It would be interesting to see how spatial information could be incorporated into the descriptors. For object recognition with local features, RANSAC is often used to remove outlier matches - is there a spatio temporal structure to certain behaviors that can be found and somehow exploited.

Secondly, I wonder if this could somehow be combined with the Lepetit et al. method that I presented earlier. Instead of running k-means on a bunch of data in the domain, construct a seperate multi class classifier for each action/behavior that will take a novel cuboid and find it's correspondence in this action, or return "no matches". You could then classify novel video sequences based on how many correspondences were found between the original action/behavior sequence, and the novel sequence.