While the paper does make a good number of assumptions, I am amazed at how it was able to continue tracking a person in the center image of Figure 10. The person has all but disappeared in that image.

The paper also makes the observation that background shapes cannot be deteremined without actual movement of foreground objects. While it may be true in the context of the paper, there does exist some information within a static background that can cue us on the "approximate" shape of the backgrounds (such as textures). I wonder how that extra information might be used to improve on this paper's methods.

I don't see why Figure 4 is such a hard problem. The object is only moving horizontally, and the background doesn't change. How would shadow affect tracking? The shadow didn't change in the synthetic video, and it's not blocking the object, so for all its worth, it's just a constant background (like trees in the other video sequences). I don't see reflection either... Also, why would transparency cause a big problem? If anything, wouldn't it aid tracking because a little bit of the object is still visible (appearence model should be able to pick it up)?

In the human tracking sequence, one person is wearing white and the other is wearing black. This should help the appearence model significantly. Would the algorithm be successful in differentiating the two persons if they are wearing the same color clothing?

Having said that, this is still a cool algorithm. Like the authors said, it models more possible interaction between foreground and background objects.