1. I think the repetitive constraint for the foreground need to be enhanced to repetition with same frequency/speed. Otherwise the optical flow component used in SSD matching will not match.

2. Is the "temporal search of best matching foreground frame" frame by frame or shot by shot? If it is frame by frame, is it possible that the corresponding matched frames of consecutive occluded frames are not consecutive(assume the video is long enough for several repetition)?

3. Human vision seems to tolerate more artifact in video than in image.
(a) fast motion (b) low resolution & MPG blocking effect (c) complex background texture on boundary.

4. For the smoothing problem /2, since the frame copy(either foreground or background) is pixel by pixel rather than block by block. Each new pixel is determined by using existing neighboring pixels.
Therefore, artifacts wont happen except the last several pixels.