A quick question about matching. When performing matches between the candidate frame patches and the target patch, how do we have optical flow values for the damaged patch? Given that the patch of pixels is missing, how can we calculate the optical flow in order to match to the candidate frames. Also, it doesn't mention this, but I would assume that the calculation of the SSD between the candidate and target patch is done across only the foreground pixels?

For inpainting stationary backgrounds:

I'm surprised there's no averaging/smoothing taking place when filling in holes. Even for static scenes, there are likely to be small changes over time. If you're filling in a hole by grabbing neighboring temporal patches, when the hole is finally filled it seems like you'll have pixel values from points originally far apart (the beginning and end of the occlusion) now adjacent. Similar situation for spatial filling as well.

1. I think the repetitive constraint for the foreground need to be enhanced to repetition with same frequency/speed. Otherwise the optical flow component used in SSD matching will not match.

2. Is the "temporal search of best matching foreground frame" frame by frame or shot by shot? If it is frame by frame, is it possible that the corresponding matched frames of consecutive occluded frames are not consecutive(assume the video is long enough for several repetition)?

3. Human vision seems to tolerate more artifact in video than in image.
(a) fast motion (b) low resolution & MPG blocking effect (c) complex background texture on boundary.

4. For the smoothing problem /2, since the frame copy(either foreground or background) is pixel by pixel rather than block by block. Each new pixel is determined by using existing neighboring pixels.
Therefore, artifacts wont happen except the last several pixels.

In the examples they show a black box covering a person's figure. This approach would only work if that black box occlusion is stationary, correct? If there is another moving person blocking another person for a few frames, it wouldn't work because the occlusion is actually moving right?

Thanks

I have a friend in the math department who has implemented code for this inpainting stuff. He is very familiar with the papers presented, and actually he feels there are much better papers out there on this subject than these 2. (he's going to give me the references) He has implemented code to do the inpainting, actually, he uses PDE's and Stokes' equation! For those interested, he is giving a talk on it next month!! Once I know all the details, I'll let you know!! =) Ciao!

One of the assumptions is that the foreground has periodic motion; all of the videos on their website are of people walking in a very regulated fashion. How much do you think this method would break down if their motion was more "natural"?

Also, other than people walking, what are some other applications of this method?

So, it would seem that using a median optical flow amount for segmenting background is reasonable in certain quite limited circumstances. Can anyone explain to me how this algorithm would work in the presence of a really 3D background (forest of trees at different depths who's velocities are clearly scaled by their depths)? Is that addressed in either of the papers?

This might be interesting as a form of video compression for cyclical repetitive motions. I don't think you can 'fill in the blank' in all cases, but if you could sense the right kind of motion, you can blank it out and recover it at the other end.

I don't have clear how they separate the background from the foreground when the motion confidence mask doesn't work. Maybe they consider the cases when is known to work?.

Two of the basic assumptions the authors mention include 1) the background is stationary and 2) camera motion is handled if it is parallel to the plane of the image.

If the background is far away from the foreground object (like in their outdoor examples), you can approximate it as stationary, but if the background isn't very far away from the foreground object (indoors), it will not be stationary with camera motion. In this case is their assumption violated and their algorithm unusable?

Paul Smith Shoes uk store,See discount Paul Smith large range of online,We have

Paul Smith Shoes and Paul Smith Shoes in all styles.