For inpainting stationary backgrounds:

I'm surprised there's no averaging/smoothing taking place when filling in holes. Even for static scenes, there are likely to be small changes over time. If you're filling in a hole by grabbing neighboring temporal patches, when the hole is finally filled it seems like you'll have pixel values from points originally far apart (the beginning and end of the occlusion) now adjacent. Similar situation for spatial filling as well.