First of all, I think that the results in the paper are incredible. The image of a neighborhood from overhead where the houses are recognized as having a smaller depth level than the ground is quite incredible as there is so little parallax in the sequence since the scene is so far from the camera.

My concern is primarily over the running time of this algorithm. I am not sure how large the gain is over a simpler approach that pools the votes for a depth layer over a segment and gives the segment the most popular depth level. For the regions in the building images which have noise when the pixels are assigned individually, a segment pooling would work well in exactly the areas that the authors point out as problematic.