I was initially surprised when reading this paper that tracking and model building could be intertwined. Like the paper suggested, I would have thought of these as independent problems. However, I do see now that these can be used together and result in a system that improves upon itself. (good 3D model leads to good tracking which leads to good 3D model which leads to good tracking which...)

One interesting comment they make is that initially the accuracy of the model and it's motion are poor. The technique that they use to improve this is to accumulate 3D information from several observations and then improve the entire system. I do wonder if there is another way the system can be improved without the need for many observations. They might be able to use a prediction model of the object's motion to aid the process but that might require information about the object itself. Anyone else have any ideas as to how they might be able to improve the system initially?

Also, did anyone wonder about the 3D reconstruction in figure 5? Why does it seem that the roof from the front of the church is not correct? Could this possibly be an artifact of going from a 2D wireframe to 3D? Thoughts?

Anyway, these results seem good. I wish there was a video so I could see this in action. I tried checking the author's website but no luck. Jing-han, any chance that you have access to a video of this working?

Edited by author 11-20-2003 09:24 PM
Matt, to answer your comments/question:
The 3D reconstruction in Figure 5 certainly seems weird. It is missing some edges. I've been wondering what the red dots in the tracking images mean. I think it is some form of outlier. Those might be the ones whose depth variance is too large. I couldn't find author's explanation. Anyway, looks that the roofs are where these outliers seem to appear. That is probably why the edges disappeared from the 3D reconstruction.

http://www.cs.ubc.ca/~mbrown/meng/meng.html
The above link has 5 short videos that I will be showing during my presentation. However, it only shows the final results. I kind of wanted to see how the model and tracking help each other in the process, but alas, the video is not provided.

A couple of questions that I am going to toss out there since I've been wondering about them myself... (Some of these may seem uber easy for some of you, but again, I don't have a strong background...)

1. In section 2.1, the authors say that in tracking single line segments, each new line segment add 4 degrees of freedom. What are these 4 degrees of freedom? (e.g. translation? rotation? scaling?)

2. Looking for explanation in section 3.2, when the authors try to derive the covariance matrix that takes account of image motion due to errors in the depth estimation of the vertices, they neglected "terms due to coupling between points", why? Why is the effect desirable?

Figured that I might as well put a little introduction to the paper.

Like Matt mentioned, this paper introduces a somewhat novel idea of combining model building and tracking. Normally in tracking, we find point correspondences, or have a pre-defined model of the object we want to track, and just search for that object according to the model in each frame. However, complex object models aren't readily available, so we usually track simple shapes.

The approach introduced in this paper is to have users input the wireframe of the object being tracked, and track the wireframes. This is better than simply tracking the input edges since it reduces the degrees of freedom.

Given 2 frames, we can find the object of interest by tracking the wireframes. Also, 2 frames gives us some 3D information of the object, and triangulation can be used to calculate the 3D position of edge intersections, or vertices. A Kalman filter is used here to maintain the 3D information (3D pdfs) for each point.

If we have the model of the object of interest, then we can easily compute the velocity of the image points from the camera's velocity (translational) and angular velocity. However, since initially, the model given by the user isn't really accurate, we need to accumulate the 3D information during tracking to improve the accuracy of the model.

The resulting framework is a probabilistic one that combines image motion due to Euclidean motion of the vertices in 3D (calculated from camera motion) and image motion due to errors in the depth estimation of the vertices.

Jing, I believe the 4 DOF mentioned is a similarity transform (someone correct me if I'm wrong). Thus, this would be rotation and translation with isotropic scaling.

Ok, retracting my previous statement about the church 3D model.

The edges that seem to be missing is because one face of the church is occluded from view. Since that edge was never present, the system does not know of its existence and cannot put it into the 3D model.

If that face were to appear in subsequent frames, then the model should incorporate the proper edges into the model.

Just so everyone is clear, the wireframe simply means connected line edges, not pre-defined 3D models. The wireframe is represented by the intersection vertices of the line edges.

I really like the concept of using tracking to improve the model building and vice versa - however, I'm wondering (because this paper doesn't explain it very thoroughly) how the application of this as bootstrapping would really fit with model-based tracking. Specifically, what applications would you have the time to manually initialize a model, then run this technique to improve the model, then track something?

I think the degrees of freedom for the line segments are the 4 points that represent the line ends, 2 for each end. (x_s, y_s) to (x_e,y_e), where s is start and e is end.

This variational approach to find the distance of edges is an interesting one. I'm intrigued with some of the calculus of variations techniques to find corners, circular objects and so on.

This paper presents an interesting idea of usingthe model parameters to determine the prior probability of the next location using euclidean motion of the vertex. For this rigid wireframe model of a wooden polyhedral object, it seems very befitting to allow some of the variables from teh model to decide where the next measurements may lie.