We submitted a paper and the comments we got from the reviewers were hilarious. They really didn't have a clue about anything! And still, they managed to be sturdy and supercilious, totally unaware of their own ignorance. Funny enough, the rating of the paper was all over the place.

Well, enough of this silly game now and back to science. It really hurts though, to see tax money being distributed into something with such a broken peer scientific culture.

I think it is on the rise, with all technology. Look at video games these days. I feel like I am really there in the game most of the time. They will surely only get better from her.
muffin9129
http://productreviewsby.me/

Some of the problems I see with SIGGRAPH:

- the core of computer graphics has diversified into image processing,
  physics based simulation, and 10-15 other topics quite far off original CG
- the A researchers in CG that are supposed to review other peoples
  work have no or little competence in the diversity of areas that
  show up at SIGGRAPH, which results in really low quality at the end
  of the day
- computer graphics researchers tend to grab results from other areas,
  making methods into their own, producing beautiful images that sell
  the work to the reviewers. Original scientific content in these papers
  is often very close to zero but is often rated higher than work with
  magnitudes higher quality and novelty. Often CG researchers even neglect
  to cite original papers in other areas, since they can always claim that
  they discovered the method independently, and in any case none can hold
  this against them. On the other hand, if you actively neglect citing a
  CG paper due to its low quality and total lack of novely, the reviewers
  point out that this crap paper should be used as original reference,
  rather than the seminal original papers you indeed have cited. This
  is absolutely rediculous, and shows how introvert CG is.
- computer graphics is using imagery and movies as empiric evidence for
  scientific results, but in practice these images and movies are full
  of hidden parameters and all kinds of weird manipulation, entirely made
  to over-sell the novelty of whatever crap is presented in the paper.
  Obviously computer graphics in general, and SIGGRAPH in particular,
  do not follow the same rules as other areas of science in this sense.
  Synergies between e.g. numerical methods, physics, mathematics, chemistry
  etc are great in many other cases, but computer graphics is special since
  it delivers nothing at all back. N.B. Visualization is NOT computer
  graphics in this context.

Nevertheless, CG is a very inspiring and exciting area and it has tremendous potential in inspiring great research not only in computer graphics but also in areas that apply themselves in computer graphics.
Unfortunately, the leadership and culture of the CG associations counteracts this potential. This is irresponsible, and also quite sad.
It will and should result in a catastrophe for computer graphics as an academic research area, and removal of grants from the tax driven funding agencies.

How do you feel about Siggraph Asia?

3D scanning for hairstyles (critiques as off-topic for SIGGRAPH below) sounds on-topic to me. Computer vision in general, certainly! It's "Computer Graphics and Interactive Techniques", not "Rendering". SIGGRAPH will certainly have many rendering papers this year as always, but it's a large multidisciplinary conference and IMHO that's the beauty of it.

For those of you who have reviewed papers for SIGGRAPH 2008, I would like to take an unofficial poll about what you think of this year's review process, in particular, the new part asking reviewers (including all primary, secondary and tertiary) to try to reach consensus before the committee meeting.

Among the 8 papers that I reviewed, 6 of them reached unanimous consensus so that's all cool. In one of the remaining 2, I had a different opinion from other reviewers and in the end the senior review essentially twisted my arm to agree with them. In the other one, the paper received a bimodal distribution of scores (2 >= 4.0 and 3 <= 3.0) but interestingly, only 3 reviewers chimed in during the discussion process. In the end, no consensus is reached (or even attempted).

I think I like the transparency of this added phase of asking all reviewers to reach consensus, and I would say at least in the 8 papers that I reviewed, I have not seen any abuse of the review process. Even for the 2 papers that did not reach full consensus, they both got accepted so at least they are erred on the positive side; I always think it is a bigger crime to reject a good paper than to accept a bad one; for the former, I lose a chance to read a really good paper but in the latter, I could always choose not to read it or in the worst waste a little bit time to figure out the paper is bad.

This also brings us to the earlier posts regarding topics of accepted papers. I believe as long as the papers have gone through a fair and organic review process, I have no problem seeing non-traditional stuff, e.g. computer vision or computational photography. In all likelihood they could broaden people's research scope, and in the worst case you could always choose not to read them.

To #274.

Yes, the topics in the SIGGRAPH this year will be very diverse. So far I see those non-graphics topics (according to your definition):

1. Computational photography
2. Perception and color science
3. GPGPU
4. Deblurring

In my opinion, they are all very valuable to the computer graphics community. Many computational photography papers relies on the light transport theory, and their results may benefit the traditional problems in the future. For example, there is a paper that uses flash/non-flash image pairs to reconstruct the depth map. Also, if you can use those devices to achieve the special effects in acquisition, why you need to rendering them in the post-processing stage?

All perception related topics are of fundamental importance, especially when the physics-based rendering algorithms are very mature (if we don't consider the speed issue) and the expressive images mostly rely on the post-processing at the pixel shaders. Finally, when GPU becomes so powerful, porting some complex algorithm onto GPU is a great idea. As for the deblurring...I have no comment. But at least the results shown in the papers are really amazing.

Most reviewers and committee members have published numerous pure-graphics SIGGRAPH papers and still decide to make the SIGGRAPH like this. Maybe you should see the quality of the rejected papers to make the fair judgement. Also, the topics you like may kill the graphics in other's point of view. For example, I can blame measured BRDF data that kill the analytic BRDF functions. We should ask more talented people to design new BRDF functions to kill the measured BRDF data :P.

Ok, so the semi-official SIGGRAPH 2008 paper list is being compiled, and it seems it's true what they say. Graphics is dead. We now have a conference called SIGGRAPH, but most papers appear to be about yet another project where somebody used a number of digital cameras and duct tape to build yet another specialized 3D scanner of some sort. Shouldn't there be some IEEE Optics Conference or something like this? Almost all of this is computer vision at best, if not just homebrew hardware engineering. You certainly would hardly have called any of this computer graphics 10 years ago. It seems the remaining people doing graphics research can now be found in venues such as IEEE Visualization and the like.

Seriously, there is a paper on a 3-D scanner for acquiring HAIRSTYLES this year. You've got to be kidding me!

Hello all,

I am definitely out of it--I had no idea this forum was going on. But then, I more or less dropped out of CG a bit before Michael (but less ceremoniously) for mostly the same reasons, and I haven't been to SIGGRAPH in a few years. It is very encouraging however, to see so many others agreeing that there are some serious problems with SIGGRAPH as well as the peer-review process in graphics. In fact, it has revived my interest in the field. Thanks!

It looks like this forum is essentially dead and I missed my chance to post, but in case anyone is still reading it there a couple of things I'd like to add. First, I have come to learn that many of the problems described here also show up in other fields. To my mind, the "peer review" process has two basic problems: we are never formally trained in how to do reviews, and we are seldom (if ever) held accountable for our reviews. That is not a good combination. My second comment has to do with another down side of the emphasis placed on SIGGRAPH papers. If work published in SIGGRAPH becomes a "sacred cow", what happens if research comes along later that contradicts it? In my experience, it is pretty hard to publish that kind of research. I agree that graphics is getting "overcrowded" and it would be nice to have other top-tier venues.

Michael, thanks for starting this forum, and I am flattered that you included me in your "A-list."

Edited by author 11-03-2007 11:53 PM
 In response to post by 'clipi': I am still "around" and, as you see, even clean up this by now dead forum once a (few) month(s), as promised. Currently I do not closely follow events in what is now "alive, well and very vibrant" CG community and although certainly "hear things", do not consider it appropriate for me to comment on CURRENT state of affairs any longer. Of course, you will be better off not asking me for any advice in the first place given my, well, rather equivocal reputation in CG community. But who knows - maybe someone else still reads this forum and can give you some useful "insights and suggestions" if you post your questions/concerns here and maybe other people find such discussion useful as well.

 Thanks for wishing me good luck - we all need it ...
 M.A.

First of all I just want to say that I am sorry tha Michael is leaving
the field. He had made such great contributions to the CG world that
it certainly will miss his expertise. I hope at least that he would
be around to give insigths or suggestions for other people and share
his knowledge. Is there anyway other way I could personaly contact him?
Thanks so much and good luck in your new field.

I think the computer graphics community is alive, well and very vibrant. However, I am a little pissed off about grad school in general. Because funding can be difficult to come by for any research program, this is not CG specific. But I hate fundraising when it gets in the way of actual research.

 Wow - two people whose work I deeply respect (both on my "A-list" of farewell acknowledgements, so I am not just saying this now) got siggraph awards!!!

 I hope that whoever still reads this forum will join me in congratulating Greg Ward and Nelson Max for getting the well deserved (and, I might add, long overdue) formal recognition. Most impressively to me personally is the fact that neither had a huge count of siggraph papers recently. Is this a sign of real change ? Surely hope so ...

 As for the forum, well, some people seem to have expressed interest in keeping this forum alive and so be it. For one, it would be interesting to hear people's impressions from upcoming siggraph as related to this forum's main topic.

 Since nobody volunteered to take it over, I guess I will keep cleaning this place from time to time (no, there is no automatic tools to do this). Apologies if I can not do this often enough and as a result you see too much junk posts here.

 M.A.

It turned out that Michael has another cult following. See
http://hardware.slashdot.org/article.pl?sid=06/08/29/0443210

I have been on the SIGGRAPH paper committee for the last 2 years. In general, I consider myself a nice guy and far from being a paper killer. However, just like many other committee members there are certain papers that are considered good by others but for various reasons I would like to reject. Here I would like to offer a glimpse on how I exploited the current system to kill these papers. Since I won’t be in the committee next year, I hope some of these exploits will be fixed so that they won’t come around to haunt myself.

1. Try to learn as much as possible of the submitted papers and identify those that you would like to reject. In some situations, you might get to know some papers even before the SIGGRAPH review process (e.g. faculty applicants listing ongoing work on their research statements).
2. Try NOT to become the senior review for that particular paper. Even though it is unknown if the paper will be assigned to you, at least the best you could do is not to request the paper. The advantage of not being a senior reviewer is that your opinions won’t be seen by the authors during the rebuttal period. Then, during the committee meeting, you could simply jump out and request to take a look at a paper already listed as accepted (this was actually advocated by the Paper Chair himself this year). Since during the committee meetings the authors’ opinions are not consulted, you can easily bring on new issues without receiving much challenge, just make sure that you possess sufficient knowledge to deal with the senior reviewers in charge of that particular paper. (In case you wonder how a paper could fall into the hands of people without the best expertise, remember that the paper assignment, done by only a few people (Paper Chair and Advisory Boards in previous years and Area Coordinators this year), could hardly be perfect.)
3. If you are one of the senior reviewers for a popular paper you would like to reject, there are multiple tricks. One option is to have really nice language in your review (accompanying a lukewarm score like 3.2) but with a fatal comment that seems innocuous, such as “I really like the main ideas presented in this paper and the results look all very impressive. However, I felt that a certain component of the presented algorithm has been done before in [a previous work that is not cited or adequately discussed in this paper].” Or something like “This is a key part of the algorithm; however, I don’t think the exposition is clear enough for reproducibility.” The key point is to obscure the severity of this statement from the authors so that they might neglect to fully address this fatal point during rebuttal.

There are certainly other tricks that I know of and/or have seen deployed by other committee members, but it will be too long a post. Here I have simply presented a few tricks that could be done mostly alone. You might note some of these exploits are benefited from this year’s particular innovations, giving the committee members much more power than in previous years.

Hi Michael

Why don't you submit the D-BRDF paper to a journal like TOG or TVCG instead. At least a published D-BRDF paper would be much better for the CG community than a repeatedly rejected siggraph sumbission! I believe there is more than one person genuinely interested in using the D-BRDF framework for BRDFs. Hence I would suggest that it should be submitted to an alternative venue.

regards
Abhijeet

I am not sure if that is a question of fairness and I have no big concerns about fairness at this point. I have not seen a final program, but if Marc really has a paper at Siggraph this year I would be disappointed. He did make a big announcement.

Regarding post #181 by Anon:

As Marc's former student, I can personally vouch on his fairness because, despite "a spirit of generosity prevailed this year", *all* my SIGGRAPH submissions this year are rejected, an standard-deviation-10 achievement given the good reviews I have received.

Even though what happened to my papers might provide some interesting case studies (I am still scratching my head in disbelief), however, I have decided not to post them up for now because it will sound too much like whining. Instead, I will try to share what happened to me this year after next year's SIGGRAPH announcement if this board is still alive.

Meanwhile, I hope my case will cheer everybody up. ;-)

PS: my posting appeared multiple times due to network issue.

Regarding post # 181 by Anon:

As Marc's former student, I can personally vouch on his fairness because, despite "a spirit of generosity prevailed this year", *all* my SIGGRAPH submissions this year are rejected. (And if you are one of the committee members who have seen the scores of my papers, you would probably agree with me on the astronomically unlikely probability of this achievement.)

John,

These are interesting hypothetical arguements of what *might* happen if the TOG eidtors were non-anonymous. But given that TVCG already has non-anonymous editors, do you have any examples of how it has hurt their careers? In your TOG capacity, have you spoken with other journals who have non-anonymous editors and found that they agree with the flaws you mention? I wonder how each journal internally makes this decision... etc...

All 5 journals that I am an editor for have me as non-anonymous. I do not fear reprisal, but rather must act fairly. In particular, I would *never* write anything even close to what I typically recieve from Siggraph! Even if a paper of mine gets accepted, the person I correspond with tends to be way more harsh than I suspect that they would be if their name were attached to the correspondance.

Ron

PS. You're overworked. That's way too many papers for you too have to read all the reviews for. The community owes you a lot for all this effort.

Glad to see this list is still active, though I'm sorry to see that the same problems still exist that prompted the list's creation. I see we also have some nice deals on purchasing computers available to us.

/m180 suggests that TOG change its policy from anonymous editors to non-anonymous editors. That's an interesting idea but there are additional concerns.

The desire to hold someone responsible for a paper's treatment in review is driving this request, and the community seems to be upset with SIGGRAPH's treatment of papers, not TOG's. The primary concern with TOG has been keeping the process moving quickly so papers can be published as rapidly as possible, which is more challenging with TOG's revolving schedule than with SIGGRAPH's hard deadlines. The primary concern with SIGGRAPH, on the other hand, is the quality of the decisions, and the primary contributing factor to this is likely SIGGRAPH's hard deadlines for decisions (and the fact that SIGGRAPH resubmission is annual).

TOG receives on the order of 100 submissions each year, and these submissions are spaced fairly evenly throughout the year, though with some peaks and some troughs. A big difference between the SIGGRAPH Papers Chair and the TOG Editor-in-Chief is that the Papers Chair oversees the decision of about 450 SIGGRAPH submissions in one weekend, whereas the Editor-in-Chief oversees the decision of 100 TOG submissions over the entire year. As such, the TOG EiC in fact is responsible for ultimately making the accept/reject decision on TOG papers. These decisions usually (but not always) follow the recommendation of the (anonymous) AE in charge of the paper. Thus for TOG, the EiC can be held accountable for the decision and should be the responsible party if something goes wrong.

If the AE made the ultimate decision based only on reviewer input, and the AE were not anonymous, it could lead to other complicating and unfair issues. For example an AE could fear reprisal from the author of a rejected paper, or could expect quid pro quo reward from an accepted paper's author. The current process discourages that as I tend to follow AE recommendations and if I disagree with the AE I have to present a strong and convincing argument why.

There is some discussion to try to diffuse SIGGRAPH's once-a-year hard deadline into several deadlines throughout the year, which would accelerate the resubmission of revised papers, allow authors to submit work closer to when it is truly ready, and reduce the workload on reviewers/deciders to a level where they can be more careful. But to work, that solution would need a lot of support from the community to drive through some pretty dramatic changes to the current system.

In related news, in message 22 of this thread (07-25-2006 04:54 PM ET), Marc Levoy wrote "Stanford will probably have fewer papers next year (not more), partly because I plan not to submit any papers with my name on them. No rule requires this, but I am committed to avoiding the appearance of bias." He also made a *big* deal out of this fact at last year's "How to fix SIGGRAPH" meeting that was held during the conference.

I like how Marc's commitment to avoiding the appearance of bias extended all the way to his submission of "Veiling Glare in High Dynamic Range Imaging"...a (surprise!) paper with his name on it that was accepted for publication in SIGGRAPH this year.

I would not have had any problem with him submitting papers as previous papers chairs have done, but this was fundamentally dishonest and in my opinion he should be taken to the woodshed for it.

Edited by author 04-04-2007 02:42 AM
Michael's last post points out that things are no better this year. In some ways they are worse. This year I got my 15th and 16th Siggraph paper, so overall I shouldn't complain. But on the other hand I just received the most uneven treatment *ever* on one of my submissions.

We had a submission with better reviews than the two that were accepted, and it had no real criticisms, and the lowest score wasn't even that bad. So we had very little to say in the rebuttal, but alas the paper was shockingly rejected. I've published about 100 papers across various fields in conferences and journals and never seen one with such good reviews and only superficial concerns rejected before.

The main reason given was that we violated anonymity in the rebuttal. Since a reviewer pointed out that there was prior art to our work, we had to cite our previous sketch to rebut this, but aha, catch 22, you either violate anonymity or get dinged on novelty. Yes, I bet the people who rejected this paper are very pleased with themselves - at the clever method they found to reject the paper. Congrats geniuses!

One of my students (first author on this paper) has 6 Siggraph papers at the ripe old age of 25, and has decided to quit academics and go to industry. He's quite unhappy with the intentional bias and stupidity that he has seen in the Siggraph reviewing process. Again, these reviewers are probably quite pleased with themselves, as they removed even more competition. The second author on that paper had wanted to dedicate his career to teaching even over research - although we could use more people like that in academia, he too is shaken by what he has seen in the Siggraph process.

To top this all off, let me comment on anonymity. In two previous years I had papers to referee that had the authors names plastered across the front, and was told to ignore it by the conference chairs and proceed anyway. Another year I found the paper on the authors website, and again I was told to ignore it. And yet a fourth time, I came across a submission that had been published almost verbatim in a Siggraph course the prior year, and again was instructed to ignore this. And... this year, I heard from the conference chair (Levoy) that he explicitly instructed the committee to ignore any breaches of anonymity.

Yet in our posting, the #1 reason for rejecting our paper was given as our breach of anonymity. Again, I can only marvel in awe at how clever these people are. Not only did they directly ignore what the papers chair said, but they wrote about it in our BBS posting. Of course, you say, we could complain, especially since they wrote about it in the posting. Well they might as just as well wrote "ha ha, we got you" as the main reason. Because there is no recourse, the decision cannot be reverted, and to save face you can bet that we will hear back... "well... there were some other issues too... sorry, try again next year".

These reviewers also know that there is no editor assigned to the paper as there is in almost every other respectible area of research. Typically an editor is supposed to be *non*-anonymous, make sure the rules are followed by the authors and reviewers, and make a final impartial decision on the paper. Well in Siggraph one or two of the reviewers get to play in the shadows pretending to do this - with zero accountability. Sure, you say that there is a papers chair. Well the reviewers also know that one papers chair can not be held accountable for hundreds of papers, and thus it is these secret reviewers with no accountability that get to pull the trigger however they see fit.

Siggraph needs to assign a name to each paper. Someone that can ensure that the reviews are fair, without bias, well informed, and geez that the reviewer at least reads the paper. Actually, I'd settle for looking at the pictures and reading the captions - some of our reviewers on other papers didn't even do this.

The primary should *not* review the paper, but instead but be assigned in a non-anonymous way to the paper as a judge, to make the final decision, and tell the authors exactly why he agrees with the reviewers. And be accountable. "Ha ha we got you" should not cut it. I am an editor for a number of journals, and have to send email accepting and rejecting papers all the time, and it is my reputation, so I am fair, and careful, *and* correct myself when I goof.

And ACM TOG has the same problem. Secret shadow editors. John Hart, you are in charge of that journal. FIX IT! IEEE TVCG does not operate that way. Maybe if you fix TOG, Siggraph will follow suit - maybe they will *have* to follow suit to stay an issue of TOG. Or maybe the powers that be will drop out of TOG then, making it obvious just who the secret system benefits - those illustrious committee members. Yes, of all the students and young people in the area, let's make sure to protect that committee.

Michael,

Please leave this board up. This year's round of reviews is coming out soon, and I'm sure people will be congregating here. I respect your courage and decision to leave this world behind, but maybe you can watch from the sidelines to let those of us still bound to it try to make some progress from the inside (well, inner periphery).

(Do you have the ability to easily filter "lenovo" or "nokia", etc.?)

Hello,

I'm a young researcher in computer graphics without any SIGGRAPH publications; thus, you probably want to stop reading at this point. Thanks for your attention, have a nice day.
 
Also thanks for this very interesting discussion. My personal suggestion to improve the SIGGRAPH reviewing process would be to establish some specific rules (as opposed to "guidelines") for writing reviews. This would require the community to actually state precisely what the duties of a reviewer are. It would also give authors a chance to mention violations of these rules in the rebuttal process and might thus help to improve the quality of reviews in the long term by teaching reviewers (and their students) how to write reviews. For the short term one might also think about a right of authors to reject particular reviews in the rebuttal process if there are clear violations of these rules. (The primary reviewer has to make the final decision and therefore should not be anonymous because he or she is responsible for this decision.)

Secondly, my suggestion for frustrated authors would be: if a video ... oops ... if a manuscript is rejected at SIGGRAPH, don't wait a year but submit the manuscript to a journal that gives better opportunities to discuss shortcomings of the manuscript and requires reviewers to defend their statements. Yes, it will take longer to publish the manuscript this way but maybe not as long as waiting for next year's SIGGRAPH. Also, journal publications are valued higher than conference papers (even if the proceedings have journal status) by many people outside of computer graphics and might therefore be even more important careerwise, at least sometimes.

Thirdly: don't expect life to be fair. As Sheryl Crow put it:

   No one said it would be easy
   But no one said it'd be this hard
   No one said it would be easy
   No one thought we'd come this far


Best regards

Martin Kraus

 Dear all,
 This place is clearly dead and is just filling up with junk advertisement posts. I guess the proper action for me is to close it down very soon assuming there is no new posts...

 Before I do so, one last story: one of CG researchers who got familiar with my ill-fated BRDF paper thought that there is value in it and asked me if it is worth to resubmit it for siggraph 2007. I replied that I am burnt out by whole siggraph system, have no time to do anything for the paper, everyone who makes decisions already saw it and made up his/her mind (so there is zero chance of success), but he is free to do whatever he feels like. So he made an editing pass, improved some images and submitted it. He forwarded me the reviews a few days ago and of course I could not help looking at them.

 It's just so sad how ABSOLUTELY NOTHING has changed in this past year. I am very tempted to post at least two out of five reviews here (reviewers #1 and #5 in the system, if these people read this) as almost perfect illustrations to everything which I was talking about more than half a year ago. I am not doing this only because this would be a violation of unwritten rules and although it would be quite difficult to screw up my CG career more than I already done on my own, this might impact the researcher who submitted the paper. Let me just say that "explain your rating" section, i.e. the actual review, contained 61 word (4 lines of text) in review #1. Prolific writing of reviewer #5 had no less than 96 words, ~6 lines of text. This is on a 10-page paper which I still think is a bit more than total junk (I do make an outrageously bold claim of knowing something about BRDFs). And somehow I do not think I am just unlucky here ...

 Finally, I just stumbled upon Lee Smolin's "Trouble with Physics" book, significant part of which is devoted to exploring systematic issues leading to what author considers a lack of real progress in fundamental theoretical physics over the past 30 years. CG is a much younger field and is really not a science, but it is scary how much of what he is writing about can be directly (or almost directly) applied to this area and echoes (in a much more eloquent way, of course) some of the things I mentioned last year. CG is a fast evolving area indeed - we managed to reach the state of stagnation about 10 times faster than theoretical physics people.

 M.A.

I think there is another bad thing also possible because of SIGGRAPH: almost all the good conferences select their deadlines close to SIGGRAPH. Look at next year's conferences:
SIGGRAPH: deadline Jan 24, Notice Mar 19
Eurographics: deadline Feb 8
Graphics Interface: deadline Dec 18, Notice Feb 19
Computer Graphics International: deadline Jan 5, Notice Feb 20
CVPR: deadline Dec 3, Notice Mar 31
...
If you want to submit your paper to any of them, and do not want to submit one paper to multiple conferences in a same time, and if unfortunately your paper is rejected, there will be few good conferences you can submit it again. :(

 Dear all,

 It has been a while since I had a chance look at this forum (new job is a new job) and the most amazing result to me is that people still post here. Although I remain pessimistic about the final outcome, I start to think that maybe there is some value in keeping this around. If nothing else, people might vent their anger and feel better (if this survives till March when siggraph results come back, I am sure there will a great need for that).

 Unfortunately, as such things go, this froum seem to be quite vulnerable to obvious spam. I just removed several recent invitations to buy junk from here (I hope people do not mind me using administrative permissions in this case) but I am sure it will come back in volumes. Since I can not clean up this place on regular basis, I was wondering if anybody has enough time to take on the task? I would probably have to give my QuickTopic login to this person, so I hope you understand that I would prefer this to be someone people know and everyone can trust. If interested, write to my old but still working account: ash AT cs.sunysb.edu (which is, sorry, not being checked very often either)

 Alternative would be moving this discussion to another place which has some kind of anti-span tools or more restricted posting access.

 Finally, this place can just be kept as it is now. From time to time, whenever I feel particularly nostalgic, I can come here and clean this place in (probably unrealistic) hopes that in the mean time span does not grow to swallow all the rest.

 Any other suggestions ? Thanks,
 M.A.

 P.S. I would like to again thank everyone who had kind works for me (posted here or send via e-mail). I only hope you will be as kind if I ever try to return to graphics.

Some informal descriptions of research challenges in the view of various individuals who have been involved in computer graphics for some time can be found at:
http://www7.nationalacademies.org/CSTB/pro...raphics_papers.html

I am not a leading researcher, but I recall a challenge of "rendering/simulating spaghetti" in an earlier SIGGRAPH proceedings introduction. A little searching reveals that it is part of Jim Blinn's 1998 keynote, which contains such a top ten list:

http://old.siggraph.org/s98/conference/keynote/blinn.html

It seems that some of these have been solved, and some are too abstract to have permanent, satisfying solutions. But it's interesting to see where the priorities were.

Thanks to Michael for starting this discussion thread. It is refreshing to see researchers introspect and question the system which is so very heavily biased towards publishing at a glamourous conference such as SIGGRAPH.

I have a simple question addressed to Dr. Levoy or other leading academic researchers in graphics. Every self-respecting active "hot" scientific discipline should have a set of top unresolved problems that people are trying to solve.

Can you give me a list of the top 10 GRAND CHALLENGE problems that the field of graphics is trying to solve at this point in time ?

Dear Veteran, thank you for your answer,
anyway, I've got good reasons to remain optimistic,
for instance, it is still possible for a perfect outsider to have
a paper accepted at Siggraph (I'm aware of several stories like
that, including my own story), this means that everything is
possible even if you are not a member of a "clique".

However, I agree that the way the PC works is not perfect. For instance,
two friends of mine co-authored a paper that got an average score above
4 and that was finally rejected, was it cornered by a paper-killer ???.
I also once reviewed a paper that was full of errors (recommanded rejection), but that had a well-designed video, and that finally got accepted. That's life, no matter how many reviewers you use,
no PC can be perfect.

Anyway, if the only way to get an academic position is to publish at
Siggraph, is this Siggraph that should be blamed ? I've seen in a post
that Peter Schroeder said that Siggraph needs to be "demistified", I think he exactly gots the point: commitees, students etc... seem to
think that Siggraph is magic. For instance, one of my Ph.D. students
asked a question to a student that did co-author a paper this year
(like: "does your method also work in some specific case"), and the
other folks just answered "Hay, you are at Siggraph !". I think this
is this perception of the conference that we need to change in the
CG community rather than the way the conference works.

All the anonymous posts in this thread give also more strength to the "mystification" (if I put my name here, a white lightning will
disintegrate me, or even worse, will disintigrate all my future
submissions to Siggraph ??)

  Why is Siggraph seen like that ? OK, it has high-quality papers, there
is a good impact factor etc..., but this does not explain everything.
I think this is also due to the fact that it is so big, maybe separating
the tradeshow from the conference would partly restore a more normal perception of this conference.

To end my post, as Veteran said, CG is not the only case, see
this thread about FOCS (Fundamentals of Computer Science)
  http://weblog.fortnow.com/2006/06/focs-accepts-and-movie.html
In particular, have a look at the posts of a folk named Mihai

And at least, since Siggraph is not as prestigious as Fields medal,
we do not have journalists, have a look at that story:
http://science.slashdot.org/article.pl?sid...21/1442239&from=rss
http://www.doctoryau.com/

Edited by author 09-17-2006 12:33 PM
I know it won't happen, but please let's not even consider having a winter SIGGRAPH!!!

We need to realize that there's life beyond SIGGRAPH. We have journals, I3D, EG... I know a lot of people in rendering that, had they just one shot at attending a conference, would rather go to EGSR than SIGGRAPH, and that's pretty healthy (publishing a paper is a different issue, though...). The winter SIGGRAPH idea will just reinforce the impression that it's SIGGRAPH or nothing. All borderline summer rejections would of course become winter submissions. All winter rejections would immediatly become summer submissions again. Tenure committees will scan CV's looking for SIGGRAPH papers only, even much more than now (hey, you now have twice the chances, how come you don't have 2 or 3 SIGGRAPH papers per year?). We either demistify SIGGRAPH or, as Peter Schröder said during the Town Hall Meeting, we'll be "eating our own children". Great intervention, Peter, BTW.

A very good idea I found a few posts below is the restrictions on video. The level of refinement expected, profesional voice-over and all, is reaching more unchartered heights of absurdity by the year. Having a committee member say "the video looks so good we have to find a way to get this paper in" is a pretty serious indication that we're treading the wrong path. Force the video to add useful information beyond what is in the paper. Don't let them be flashy showoffs.

As for reviews, I have nothing to say. My SIGGRAPH reviews have been fair so far, both for courses and papers, even though I of course didn't always agree.

Bruno states:
"Do you think that simply expressing your views in this forum can
have a bad impact on your careers ? Do you really think that
the situation is *that bad* ?"

Yes Bruno, that is exactly what I think. I hope your optimism turns out to be valid, but I doubt it. Not that graphics is different in this regard than any other field...

Edited by author 09-07-2006 08:10 AM
I'm also a new researcher, and I would also like to add a few thoughts to the existing list of excellent observations and suggestions.

First of all: On a paper-by-paper basis, the individual acceptance decisions at Siggraph probably are not always fair. This is as true for Siggraph as for any other scientific conference. Such is the nature of peer reviewing; the evaluation of the novelty and usefulness of a submission is in the eye of the beholder. Does this mean that the paper selection process at Siggraph is worse than in other places? Personally, I do not think so. The reviews I got for my Siggraph submissions so far have always been detailed, and it was clear that the reviewers spent a lot of time reading the paper, pondering its contribution, and writing the review. Although I did not always agree with the conclusions the reviewers had come to, the diligence with which the reviews had been prepared definitely matched or even surpassed the level of quality I have seen in many other places (conferences and even scientific journals).

Still, there are some aspects of Siggraph which I also find annoying, and although these have been mentioned before, I would like to put an emphasis on them:

1.) Computer vision / image processing papers. Here, I disagree with post #75. I do not think that pure image processing and computer vision papers should be presented at a computer graphics conference. The number of "paper slots" at an event like Siggraph is small enough already. You have to keep in mind that every image processing paper which is accepted at Siggraph possibly leaves another researcher in core computer graphics, who may have done very good work in his field, frustrated and disappointed. I do agree that many of the CV/IP-oriented papers are interesting and useful for computer graphics. The same, however, is true at least to a certain degree for developments in a.i./machine learning, hardware design, software design and programming methodology, networking, sensor development and so on. I am pretty certain that nobody here would support a pure "software engineering" session at Siggraph. So why should we have one on image processing? Computer graphics is the process of converting abstract scene representations into images, as well as anything which is more or less closely related to this. This is what you are taught in your Introduction to C.G. course, and I think we should stick to this definition.

As a random example, "Fast Median and Bilateral Filtering" presented a technique which converts one image into another. Is this a good paper? Yes. Is it useful? Yes. Will it probably be helpful for future computer graphics research? Yes. Is it a computer graphics paper in its own right? Definitely no. Not even in the least.

2.) The video mania. The custom of having to send a video demonstration along with your paper submission is really getting out of control. The official Siggraph submission guideline urges researchers to send a video demonstration, and most submissions are accompanied by one. These videos are getting more and more elaborate, almost to the point of being ridiculous. These days, a Siggraph paper video has to have professional-quality voice-over, must be several minutes in length, and should have professional-grade special effects and editing. This leads to a number of undesirable consequences:

- It is not rare that researchers have to spend as much or even more time on the "presentation" aspect of their Siggraph submission as on the research itself. I have experienced this myself and heard similar stories from several colleagues. This means that highly qualified and motivated researchers in our field are forced to spend a considerable amount of time each year on this task, which is certainly not helpful for the overall progress.
- While the video demonstration is only meant to accompany a paper, its actual significance is continually increasing. As a reviewer, most people have a look at the video first because watching a video is easier and more entertaining than reading a multi-page scientific text. This, however, means that the reviewers have a strong opinion about the submission before they even read the actual paper. Given that a reviewer evaluates several papers, those with comparatively polished video demonstrations stand out, regardless of their actual scientific value. This is damaging to the scientific quality of the entire paper program! We should select the best research achievements, not those submissions with incredibly elaborate video demonstrations.
- Most people in English-speaking countries are not aware that the emphasis on the accompanying videos represents an additional unfair disadvantage for non-native English speakers. While it is relatively easy for English-speaking groups to produce a good voice-over, which is now an absolute necessity, this is very difficult for other researchers. Any verbal explanation recorded by an author or other lab member who is not a native English speaker will typically have some sort of accent. This makes the resulting video demonstration appear unprofessional and second-grade compared to the tv-commercial-style videos submitted from English-speaking countries. This represents an additional obstacle, considering that producing a high quality scientific text in a foreign language already is challenging enough! As a matter of fact, I have more than once had the impression that certain research groups team up with colleagues from English-speaking countries mostly to get "language support" for presentation aspects like these.

While it would be silly to demand a ban on accompanying video demonstations, I think Siggraph should impose certain limitations on the submitted videos. These would significantly reduce the amount of time required to produce them and increase the fairness of the process for foreign researchers:

- Videos should not contain an explanation of the entire algorithm or method! This is what the paper itself is there for.
- Only results should be presented, in the sense of an animated version of the images in the Results section of the paper.
- NO AUDIO TRACK AT ALL! No music, no sound effects, and no voice-overs.
- Videos should be short, maybe 1 or 2 minutes at most.
- Maybe it would even be possible to somehow supply some sort of a "video styleguide" with defined title screens, fonts, video resolution etc. This way, the presentation of all videos would look the same. People would not have to waste time on editing and special effects. This is the reason why there is a styleguide for the actual paper, so why should we not apply the same principle to videos?

I'm a new researcher in the field (I've been working for a tenth of
years, published a couple of papers at Siggraph and tertiary-reviewed
approx. 5 Siggraph papers per year since 98).

First, there is something I want to say:
There is something that strikes me, most people posting in
this thread remain anonymous. What you guys are afraid of ? Do
you think that simply expressing your views in this forum can
have a bad impact on your careers ? Do you really think that
the situation is *that bad* ? I do not think so!

I've got an experience of Siggraph from both the paper author and tertiary
reviewer point of views. As a paper author, most of the reviews I
received were well detailed and fair. The only thing I regreat is that
the only way of getting a paper accepted is to present a completely
finished "product" (including the "TV commercial") rather than a new
idea. This sometimes closes the door to fresh ideas, but fresh ideas
can also be submitted to other more specialized conferences / journals.

As a reviewer, I'm always trying to do my best. It's a hard work,
in most cases you know where the paper comes from, and you need
to avoid getting influenced by the fact that you know who the authors
are (it's especially true when they are well renowed people).
I think that the anonymous submission is only effective for newcomers
(but this is not specific to Siggraph)
Then, I think that it may a good idea to involve the tertiary reviewers
in the rebuttal process, as suggested in a previous post. When you write
your review, you do not see the other reviews, and sometimes discussing
with the other reviewers and authors may help better explaining / detailing
the reviews and improve the quality of the paper selection process.

I see some people argued that papers about image processing/computer vision should not show up in Siggraph; on the contrary I think we should thank Siggraph to accept these papers. I try to explain the situations of these two communities, so people can understand why so many papers about IP/CV submitting to Siggraph. I think on this topic, people should blame conferences of IP/CV instead of Siggraph. There are many things in other communities worse than Siggraph in graphics community.

Let's take a look at image processing community. To be honest, the quality of most conferences directly related to IP (ICIP, ICME...) is much lower than that of Siggraph. The acceptance rate is higher (so is the acceptance number), the review process is worse (author name is visible to reviewer), and usually you can only have 4 pages to describe your work. Most people will try to find a better place to submit their papers if they really do a nice job.

For computer vision, it is a different story. The acceptance rates of top conferences (like ICCV, ECCV, and CVPR) are amazingly low (see the website in post 74). Few people will check the paper list of CV conferences, and it means few people can see your work (Here I mean normal people and investors). So here is the problem: Siggraph has higher acceptance rate, Siggraph has more attendee. I can easily put the term “image-based xxxx” in my paper, and even more, I use OpenGL in my demo program! Why can’t I submit my paper to Siggraph?

Ok this is too ironic, but I really think there are too many "image-based" now. Most papers are great and they can be accepted even without that term. As long as the paper is good and related to human perception, I don’t think Siggraph should exclude any specific area.

It is not just in Siggraph but in almost all computer graphics conferences the acceptance rate is not very high. I review for many conferences, I see many papers with good ideas rejected. I think that this is very healthy for the computer graphics field.

I discovered the following page that can give you an idea about acceptance rates of many graphics conferences.

http://vrlab.epfl.ch/~ulicny/statistics/

As you can see, acceptance rates for almost all conferences are lower than %40. This list does not include many graphics related conferences such as Solid & Physical Modeling. But, if you even look at those conferences you will see that acceptance rates are also at most %40.

I think this is an important information for the tenure track faculty or the PhD students who consider academic careers. Siggraph is not only venue for publication of your computer graphics research. There exists many conferences you can publish. It also mean that if you send your paper only one conference it will be rejected with at least %60 chance. You should not take it personal. Read reviwers' comments and improve the paper and send it to another conference.

One good thing about high rejection rates, reviewers are friendlier (seems controversial but that is my experience). If the reviewers know the field, they usually try to be helpful. After carefully respond the reviewers' comments and sending the paper relevant conferences, if your paper is still rejected (say more than two conferences), you can safely assume that the paper is in the lower 10%. Then, you can retire the paper. But, this still does not mean that the ideas in the paper is not good. There may still be a good ideas in the paper but the paper is probably not a good fit for existing graphics research conferences.

The main problem with retiring the paper is that you may lose ownership. Even if the paper is not accepted, you may still want to say that you are the first to find these ideas. The simplest solution is that as soon as you finish the paper, make a technical report and put pdf file to your homepage. Don't be afraid of people who can steal your idea. It is very hard to reproduce the results in a short time.

If a paper is rejected only from Siggraph, it is not a good idea to retire the paper. Submit it at least one more conference. It is better not to make visuals Siggraph related. So, you do not have to create new visuals for new submissions.

An important concern for academicians is the effect of publishing in Siggraph on tenure & promotion. For tenure & promotion, outside reviewers are tenured graphics faculty. They have a good understanding of the field. I do not think it will matter much where you publish. They will mainly look at the quality of your publications. I have been in Tenure & Promotion committees for several years, I can say that for P&T committees and university administration, Siggraph does not mean much. As far as outside reviewers like your publication record, you are fine in terms of publication.

I think the real problem is not really tenure & promotion or publishing. The real problem is that perception of the Siggraph among graphics practitioners and graduate students. Without a Siggraph exposure, many interesting ideas cannot find market easily. Better students choose to work with faculty who frequently publish in Siggraph. This is really a problem for ambitious researchers who wants to make a quick impact. But, I believe that strong ideas will eventually be known wherever they are published.

Siggraph cannot find any real solution to this perception problem. It is hard to change our perception. It may even be better if Siggraph does not function well. Then, the other conferences will start to be perceived more important. In fact, since it is possible to meet with more people with similar interests, smaller conferences with high rejection rate will be much better than Siggraph for a tenure track faculty.

So, to me nothing is broken in the system. The real problem is our perception of Siggraph.

This is a reply to "insider". I do not think the "few really good papers get in". My own stats are that I have around the same number of my papers in SIGGRAPH and really good papers submitted to SIGGRAPH. The overlap of those groups is at most one. SIGGRAPH is simply a mess. Where the weights are on incompetence and corruption is debatable. I do agree that people put a huge amount of honest effort in. I also think that effort is worse than wasted. SIGGRAPH is a cancer on the community. It is like the Iraq war. The sooner we take an honest look in the mirror, the better.

Edited by author 08-29-2006 08:03 PM
Computer Graphics research has, sad to say, been much less exploratory, less cutting-edge, than it could be. There are relatively few radically new ideas. The law of diminishing returns places an absolute limit on what can be obtained by reification of prevailing theory or refinement of prevailing techniques.

Normally, in any arena, you can expect a totally revolutionary concept every few months. Of those, a few may emerge into the mainstream with perhaps one a year gaining dominance in some field. Of those that succeed, a few will become so dominant that they become synonymous with that field for a while.

You should also expect an idea to mature in roughly ten year cycles, starting from pure theory and evolving to become a polished commodity item that is simply a part of how things are. It took about six such cycles to move from written theory on what a computer was, along with prototype proof of concepts, to machines that were almost ubiquitous.

How does the field of Computer Graphics live up to this? Very well, for the most part. It is relatively easy to track the history of image rendering, display technology, scene description, graphics languages, graphics acceleration, etc. There are hiccups, where an idea doesn't survive for any one of a number of reasons - including political one, but in general progress has followed the path one might expect of any technology.

"Most part"? Well, over recent years, some areas are not progressing at the rate that might be expected, and new areas are not emerging at the rate one would anticipate for a field not short of intelligent and creative people. Based on both the typical and observed rates of progress in computer graphics, this is what the state of computer graphics should be:

• Research "raytracers" could be expected to use the wave theory of light, support interference and treat frequencies as relatively continuous and shiftable. This allows for the correct handling of edges (where light need not travel in a straight line), diffraction, the impact of the difference in path lengths, the ability to visualize in different ways*, etc.
• *I'll add this as a seperate item, it is so important. Not everyone has three colour cones. Some people have only two, some people have four (so-called tetrachromats). Existing rendering systems are largely incapable of optimizing for such vision without a lot of hard-coding. Nor is there any display technology optimised for such users.
• Talking of display technology, it seems to be either/or. Either you have a higher than normal resolution display (eg: the double-width systems fashionable in CAD), High-Dynamic Range (eg: 32-bit floating-point colours), or an extremely high refresh rate. Sometimes the problem is refresh rate, sometimes it's mechanical, but it's always avoidable.
• Commercial "raytracers" would be expected to replace both rays/cones and radiosity with a statistical function showing percentage reflected for a specific channel (red, green, blue), providing diffuse and direct reflection in a single operation. Discussed in SIGGRAPH in the 1980s, I seem to recall.
• Full "CAVE"-type virtual reality should - by now - be buildable in kit form in the garage, in much the same way early radios, televisions and computers were in their respective developmental eras.
• "Line of sight" calculations (for clipping, etc) are correct for engineering purposes but are incorrect for photorealism, because a plan view is not the same as an eye's view.
  

Modern computer graphics are really not significantly further on than they were 5-10 years ago. Progress has occured, but at a snail's pace. What is now research should have become a standard feature. What is merely imagined should have become a hot-ticket research item. Oh, sure, there may be counter-examples, but my point is that it's not happening on nearly the scale one would expect or at the pace one would expect. The past 5-10 years has been essentially stagnant, and that is NOT good.

Edited by author 08-29-2006 06:03 PM
I am an artist in the Movie industry and have a group of outcast talents forming a company. We were outcast due to our effectiveness and the hurt egos that our success engendered on an upcoming project. We haven't been to Siggraph in the last few years because we believe that E3 has a better look at what's Happening for the future, That said, As I am starting a company I may have to present or be present at a future Siggraph so my real name is not posted here. In any event we are trying to do breakthrough work in Mental Ray on Human Skin using what artistic skills we have, with very little in the way of math to back us up. I wonder if The anger that drives us to "Show these big companies a thing or two" also would drive you in a similar way to bypass the academic route and instead band with us to make history in spite of them.

Contact me to establish a dialog. MentulRaymond@radiofree.tv

Having served on the SIGGRAPH papers committee multiple times and having been its chair in the past let me provide my own "inside view" of what goes on. First, the chair volunteers a good portion of 2 years assembling and chairing a committee of 50 peers. The committee voluteers most of 2 months of their lives to reviewing and soliciting reviews and then ultimately deciding the fate of 500 papers submitted. Tertiary reviewers contribute untold hours of their time. Of the papers chairs I have witnessed work, every one of them has, to a first approximation, done everything in their power to create and run a fair process. Of the 50 committee members, I would say on average 49 out of the 50 are equally committed to performing in an honorable and fair way. They do the best they can given their often diverging interests, knowledge, and personalities. They should all be applauded (except that 50th member perhaps...).

That said, the process is very noisy and often inconsistent. Many good ideas have been expressed below by others on how to make it a better process, but none of the ideas will really "solve" a lot. In general, the few really good papers get in. The really bad papers get rejected. But as in most distributions, the bulk of the papers are somewhere in the middle. There is no way a group of 50 humans can, in any truly objective manner, separate the pretty good, from the good, from the almost good. But those 50 committee members and the hundreds of tertiary reviewers do their best. If you could be a fly on the wall at the committee meeting, I am certain (almost all) of you would be impressed with the honesty and integrity and diligence with which the work is done.

Keep the good ideas coming, but please rest assured that those in charge of the process are both listening to you and trying to do their best.

Edited by author 08-29-2006 05:04 PM
I think a large part of the problems seen here with SIGGRAPH is due to its cross pollination with the arts community. To put it in perspective, one should take note of the artistic pretention required to get something into an art gallery or museum, regardless of whether or not it is computer related.

One specific example I saw a year or so ago at the Fowler Museum in LA. The museum was doing a themed set of exhibits on rice and culture. There was an interactive exhibit that used videoprojectors and cameras, that would track the motion of people in front of a large wall, record some of their movements and then correlate the pattern created against some of the patterns in rice DNA. My first thought was WTF? What does the movement of a visitor in front of this wall have to do with rice DNA? But then I read the exhibitors description and realized the significant-sounding mystic-feelgood-psychobabble description had really been tuned to a "t" and by an expert (wish I could make up BS like that!)-- from just reading the one page description on it I could see how the concept of the piece would have won many reviewers over with an "oh wow-- sounds like it has important observations, and is right on topic" response. My guess is the interactive construct was already set up for some other purpose, and all that was done was to re-explain it so that it seemed integrated with the theme of the exhibit, and made "unique observations about the human condition and culture" (or some such, that's about the best I can do with that stuff-- my heart is never in it) in order to improve the chance for acceptance...

I think this sort of thing is not atypical in art settings-- overall concept and presentation are often far more important, certainly than aesthetic or technical prowess. Art is about *selling* something as art, and often without the sideshow pitch and gallery setting it wouldn't be recognized as art at all. SIGGRAPH some time ago chose to follow that example, so there's no real surprise here IMHO...

Art is about aesthetics, which is largely subjective-- and where subjectivity rules, objectivity is left behind...

SIGGRAPH is like other major conferences in that they receive a very large number of papers for review, and can accept only a relatively small number of them. The program committee acts as a gatekeeper for the themes and areas, so it is natural that certain areas will be favored because they fit the themes. The result is that even very good papers do not always appear in the conference. Even if people all acted in the best way (which they don't), the conference would still have to reject good papers. Then the essentially stochastic nature of review can explain why not all papers that appear in the conference are the best ones. I say all this not to defend the practice, nor even to apologize for it, but just to explain what actually happens. It is not a conspiracy, but it is really unfair, and probably inevitable. Resistance is futile. SIGGRAPH is not even the worst in this respect. Ask a neural net researcher about NIPS, for example.

I'm going to jump on the bandwagon and thank Michael for expressing his views. I have to say that I agree with several of his points, and several made by others in this forum.

I have had one SIGGRAPH paper accepted, and another rejected. The rejected paper deserved to be rejected because, while the ideas were good, the paper was in poor shape.

The accepted paper was much more of an interesting study. The paper recieved mediocre scores between 2.5 and 4. The average score was 3.3. Among the reviewers was one who knew our work - his was among the lowest scores. Many of the other scores were due to the reviewers not knowing the background work, and therefore not thinking that the work was new (ie. it was so clear and simple, how could it not have been done before?).

In the end, I believe that the paper was accepted despite the mediocre scores due to the nice images in our paper.

This is an example of the review process going wrong both ways. While I think it's nice to have pretty pictures in graphics papers, they should not be the sole reason for acceptance. I believe that the paper should have been accepted based on it's technical merit. In fact, the paper generated a lot of buzz, produced a lot of interest, and there have been several follow up papers in the short time since the work was presented (which justifies acceptance).

Now for some suggestions (some of which have been mentioned before):

*There should be at least one editor/reviewer who is responsible and can be contacted by the authors.

*Jonathan's idea of a winter SIGGRAPH is a good one... I'd like to take that even further:

*Eurographics has several workshops/symposia for special topics. If SIGGRAPH had such counterparts (eg. The SIGGRAPH workshop on Rendering), this would give a home to many high quality submissions. Moreover, they would provide a high quality delivery venues. I have been very impressed with EGSR, and have learned more about new rendering work attending one EGSR than 5 SIGGRAPHs.

This doesn't mean we eliminate SIGGRAPH. It's still good to get a smattering of different topics (vision, image processing, rendering, animation, etc...) in one venue, but we should provide more venues. Since they would be attached to "the SIGGRAPH name", presumably they would inherently carry weight in the community (which is perhaps sad).


One last point I would like to address: Many people have suggested to start sending submissions elsewhere. This is fine for professors who are already established/have tenure. They want to make a statement, and that's fine. This is a problem for new professors and graduate students. We need to published our best work at the best conferences to survive in academia. It's not so easy for us to just pick an appropriate venue in liu of SIGGRAPH because it can lower our percieved worth. This is why it's very important to help solve the problems of SIGGRAPH, and possibly open up new high quality (where the quality is also percieved to be high) venues.

I personally believe that EGSR is top notch, and that some of the most useful and visible(though not most theoretically novel) work is in JGT. But I am hesitant to submit to these venues because there is more recognition for publishing in SIGGRAPH.

Same Anonymous Coward as below...

I think that a major problem is that most people outside of one's specific field judge the quality of a researcher by the weighted sum of the publication list (higher weights for top-tier conferences). However, I think that it is more important to try to figure out if the researcher has made significant contributions to the field. Once you've got a topic, it's quite easy to incrementally re-spin it a few times over a 1-2 year period, and maximize the publications; however, doing this prevents you from working on something more novel and interesting in the meantime. It stifles innovation on a personal level.

For example, suppose that I discover some new application domain where an NP-Complete problem of great importance needs to be solved. Something big, along the lines of register allocation in compilers.

Paper #1 - introduce the problem. Model it as an integer linear program or satisfiability problem and solve it using a commercial tool. Minimal implementation effort needed.

Paper #2 - Solve the problem with a backtracking search. Because you know something about your application domain, you can find a nice pruning criteria to speed things up a bit compared to #1.

Paper #3 - backtracking search is exponential in the worst case, even with your pruning criteria. Now let's solve it with an iterative improvement algorithm such as metropolis, simulated annealing, or a genetic algorithm.

Paper #4 - speed is important. Resolve the problem with a polynomial-time heuristic based on a network flow or dynamic programming.

Paper #5 - #4 is still too slow. You've found an online application of the problem (e.g. just-in-time compilation), so now you need to focus on speed rather than solution quality. Greedy heuristic.

Paper #6 - You've analyzed your application domain, and found that 50-70% of all problem instances belong to a special class that can be solved optimally. (say, graph coloring on interval graphs) New paper.

Paper #7 - It's time for an approximation algorithm.

...

This is how you maximize your publication list. Beat the same problem to death. Of course, we know that you can do #1-7 for ANY NP-Complete problem., so at some point, you're not really adding much to the community (although #6 could be valuable, especially to industry).

Then, find some variation of the original problem with a different objective function, and repeat #1-7. For your typical embedded/CAD example, maybe the first iteration was optimizing performance, given some area and energy constraints. The second iteration, you optimize for area, given performance and energy constraints. The third iteration, you optimize for energy...

THIS, my friends, is how you get tenure. Taking the John Nash approach from A Beautiful Mind--find a really really really important problem and solve it--is a real crapshoot. Either you prove or disprove P=NP and get uber-famous; or you don't get tenure. Typically, this is a matter of being in the right place at the right time, and being smart.

Edited by author 08-29-2006 05:24 AM
I believe that there is a similar problem in the architecture community with ISCA. If you don't believe me, check out the ISCA Hall of Fame:

http://www.cs.wisc.edu/~arch/www/iscabibhall.html

The same people repeatedly publish there every year, and its hard for newcomers to break through.

The difference, however, is that there are some alternatives in the architecture community, although they generally aren't considered to be as important as ISCA. For example there are MICRO, HPCA, and to some extent, ASPLOS if you have the right application.

Secondly, any architecture researcher who is truly in a bind can switch to embedded applications quite easily. That opens up the possibility of publishing at DAC or ICCAD, which are generally larger (similar rejection rates, but significantly more overall sessions--in other words, if your paper is of high quality, they are likely to find a place for you).

But, back to ISCA, one of the big problems there is that the papers are not reviewed blindly when the paper is discussed at the table.

---------------------------------------------------------- ---------------

As far as getting your ideas stolen from someone on the committee, that is a definite risk. The best advice that I can give anyone is to assume the worst. If you believe in your paper, and it gets rejected, publish it online as a technical report ASAP. You can usually republish technical reports in conferences and journals anyway. This way, if someone tries to steal your idea, you can wave your technical report in the air and cry bloody murder. It is highly unlikely that the "stealer" will be able to get a submission out and published before your post your technical report.

----------------------------------------------------------- ---------------

I've heard of a number of situations where two papers proposing similar techniques have been submitted, and the one who was authored by a program committee member gets in. I have personally been involved in situations where I have developed the same result (simultaneously and unknowingly) as someone else. In this case, I think that either both papers have to be accepted (or rejected).

---------------------------------------------------------- ----------------

The community needs to find a way to promote collaboration rather than competition. The Berkeley group and the MIT group that have similar research topics should be encouraged to work together, not against one another.

www.tipcell.com sell: kyocera 2135 housing, kyocera 2235 housing, kyocera 3225 housing, kyocera 3245 housing, kyocera 3250 housing, kyocera kx17 housing, kyocera kx9 housing, kyocera kx13 housing, kyocera kx440 housing, kyocera kx494 housing, kyocera K433L housing, kyocera K434N housing, kyocera KE433 housing, kyocera KE433C housing, kyocera KX 434 housing, kyocera K7 housing, kyocera K9 housing, kyocera K433 housing, kyocera K434 housing, kyocera kx1 housing, kyocera kx1i housing, kyocera 1155 housing, kyocera se47 housing, kyocera 1135 housing, kyocera 2119 housing, kyocera 2235 housing, kyocera 2345 housing, kyocera 3245 housing, kyocera 3250 housing, kyocera 5135 housing, kyocera 6035 housing, kyocera 7135 housing, kyocera k10 housing, kyocera k112 housing, kyocera k404 housing, kyocera k433 housing, kyocera k434 housing, kyocera k454 housing, kyocera k484 housing, kyocera k490 housing, kyocera k493 housing, kyocera k494 housing, kyocera ke413 housing, kyocera ke414 housing, kyocera ke424 housing, kyocera ke424c housing, kyocera ke433 housing, kyocera kx1 housing, kyocera kx16 housing, kyocera kx2 housing, kyocera kx424 housing, kyocera kx433 housing, kyocera kx434 housing, kyocera kx440 housing, kyocera kx444 housing, kyocera kx5 housing, kyocera QCP-2035 housing, kyocera QCP-3035 housing, kyocera QCP-6035 housing, kyocera S14 housing, kyocera SE44 housing, kyocera SE47 housing
mail: sales@tipcell.com
name: keiresing
MSN: keiresing9988@hotmail.com
http://www.tipcell.com
ICQ: 239470786

Edited by author 08-19-2006 01:44 AM
My somewhat off-topic response is to message 61 (industry person 2). Upstill's 1985 thesis was on tone mapping and was ten years ahead of its time not only in recognizing HDR as an issue, but in taking a very sophisticated approach to dealing with it. Unfortunately it is only available from University Microfilms. And I would not bet against something else I haven't heard of being first! Some might argue it was Stockham. For a more well-known work that I would argue defined the field for graphics was Tumblin and Rushmeier's work in the early 90s. Clearly you had to be doing HDR in order to do tone mapping!

Obligatory SIGGRAPH comments: there are good reasons other fields have conferences with abstracts, and journals with articles. That Genie can probably not be stuffed back in the bottle very easily...

Just an aside, the story several posts below is not the first time that the
SIGGRAPH academic side has failed to mention prior work in the industry.
Academic people talk as if they invented the HDR idea, but actually it was
introduced by Glenn Kennel in 1993, as part of the Cineon file format (and
Cineon compositing system). Both were in wide use in the mid-90s, several
years before SIGGRAPH papers had heard of the idea. The Cineon compositing
system died out several years later (though other systems have picked up the
idea), but the file format lives on as "DPX", which is a superset of the
original format, (and in practice is usually the same).
No doubt, the academic side has taken this idea much, much farther.
But it's never mentioned where it came from.

Cineon/DPX is 10 bits log per channel, with an adjustable "white point"
usually at 685 on the 0..1023 scale, so values above 685
are the HDR values. Since this is a log scale, that is an adequate
range, since for typical purposes the >white values will be clipped
after processing anyway.



From wikipedia,
   "Pixel values above 685 are "brigher than white", such as the sun, chrome highlights, etc"

Now let's count the days until someone edits this wikipedia page.
But watch out, wikipedia has an revision history.

I think the idea of having online reviews/discussion will not help much
(though it will not hurt either).

The problem is that your paper is entirely reviewed by your direct
competitors, who have a shared interest in having your work rejected,
and there is no process to hold them accountable to producing a fair review.
Simply having the discussions online does nothing, unless they were
to be made public. In rare cases a chair might decide to take
action on a review, but the chair is often a political type
who is more interested in avoiding conflict, so this is unlikely.
And they are probably way to busy to do anything even if they
noticed and were inclined.

I do not have any great solution, but what about this as a good first step:

For the primary reviewers, have an explict policy for the two reviewers:
1) is an expert in the subfield, i.e., your direct competitor
2) is outside this subfield, is a generalist who is not competing with
   this particular work, and maybe is a more established/tenured sort
   of person who has nothing to loose in the outcome of this paper.

Then reviewer 1, who is the competitor, will have to justify their
comments to #2.

This avoids the current problem of the easy dismissal of a paper
by two reviewers who are both in competition with you.

More generally, most of the suggestions offered below will
have no major effect as long as the two primary reviewers have
the power that they currently do, so the first thing to change
is the balance of power with these two reviewers.

I just read through the whole archive and there were a number of great ideas. Let me add my own bias to them :-)

Yes, make the review process an online forum where reviewers have to stand by and defend their reviews. This will take out some of the biases (if there are any) and will converge to a more objective review, which is to everyone's benefit. I would add to this: have an initial period where nobody can see each others reviews. This will avoid initial biases towards other already written reviews and produce a more honest one. This way late, hesitant, or cautious reviewers can not just copy (opinions) from existing reviews. Ideally, all reviewers will give an honest opinion in this time, and after this private period the shroud is lifted and everyone (in the paper's reviewer set) can see the collection of reviews with names attached. Then comes the online review refinement period, where online discussions emerge, resulting in a final opinion which is then used by the senior committee and papers chair to select the ultimate collection of invited siggraph papers. This opinion could even be published as well (online, not in the procedings), so the perhaps less-expert reader is more informed about what expert reviewers felt the strong and the weak apects of the paper were.

A second idea that I found very good was to select reviewers in (computer) randomized fashion from a pool of suitable ones (where only the computer knows the authors of the anonymized paper). This would remove the bias in the reviewer set considerably. I am not saying that there was one before, but this way authors could feel more secure on that issue.

Computer graphics is no different than anything else. Have you ever
heard of a field of human activity where politics does not enter?
Of course not. So of course the reviewing process is somewhat "broken".
Anyone who says otherwise merely flags themself as a politician.

You may have heard about the incident this year where one of the SIGGRAPH
papers was largely based on material in the courses several years ago
introduced by a well-known industry person (a rare case where industry is
leading the academic work). The paper this year did not mention or cite the
earlier work. Probably this was intentional, since SIGGRAP paper candidates
are rejected for much less than this. The upsetting thing is that the papers
committee was alerted to the problem, but did nothing. Whether these full
claims are true or not, it certainly that the industry work should have been
cited and fairly discussed.

So while there are ethics guidelines, they are for show, and are not meant
to trouble anyone to actually enforce them.

But let's not make so much of this. The old boys of SIGGRAPH are going
to retire in another 10 years, and the field is changing rapidly.
SIGGRAPH's policy of accepting papers from other fields only makes the conference less
relevant ultimately. We're all more aware now of the source conferences in other fields, and are more likely to turn to these to
see the new results published first (and only later in a polished
demo at SIGGRAPH). Simultaneously, the actual graphics work will increasingly
be done in the games community, and forums are evolving to showcase this work.

I also thank Michael Ashikhmin for this discussion opportunity.

One thread in this discussion is the issue of if
original ideas occur at SIGGRAPH, or even in graphics.

Let's look at two examples-

The original Effros texture paper provided the basis for 100s of graphics
papers, with very many in SIGGRAPH. Some of these, like one by Michael
Ashikhmin, extended the basic approach to be practical and efficient. These
have been followed by dozens of others where it is applied to this and that,
mostly obvious and relatively unimportant problems, and it is these other
papers that appear mostly in SIGGRAPH. The original paper did not appear
in SIGGRAPH, nor in the graphics literature.
  Second, the paper that introduced the bilateral filter. It has also
inspired dozens of efforts in graphics, and it too, did not appear in
SIGGRAPH, or even in the graphics literature. I will bet you that if
it had been submitted to SIGGRAPH, it would have been rejected!

Certainly there are a few examples where graphics has devised things that
have been recognized in other fields, as Rushmeier mentions. But these are so
few that we can probably count them on one hand.

When I was in school one of our exercises required applying the wavelet code
from the new Numerical Recipes 92 book to sparsify a linear system. I was
already interested in graphics then, and was quite interested in one of the
papers in SIGGRAPH around that time. It was quite appealing that in graphics,
one could do what would be a homework exercise in engineering, but get pretty
pictures and a publication too. This was the good side of graphics.

The bad side is what it has become- the shared brainwash
that this is such important field that only a select crowd of
old guys can be on the PC, when in reality it is turning
into a sad scene of increasingly large teams making increasingly
absurd video demos. We must thank this discussion board for holding
a bit of a mirror up to this arrogance.

Looking at how the Chinese government modernizes the country, you will see that most operations are incremental and progressive; this is to ensure the greatest chance of success without derail the entire process.

Similar theory might apply to SIGGRAPH. It is unlikely to dethrone it as THE conference for graphics or even to have a winter session in the near future. Instead I recommend the following incremental and progressive improvements of the review process that could easily be implemented for NEXT SIGGRAPH.

The majority of the process has been done for EGSR and I3D already (as posted by Jonathan Cohen earlier), but I reiterate them here for completeness.
1. Reviewers know each other’s identities, even though the paper authors remain anonymous to tertiary reviewers.
2. There is a discussion board for reviewer to exchange opinions. They could also see each other’s reviews.
3. The rebuttal period should be extended longer, probably at least two weeks. The paper authors use the same discussion board, and can communicate with all reviewers.
4. Prior to the committee meeting (unlike Jonathan Cohen, I did not propose removing that), the senior reviewers inform all other reviewers of the same paper his/her decisions for acceptance or rejection, including his reasoning.
5. If a paper is unanimously accepted/rejected (i.e. all 5 reviewers agree with each other), the paper’s fate is sealed at this stage. There is no need to discuss it during committee meeting.
6. Committee meeting only discusses papers that remain controversial (i.e. 3-to-2 or 4-to-1 voting).

Justifications for each point above:
1. Peer pressure discourages irresponsible reviews, e.g. reviewers who are too busy to read the paper and put out superficial comments or people who get personal/emotional and simply rant.
2. The discussion board facilitates communication and removes potential misunderstanding of the paper.
3. There is no reason why the authors could only communicate to the senior reviewers like the current system. Also, one week rebuttal is too short. I have seen one case in MSRA where a senior reviewer asked for a comparison with a previous work and the first author of that paper has to work straight for 2 days without sleeping just to implement that previous work. Allowing longer periods of rebuttal should help reduce this stress.
4. Obviously this step reduces the chance of committee member playing politics, making the decision process more transparent.
5 and 6. Again this reduces chances of playing politics, reduces workload during committee meeting, and allows more discussion spent on controversial papers.

Since the majority of these are already in EGSR system (I don’t know I3D since I never reviewed for it), so I don’t think it will be too hard for SIGGRAPH.

This is in response to what "Industry Person" said below. I must start with saying that I feel this year's paper selection was FAR better than the last couple of years. It all depends on how you define a real "_graphics_" paper. Isnt image processing, computer vision etc. a part of graphics? And who said that it has to be interactive to be classified as graphics? I believe that the "Image Capture" session this year was perhaps the best siggraph session I have attended in the last three years.

I think that saying image processing, computer vision, human perception, haptics etc. shouldnt be a part of SIGGRAPH is wrong. I am sorry if you didnt find what you were looking for in this year's SIGGRAPH, but I certainly did, and I hope we continue to see such brilliant papers in the coming years.

This is not to say that the system is perfect. In fact, far from it. But criticizing papers based on personal preferences on what we would like to see at SIGGRAPH is a bigger evil.

I am very glad to see all of the postings on this topic. I find that the timing of this discussion is absolutely wonderful - as Siggraph 2006 seems to be the apogee of the problem. I attended the conference, hoping for the usual burst of inspiration. Unfortunately, I was sorely disappointed. And even though what I say here is just my opinion, everyone who I have talked with after the conference seemed to think that unfortunately this year's Siggraph was paltry in comparison in terms of papers (and even sketches, to some degree).

Is this the graphics conference that it claimed to be? It's hard to imagine. Now there are papers on sound, perception, human interfaces, haptics (which were present for many years), image processing. Look at the percentage of actual graphics papers that are currently published at Siggraph? This year it seems to be an image processing conference. And how many papers make a claim to be "real-time" and applicable to interactive applications while in reality using "practical" amounts of memory such as 4GB of textures or a 600MB database of small data to lerp from for real-time performance?! It's absolutely ridiculous. To a degree,I have to tip my hat to Jonathan Cohen - because in the last two years I3D has really become the conference to watch for releveant _graphics_ papers. I find that if one is looking for good, applicable, implementable rendering techniques, then one shouldn't look at Siggraph papers. Instead, we turn to I3D or Eurographics (which has its own set of problems).

Siggraph has been a great source of inspiration for many years. But having been through the process myself and having witnessed people submitting their work it seems that the process is indeed very broken. I personally like the suggestion of building up other conferences meanwhile. And the suggestion of refusing paper submissions from committee members seems reasonable - specifically as a method to build up other conferences.

After very hard work, I got a paper in this year. The natural thing was to work VERY hard on the talk because so many important people would see it. AFTER my talk (which went well due to A LOT OF WORK), I found out that almost nobody saw it from the audience I had aimed for. Ironically, few relevant people saw it because SIGGRAPH is so big (other concurrent sessions). So at some point, a bigger conference means a smaller relevant audience. :(

I was also at the Town Hall meeting, and thought the idea of making SIGGRAPH more like CVPR made a lot of sense. Accept more papers, but only orally present the best papers (present the rest as posters), hopefully this way papers with less polished results but with interesting ideas can get in more often. If these papers lead to polished work, they could then be published in TOG or other journals. Perhaps the orally presented work at SIGGRAPH could also get automatic invitations to be published in TOG, let TOG be the end of a research project instead of SIGGRAPH. The longer review process of TOG should be able to reduce much of the noise generated at the SIGGRAPH review process.

I attended the Town Hall meeting in SIGGRAPH. Most discussion there were about reviewing processing. However, all technical measures to improve the review process will not solve the problem. The fundermental problem we are facing is that SIGGRAPH is THE ONLY dominate conference in computer graphics. That is the most dangerous thing.

Is it viable for Siggraph to publish the reviews, as they do in other venues? Maybe not in print, but certainly online. This may induce some extra responsibility when submitting reviews and, in the long run, it should improve the review process. Any thoughts on this?

3. Third suggestion is to make the author names blind even to primary reviewers. It is almost a mockery fo the "blind" reviewing process if the most important person who makes the final decision about the paper knows the author names. This can possibly be done by pseudo randomization of the paper numbering, mapping of which is known only to the Papers Chair (or only to Price Water House Cooper ;) )

It wouldn't be too hard to increase the anonymity of the process with a little cleverness and a couple simple black boxes. For instance, suppose the authors' names were unknown to the primary reviewers. They would choose a reviewer and give their name to a black box, which would complain only if the name matched one of the authors' names. Otherwise the black box would tell the primary that the proposed reviewer name is acceptable. Of course, they might get (un)lucky and pick an author as a reviewer.

A more robust scheme would be: a primary reviewer selects a large number (~5-10) of potential reviewers, and the black box chooses a subset of reviewers which are not authors. There's some small chance that all of the proposed reviewers are authors, but on average this procedure would improve anonymity.

These black boxes could be implemented with a simple perl script or even a human moderator.

There have been very good suggestions in this discussion board.

1. Have online discussion system for even tertiary reviewers to participate. Primary reviewers should take this discussion seriously and respect the consensus among the reviewers that is evolving in the discussion board.

2. There were suggestions to avoid submissions from the program committee, and understandable reluctance (difficult to staff) to do that. There was also a suggestion to have two SIGGRAPH conferences per year. These are all good suggestions and reasonable concerns. I heard that Eurographics is going to shift its submission date to October 1, and move the conference early (During this transition, there are going to be two Eurographics deadlines in 2007). If we are reluctant to have two Siggraph conferences and to prevent the PC to submit papers, we can make use of this opportunity to encourage Eurographics to become a quality alternate venue, so that PC memebers need not wait for a year to submit their work, but just submit it to Eurographics. If possible, we can encourage EG to time their conference appropriately to have close to 6 months spacing from Siggraph.

3. Third suggestion is to make the author names blind even to primary reviewers. It is almost a mockery fo the "blind" reviewing process if the most important person who makes the final decision about the paper knows the author names. This can possibly be done by pseudo randomization of the paper numbering, mapping of which is known only to the Papers Chair (or only to Price Water House Cooper ;) )

4. Have no limit on acceptance rate. If a paper is good, it should be accepted. Although it is made to believe that there has been no fixed "acceptance rate" enforced during PC meeting, it is hard to believe that there can be so many rejects especially when our community knows the standard of SIGGRAPH and makes all effort to submit only highly polished papers. My guess would be around 30% of the submitted papers would be of very good quality.

If these suggestions are implemented, I strongly believe that the hard work put in by the PC, tertiary reviewers, and not to mention the authors will be well-rewarded.

Having said that, I would say around 50% of papers in this siggraph were of very high quality, and the rest were mostly pure hacks and problem specific fine-tuned implementation. Unfortunately, this is the randomness in the selection process that has creeped in due to low acceptance rates, and also our fatal attraction to flashy images and videos.

Looking forward to Siggraph 2007! Good luck Marc! You are in an unenviable position that invites too much scrutiny.

I am sorry to have missed the BOF on this topic - I heard about it too late and had other committments.

I should note that, as someone who works in the visual effects industry, I have no career-oriented need to publish anything I do. However, I do want to give papers/sketches at siggraph because two-way sharing of technical information improves what I do, and because it is intellectually satisfying when my research grows with the help of others.

The early note in this forum about the early history of siggraph was very helpful. For a fledgling field of study, it makes sense to build up a forum with a rigorous process that can build respect from the larger academic community. That approach has been wildly successful. The graphics community is now a large academic field, but also drives several large commercial industries (e.g. visual effects and graphics cards).

And now the community is VERY different from the audience it began with. On the technical side, my experience is that visual effects companies are conducting basic graphics research that is at least as good as the academic community. And although that research is closely held for a period of time, it eventually can be published and/or openly discussed. Siggraph is the natural forum, but we have all been reading about the barrier to entry. And since publishing has a more altruistic drive than in academia, going through this broken review process is not worth the effort.

While siggraph is very aware and sensitive to the non-academic community members when it comes to accepting registration money, we are very underrepresented in the review and organizational process. I have reviewed 2-3 papers every year for the past 5 years, and have asked to be on the review committee so that I could better understand the baffling results I have witnessed, but there does not seem to be any interest from papers chairs. This lack of representation substantially limits siggraph's actual value and potential.

I beileive the goal siggraph should adopt is to provide a forum for presenting as many ideas and concepts as possible. This not only better serves the rapided improvement of graphics research, it also is more in line with the needs of the larger (non-academic) graphics community that supports it. I am not sure why, but the focus in this forum has been on the details of how to implement a better review process. No one has addressed the idea suggested twice by Michael to operate the conference more like scientific conferences: completely eliminate the review process. I used to attend quite a few engineering and scientific conferences, some smaller and some much larger than siggraph, in which the process required you to send in an abstract, and that assured that you would recieve 15-20 minutes to present your ideas. Yes, some turkeys do get in, but the community is capable of deciding for itself - a review committee is unneeded for this. And I dont think the current system in entirely turkey-free anyway. So longer term, I think siggraph and graphics in general will be more healthy eliminating the review process entirely. Michael was correct in his original note, the physics community uses the arXive for distributing papers, leaving reviews for publications that take place well after the community sees the work and comments on it. That is a much more sane and efficient process.

Conversations I had after the BOF confirm what I was afraid of: there is strong resistance to changing the process because tenure positons are on the line. That rationale is illogical from my point of view. For one thing, that means that all of the graphics community must be suppressed by the needs of the few people who have tenure issues to attend to. The tenure process functions just fine in other academic communities that dont have a peer-reviewed conference. For them, peer-review happens in journals. Conferences are an opportunity to learn about the ideas of others and share your ideas. And that is an important contributor to the health of those communities. It would be a shame if, in a misguided interest in a flawed form of tenure seeking, graphics is held back. And it is being held back by the current process.

Certainly we cannot switch to a non-reviewed system overnight, but it should be entirely workable to decide to do so in 2010, for instance. That would give the academic community time to switch over to a journal based (or some other mechanism) approach for their needs.

I certainly look forward to seeing what comes from this discussion and any changes that might be made to the system. I am always optomistic that things will get better. I am writing this on the last day of siggraph, and I was quite discouraged after seeing a lot of talks with severe flaws in their reasoning and/or technical implementation, because we are told that they are the best the graphics community has to offer. I know that is not true because I have seen much better, so I am optomistic that changes will make siggraph better.

In response to the claim that acceptance depends largely on visual appearance and less on scientific content, I propose a control experiments: In parallel to the committee, form a committee of visual arts people (from academia and graduate students), tell them how many papers to accept from each category and watch which papers they “accept”. These people should have internet access like the reviewers, but be people who don’t know computer graphics.

Another idea: do the review process in two phases. In the first phase, reviewers get the paper with all figures removed (without videos), and review based the text only (it might be needed to ask authors to submit text-only version). Papers that survive this phase go through a second phase which considers the graphical content. The idea is that a text-only version requires a more scientific job both from authors and reviewers.

I believe the review problem in our field is predictable.

In a mature field,

  for any topic, there are several groups working on it, and

  every new development is necessarily "incremental", because
  the big new ideas were discovered long ago.

This means that:

  your paper will be sent to your competitors,

  who can trivially reject it because it is indeed incremental.

This also means that papers that are well constructed but
do not mean much to anyone (as a previous post indicated)
will have more success. And the opposite, Papers that
have a bit of new idea will have a particularly difficult time.

Despite all this, papers do get accepted. Unless there
is this body of great unpublished papers out there?

I knew this forum from siggraph yesterday. I just read most of existing posts here, and want to add two points (could be wrong, just for discussions).

(1) Some posts mentioned "siggraph put too much weight on superficial video/images". I agree! It is true for recent siggraph, but not true for siggraph long time ago. There are various possible reasons. The first possible reason is that traditional graphics runs out of topics and need to bring new things from other fields. Vision certainly is one option, But i feel it is going too far now. You have to twist video to get siggraph papers in. Siggraph is quickly becoming just another CVPR/ICCV style conference which publishes hundreds of papers but the generality of most of the works are weak (i.e. only work on several intentionally selected video/images/models).

The second possible reason is that siggraph paper committees who have super power on paper selections are over visionized. Just check the composition of recent siggraph paper committees, and you will find the percentage of vision experts in siggraph paper committees. So, it is not a surprise that more and more boring vision/image papers got in.

(2) Online discussion among reviewers are certainly better than in-person PC meeting, which has been successfully used for many other graphics conferences (except siggraph) for years. I am very surprised to see the siggraph is just indifferent to it. As pointed out by other posts, the similar thing happens to TOG that does not show the name of assigned editor to authors, while TVCG certainly is taking the lead. I wish it is not because some powered people want to keep their hidden (thus safe) super power on paper selection of top confs/journals. If so, it is really bad to the whole community.

Edited by author 08-02-2006 01:14 AM
I have some suggestions to address some of the current problems with the SIGGRAPH review process.

1) Eliminate the physical meeting of the SIGGRAPH papers committee.
2) Replace the committee meeting with an online discussion among all the reviewers of the paper (not just the committee members).
3) The job of the senior reviewer on a paper is try to generate consensus. That may ultimately require more reviews. Because there is not the intense time pressure of an in-person committee meeting, these reviewers should be best experts on the subject matter, not necessarily other committee members.
4) A job of the papers chair is ultimately to make sure that each decision is made in a reasonable fashion, as indicated by the reviews and subsequent discussion. It's a big responsibility, but it is essential. At least she should have sound information to either approve of the senior reviewer's recommendation or the necessary ammunition in writing to overrule.

This process has a number of desireable properties.

A) All discussion is in writing and subject to the scutiny of a number of people.
B) All reviewers can see and participate in the whole review process.
C) We are now free to turn the often-present criticality on each other as reviewers and not just as paper authors. So if I see that reviewer #2 wrote a really sketchy review, I can say so, and suggest either that he go into more depth or perhaps that we need another review.

This process has worked well for at least the past 2 meetings of the ACM Symposium on Interactive 3D Graphics and Games (http://www.i3dsymposium.org -- sorry for the shameless plug). And what's more, saves travel time and money for the committee. It may seem a large departure from the historical SIGGRAPH review process, but I don't think the change is prohibitively large.

This will not fix all problems of how papers in our field are reviewed and what the standards should be, but at least it puts it all more in the open. Also note that property C has the flaw that if I am Joe Schmo assistant professor and reviewer #2 is graphics pioneer god-incarnate, I may be reluctant to speak up.

Well, that's all I've got tonight.

Heard about the forum on the floor today. Levoy's session might help but I expect it'll mostly be people who've succeeded in the system protesting that minor changs might be good but the system is sound.

The posts so far have ignored the "Glam" factor in siggraph, a thing I think broken. Papers are accepted/rejected on their images or videos. I heard a senior member of the papers committee say one year "the video is good so we have to figure out a way to get the paper in." The paper, needless to say, has never been reproduced becaus the intense amount of handholding and manual labour required to get the technique to work. I've also seen papers that represented outstanding engineering efforts rejected because while the authors did a reasonable job with their video, they didn't spend a couple of weeks tweaking Renderman to get the best possible results out. "Find a reason to reject" prevailing in these cases. I don't expect that will change about siggraph but they ought to be upfront about it. Requiring more disclosure of how the results are generated would be good. Special evaluation of the video could be included in the reviews. And its daft to penalize those authors who hold their papers so that they can polish their videos as some hints in this forum have suggested.

John Hart's comments seem off the mark, too. Graphics is not healthy, siggraph is not healthy, and supporting elitism and not repairing siggraph means that in the long run you'll only be tenuring 18% percent of your academic-minded graduate students. We already see that in the last few years.

As someone who has had one siggraph paper accepted and several rejected, I've read this forum with great interest. I have a couple of things to offer to what's already been said.

The biggest problem with siggraph is that given the amount of material submitted, the time alloted for the review process is disproportionate to the reputation of the conference, and encourages careless reviewing. A reviewer often reviews more than one paper, and there's simply not time to try to test any of the ideas to see if they work. There are several notable examples of siggraph papers that don't work, as anyone who's ever assigned a new graduate student the task of implementing a siggraph paper can attest. These "flukes" are more common than the siggraph community likes to admit. The harsh truth is that most siggraph authors probably don't want their results reproduced.

As an aside, if one thinks that testing any of the methods or algorithms in the paper is impractical, then they should be advocating for the removal of the review criterion "is this work implementable by a competent graduate student?". Why does anyone care? Most graphics is engineering, some is science. A good engineering job may not be implementable by a competent graduate student. A good engineering job may require a team.

Prof. Levoy remarks that he intends to bring the tertiary reviewers in on the rebuttal process. I think that's a good suggestion. Another good suggestion is that if the committee asks for additional reviews, as often seems to be the case among the program committee, then those additional reviews need to write a review that the author can see. The primary/secondary paragraph summarizing the findings of the committee is insufficient to give exposure to the process that ultimately accepted or (more commonly) rejected a paper. Also, Prof. Levoy remarks that he will implement new measures regarding papers with high variances in their scores. High variances are not always bad, and in fact often indicate that there is disagreement in the community on some issue. Those papers ought to be published, and squashing dissenting opinions doesn't do any community any good. Prof. Rushmeier remarks that outsiders think that siggraph reviews are often more thorough than in other areas of computer science. That may be true for conferences but is not true for many journals in other areas of computer science. Such reviews are often more insightful than any siggraph review I've received, simply because (I think) that the time pressure is not as constraining.

You are making this harder than it needs to be. There is one simple secret to
getting in to the papers program, and that is to pick a topic that no one has
interested in, and then make a really polished paper and video about it.

Here is a practical step-by-step formula for easy paper acceptance:

1. Scan CVPR, ECCV/ICCV, or NIPS for the algorithms they use. Pick one that
the SIGGRAPH audience has not heard of yet. This is not too difficult, CVPR
is huge.

2. (This is the creative step)
INVENT A "CUTE PROBLEM" THAT IS HAS NO PRACTICAL VALUE. It does not even need
to be a problem in graphics. But if you choose an area in graphics that has
possible practical value, the reviewers will kill it in a second. We're smart
and can easily detect any true relevance, and we want to save those problems
for ourself.

Example: do texture synthesis on some sort of spline or manifold that is
little used. Apply someone's vision segmentation code to make a system that
would allow even your grandmother to make a video of a virtual fishtank from
clips downloaded from the web. Build a clever device for capturing the BRDF
of some phenomenon that is of little interest.

I'm not going to give you any better examples, because I have to admit that
this step is not trivial. But by definition it is not as difficult as getting
a relevant paper accepted to SIGGRAPH, since that is impossible.

3. Download the Matlab code from the identified CVPR or NIPS papers.

4. Add a scale parameter. Now you have an innovative new algorithm.

5. Apply it to the bunny and the dragon.

6. Put ATTITUDE into the video narration. The tone of voice in the video
should convey how both clever you were to download that Matlab code, and how
effortless it was. Make it seem so simple that anyone can do it, yet so
clever that only a SIGGRAPH author could have thought of it.

I have had papers rejected, and one accepted. Trust me, it does work.

I urge Michael to leave this message board up (and also admire his courage for doing
this). Or transition the board to someone else if he receives too much pressure.

This is the first time we have had an open, public discussion about the review process in
our field. Note that, it is an important statement.

These are things that have been said time and again at dinner meetings around the world
every March or April, but with the common understanding that none of it can be said in public.
Let it be said in public. SIGGRAPH will only improve from some self criticism.

And the discussion is clearly having some impact, with the response from Levoy and others.
Unfortunately, I have to make note of Levoy's response. He says he thinks that the review
system "is not broken".

Although it may be no more broken than some other conferences, this is a disappointing response
from him. The impact of its existing problems can be larger, and they impact us.

So despite whatever the intent is, Levoy's statement just makes me believe that the actual effect
of his BOF will be to pull the discussion away from this open public forum to a quiet room
where it will not be recorded, and were people cannot speak freely, and where the discussion
can be generally ignored.

So again, this message board has a major value for the community.

Being someone totally outside this field (from computer networks), it was very interesting to read Michael's writeup on the state of SIGGRAPH. I couldn't help but wonder how replacing the word SIGGRAPH with SIGCOMM in Michael's writeup would give an (almost) accurate description of the state of affairs in networking community - one of old boys exclusive club. Perhaps the same applies to other SIG*s or their equivalents in other CS areas. Perhaps this is now a wide enough problem in Computer Science community that any solutions you SIGGRAPH folks come up with would greatly benefit the entire CS community.

It's encouraging to know that there are people like Michael who are willing to say something about problems like this. I'm an undergrad CS student who's dream is to someday become a big graphics researcher. I've been writing graphics code since I was 14 and I wrote my first GI ray tracer just two years later (man, those noisy caustics look so cool when you're 16!). Ever since then, I have always wanted to make a serious contribution to the field. After reading what Michael has to say about the current state of CG research, the future looks bleak for my dreams of becomming more than an amateur researcher. I can only hope that in the years to come this can be changed.

Michael: Rock, rock on.

Unlike Aaron I "grew up" outside of graphics, and so mainly read a lot of papers from other fields (mainly engineering.) A lot of my work has been with people from outside of computer graphics. Papers and reviews in other fields are all over the map in quality. I find people from other fields have a lot of respect and enthusiasm for the algorithms we develop, and the new ways to visualize data and solve problems with images. I see graphics papers referenced in journals as diverse as the "Journal of the Optical Society", the "Engineering Analysis with Boundary Elements", the "International Journal of Heavy Vehicle Systems" and the "International Journal of Diabetes and Metabolism" (just a quick check of some of my own papers on Scopus -- other people have been much more influential.) Early papers in radiosity used to always reference texts like Siegel and Howell, now Siegel and Howell reference us back. A lot of excellent contributions have been made by computer graphics researchers, assisted by the reviewers that help get their manuscripts prepared for publication. I think it is great to discuss how we can do things better, but I think we should remember that we have a lot of good work to build on.

Edited by author 07-27-2006 11:06 PM
A few comments and a proposal:

1. The reviewing situation is bad in other fields --- the number of submissions has been growing exponentially, while the number of "good reviewers" (i.e., the number of people you feel comfortable assigning reviews/editorial positions to) grows linearly. Although it seems to have stabilized. The reviews I've seen in vision and learning conferences have been very mixed in quality, but the average vision conference review is often much worse than the average SIGGRAPH review. On the other hand, the average vision *journal* review is of *much* higher quality than the average SIGGRAPH review.

2. It is a very human process (as, I think, Marc said), and most people I've seen really try to do their best. I think the right decision is being made in the majority of cases.

3. There's a big cultural problem. I "grew up" as a graphics researcher, and learned that all that matters in a paper is how good it looks; I didn't learn much about science. Reviews are often about feelings; "I do/don't like this paper," is a common sentiment. As I've been exposed to other fields, SIGGRAPH sometimes seems more like an art gallery with equations than a scientific conference. As a student, I never submitted a journal paper, and nobody ever explained to me the importance of journals in other fields. I never read really scientific papers (except as my interests grew to other areas), and never considered myself a scientist. I suspect that a lot of other people in graphics are in a similar situation.


4. Here's my proposal for how to fix the system:

* eliminate the main SIGGRAPH papers program

* add a collection of collocated conferences on topics that roughly span the field: SCA, NPAR, Hardware, Games, SGP, rendering, vision/graphics, etc. Combine logistics, so that attendees pay one fee to attend all conferences, but keep reviewing separate. Create a process that makes it relatively easy to add conferences as necessary.

* optionally, have another set of collocated conferences in the Winter.

There are many advantages: the quasi-journal SIGGRAPH track is eliminated, at which point the conferences will become more selective, but - we hope - not SIGGRAPH selective. There will be some incentive to write journal papers. The committee meetings become smaller and more focused. There is less concern about one community being less fair than another (they may remain different from each other, but I don't think that's as bad as having different SIGGRAPH areas seen as having different standards).

The disadvantage, obviously, is that we lose the showcase papers track, and that there's more possibility for good - but uncategorizable - work to get lost between the cracks. As long as everything is flexibile and co-located, I'm not too concerned about these things.

Just a note on how things get decided in SIGGRAPH:

Key conference positions (chair, papers chair, courses chair...) are approved and policy decisions are made by members of the SIGGRAPH Executive Committee. Members of the Executive Committee are elected by popular vote by the SIGGRAPH membership. One way to influence what happens in SIGGRAPH is to volunteer or nominate someone else for an Executive Committee post :
http://www.siggraph.org/newsfeed/call-for-nominees/
(yes the date posted there is passed, but it is not too late to contact Alain Chesnais and express and interest.) If you or your nominee is not selected by the nominating committee, it is possible to petition to put someone on the ballot
http://www.siggraph.org/gen-info/bylaws.html#article_10
This has been a successful approach in the recent past. Note that you have to join SIGGRAPH to vote.Finally, when it is time to vote, please consider who you think will make the best decisions about how the conference is conducted.

The EC does not dictate to the papers chair how to run the review process, but clearly approving who the papers chair is and determining when and where SIGGRAPH is held and what other events SIGGRAPH sponsors are important influences in determining the opportunities for all of us to publish and present our work.

 Dear all,
 First of all, message #28 in this thread has a note that it was "Deleted by topic administrator 07-26-2006 11:12 AM". I would like to assure you that it was not me. I have NOT deleted any messages and have no idea who deleted it and why this happened.

 Second, I certainly wish Prof. Levoy best of luck with organizing next paper revision process as well as a session at siggraph-2006 (information about which is, by the way, is currently not present on the web page where BOFs are listed, i.e. http://www.siggraph.org/s2006/main.php?f=conference&p=birds ). I do have some doubts that people, especially younger ones, will be willing to speak openly when everyone sees who they are. Still, I wish I could attend what might end up being an interesting discussion but have already started at my new job.

 Finally, the document which initiated all this, i.e. http://ash.5000megs.com/leaving.html
has disappeared for some reason along with the whole site. I was NOT planning to remove it and hope it is just some temporary technical difficulties - you get what you pay for and I paid exactly nothing. However, given the upcoming siggraph session, I think it is useful to have it available. Plus, it contains probably the only public link to this discussion forum. I will therefore put this document to my very old Utah web page for now. See http://www.cs.utah.edu/~michael/leaving.html

 I hope you understand that U. of U. CS Department by no means endorses it. Therefore, I would appreciate if those who disagree with my views do not ask Department officials to remove it, directly or indirectly (as some of you apparently did a year ago concerning my Stony Brook "siggraph broken" note).

M.A.

 P.S. Dear Alex and others. Thanks for nice words, but inability to shut up for one's own good is hardly courage, it's borderline stupidity. It should be highly discouraged if one can help it. It my case, however, "the leopard cannot change his spots" as they say although I personally prefer the more "optimistic" Russian analog which in exact translation sounds "Grave will fix the hunchback" ...

>>4. A discussion forum between reviewers before making acceptance/rejection recommendation is very helpful. Many other graphics conferences have this feature. This should be adapted by SIGGRAPH.

I'm in favor of communication among reviewers for resolving technical issues, but *dislike* review processes which let reviewers see others' actual reviews because it encourages conformity.

I'm more interested in increasing the amount of communication back and forth between reviewers and authors. It's difficult and frustrating to have a single rebuttal and hope the reviewer properly decodes what you're saying. Something like an anonymous message board (or even chat) between reviewers and the author would be great.

Marc's small adjustments for SIGGRAPH certainly sound like good ones. He cannot do a lot in the time he has but doing what is possible is good. Some other suggestions possible this year:

1) eliminate the in-person PC meeting. Having more latency and a good bboard system gives the tertiaries more of a voice and eliminates some of the "we have to decide by Sunday mistakes". This is just a stronger (and more resource efficient) version of what Marc already plans.

2) avoid "paper killers" on the PC committee. I believe much of the variance in the decisions is caused by whether you get a person with a strong personality that wears down the other people until a perfectly good paper is rejected. We all know who these people are (if we have served with them). Just don't put them on the committee.

Michael, thank you for creating this forum.
Here are my two cents.

(1) Submitting source codes is not quite practical. But submitting screen-captured video is feasible and helpful (not crafted video by premier software), e.g. if it is 5-minute siggraph video, 3-minute portion of the video must be screen-caputred real-time video that shows how the system is doing real work. Submissions with real-time captured video should have higher priority to be accepted. Manipulating the 3-minute video is much harder than pictures.

(2) My last suggestion is that when authors are presenting siggraph papers, audience has the right to request them to show their live system, and the authors have responsibility to prepare and demonstrate their system on the spot. (Maybe for some rendering papers, this suggestion is not practical, but for many other papers, it is reasonable.)

Michael, you have a lot courage to do what you did - although you probably shouldn't have. After all, you apparently are/were one of those researchers in the field who actually cared about *content* (as opposed to getting tenure as fast as possible with pretty "designer" papers), and who genuinely tried to advance the field by making contributions that were actually useable.

Most of the comments here on this board are focused on Michael's comments on the SIGGRAPH review process, which IMHO just underlines the absurd level of importance attributed to this one conference. In my opinion the current problems of graphics run deeper than just the way SIGGRAPH is done, though.

The field I'm working in - the problems of which are probably fairly typical within the graphics realm - is physically-based rendering and global illumination.

This is an area that has ceased to be considered "hot" quite some time ago; after all, we have long had ray-tracers and (bidirectional) path tracers (and we were able to do those pretty pictures with glass spheres floating over a checkerboard *ages* ago), so the problems associated with computing arbitrarily accurate images are largely considered to be solved by the community.

There is only one small problem with this, though: if everything in the field of non-interactive global illumination rendering has been solved - where is the software that implements these fancy methods we supposedly have at our disposal now?

If you look into this more you'll find that all we really have are some pretty slick-looking academic papers (a large number of them of course published at SIGGRAPH), and very few books on the topic.

And as far as useable open-source GI software you can download and use for research purposes (i.e. the "live" version of said research results) is concerned: discounting PBRT (which is really nice but still incomplete, and most importantly lacks some more advanced methods) and RenderPark (which was unfortunately abandoned years ago in an incomplete state) the only software out there that is capable of reliably computing global illumination is still Radiance.

Which was written by a guy with a B.S. in electrical engineering in his spare time over 15 years ago.

<cynicism> <slight exaggeration>

Way to go, CG academia - all that those top labs have produced over the past 25 years or so is tons of exquisitely crafted SIGGRAPH papers, but the only working example of all this was done by someone who - at the time he did it - was outside the field.

</cynicism> </slight exaggeration>

(And yes, I know that the actual rendering algorithms in Radiance are somewhat dated, and that Greg now belongs to the community. My point is that practically nothing done since Radiance comes close in terms of robustness and reliability. Which should make anyone in the field think twice, especially given the boisterous claims made in some SIGGRAPH papers published since then.)

To make matters worse, quite a lot of the top publications on global illumination do not really contain all the information needed to duplicate the work supposedly done by the authors - quite a lot of them actually not by a wide margin.

You only notice that the really tricky aspects of the problem have been omitted if you try to implement the stuff yourself, though - a well-written paper with cool result images can hide a lot of missing information quite effectively.

Also, quite a number of these papers fail to discuss the relative merits and shortcomings of the various techniques in an even remotely fair way.

But both these problems make sense given the way current CG academia works: your goal when publishing a paper is to

a) convince the reviewers that you did what you claim (you'll of course have to brag, and show flashy images), and

b) give the other top labs in the field as little information as possible on *how* you did it in order to maintain your lead over them.

Item b) is of course exactly the opposite of what a scientific paper should be about, but given the competitiveness of achieving tenure and getting funding, this nasty tendency is hardly surprising.

And you of course should never mention if your method has any shortcomings at all - you might as well not bother to submit it to SIGGRAPH if your new approach is anything but the absolutely best thing since sliced bread was invented.

(The last paragraphs are of course also slightly exaggerated. But I guess you know what I mean - and unfortunately there really is some truth to this.)

How could this not-so-optimal state of the GI Community be improved, then?

Apart from the suggestions made in other posts (most importantly, improvements to the review processes; this is crucial, since the general quality of reviews is really exceedingly poor these days) my suggestion would be to start to demand source code along with submissions.

If someone cannot submit a working prototype that can be compiled, run and inspected by the reviewers along with his/her paper, then the claimed contribution should be taken with a sizeable grain of salt - at least in the area of global illumination.

Given the complexity of modern GI algorithms it very often pretty absurd for a reviewer (even an expert in the field) to be expected to competently judge the merits of a proposed improvement based on 8 pages of text, and probably half a dozen result images which were mostly picked for their good looks. And in some rare cases, maybe slightly "improved" before submission as well - which is often impossible to tell without access to the renderer that allegedly generated them.

Of course source code submission is not always possible or practical, but my two cents is that we'll have to go in this direction sooner or later, and that the conferences and journals out there should perhaps start to encourage such behaviour in the future. Or at least start to think about how this could be done effectively, since there are of course a number of problems associated with this.

At the moment Graphics in general terms still is great fun - but Graphics Academia definitely is not.

Hopefully we can also manage that academia eventually becomes fun (again), and that people like Michael are not compelled to leave out of disgust any more.

A.W.

P.S. as someone who has never had a paper accepted at SIGGRAPH I probably should have kept my mouth shut about all this, or perhaps have posted anonymously. But somehow I did not feel like hiding behind a nickname today... :-)

I am a new researcher in the field with only less than 5 years of experiences. I appreciate Michael Ashikhmin who initiated this discussion. I admire your courage. I appreciate everyone who post here. Here is my two cents:

1. The junior reviewer.
Many tertiary reviewers are graduate students. Totally eliminate all student reviewer is not realistic. However, something we could do: student reviewers must have a good publication record; faculty advisors should take the responsibility to teach their students how to be a good reviewers; SIGGRAPH could have a short tutorial for students on how to be a responsible reviewers for SIGGRAPH; a student can be selected as a paper reviewer unless his/her current research is very very relevant to the paper to be reviewed.

2. A feedback system
I hope the conference could have a feedback system from author side. If they feel the review is “unfair”, they can file a complaint. For most case, no action needs to take. However, if a committee member or reviewer consistently received significant amount of complaint, he could be removed committee next year. This will not fix the problem for every paper. It does help on eliminate some “bad people”.

3. Now SIGGRAPH reviewers care too many superficial things. Perfect rendering, professional level video, etc. They are good, but not essential. At least for non-rendering paper. Those should not be used to judge the technical soundness of a paper. I remember one year I attended Visualization conference, in a talk to all conference attendees, the paper chair pledged the reviewers not reject a paper based on the limitations the authors discussed in the paper. Unfortunately, many SIGGRAPH reviewers use that as an excuse to kill papers. Indeed, a paper clearly states its limitation should be encouraged.

4. A discussion forum between reviewers before making acceptance/rejection recommendation is very helpful. Many other graphics conferences have this feature. This should be adapted by SIGGRAPH.

The graphics field is not perfect, but we could improve it. I see the hope. I wish SIGGRAPH 2007 a great success.

to 2007 SIGGRAPH paper chair, Prof. Levoy,

Thanks for your great post. You mentioned the system of area chairs that is being used at least in Computer Vision community. Now I try to argue that
the area chair system for SIGGRAPH could be much better than current siggraph paper committee system, if it is properly implemented.

(1) First, significantly increase the number of SIGGRAPH Committees, and set area chairs for each graphics subfield (or set two area co-chairs for each subfield). Both names of siggraph committees and area chairs should be publicly available on siggraph website on submission time. Right now, only paper chair is public, other paper committees are *hidden* super men. It is very weird for me.

(2) For each submission, one or two area chair (or co-chairs) are the primary reviewers for the submission, and the other three tertiary reviewers for this submission ONLY can be selected from paper committees (this is different from current system). It can avoid assigning review jobs to non-committee students or even research interns at XXX research center.
In current siggraph review system, it is not a secret that many paper reviews were done by junior graduate students or even research interns! That is one of the sources why many complaints come up. One big problem of current siggraph review system is that the quality of siggraph paper reviews has unbearable variance and noise. This rule will decrease a lot of noises. Because I assume siggraph paper committees (even significantly increased) are delicately selected based on their expertise and research/publication record.

(3) After reviews are done and authors submit their rebuttal, SIGGRAPH should provide an online (anonymous) dicussion forum for all reviewers of one paper (they can see all review comments). At this stage, area co-chairs
should participate in the discussion too. By interactively discussing this paper/submission online, they can form a semi-final decision on the acceptance/rejection of this work. This semi-final decision is important and it should not be changed generally.

The reason for this design has two important advantages that are ABSENT in current siggraph review systems. The first advantage is if some *evil* reviewer intentionally wrote an unreasonable review, it will make him/her bad in discussion (although still nobody know who is him/her). Also, since this reviewer is intentionally *bad*, he/she will not be confident enough to argue strongly for his/her stupid comments, so the *fair* reviewers (if any) for this submissions still have a chance to rescue this work (if this work is indeed good) and decrease the damage caused by the *bad* reviewer.
It will help to make a more *fair* decision for each submission.

The second advantage is that it can *really* exploit the expertise of ordinary paper committee (tertiary reviewers). Current paper review system really piss tertiary reviewers off, because they are essentially neglectible. I believe most of current siggraph paper committees (or area co-chairs in the proposed system) are *famous* and *ethical* people in the field, but it does not mean in each topic/subsubfield they are the top-level (not even talking about the best) expert, because the field is so broad! and essentially they are not super men either. For example, although I never served in any siggraph program committee, I can safely say that I am not worse than (if not better) most of siggraph program committees in recent years in some specific topics I had worked on for years. In this way, ordinary program committees could say louder on specific topic/paper than area co-chairs.

(4) After semi-final decisions are made, area co-chairs (tens of them) attend a siggraph paper final meeting in person. They checked these semi-final decisions. If there are some wrong decisions or
too many (few) papers are accepted in semi-final decisions, they still can fix it. But in general, they should follow the semi-final decisions which are *truly* peer-reviewed.
     
Finally, I sincerely wish SIGGRAPH could be better. Current situation is not good.

First let me note that I am a relatively new researcher and only worked in the field for about ten years.

I agree to Jonathan's suggestion. SIGGRAPH is a big brand that was built by graphics community for many years, it is not a good idea to give it up. Twice-annual siggraph will make everything easier.

Actually, as pointed out by Jonathan, besides theory (FOCS/STOC), other CS subfields also have two top-tier conferences each year and it works well. For example, computer vision community has two top-tier conferences each year (ICCV/CVPR, or ECCV/CVPR), and Database has even more (SIGMOD/PODS/VLDB/ICDE). So, if one paper is improperly rejected by SIGMOD/PODS, it still have chance to be published in VLDB or ICDE. Authors don't need to hold a submission for one year long.

I have a suggestion that I think will solve many of these problems. The SIGGRAPH board should start a new annual conference that takes place every winter. The new conference will be called "SIGGRAPH".

Although the winter SIGGRAPH will be smaller and more intimate than the summer SIGGRAPH (as it consists primarily of a Papers Program), it will publish the same number of papers (also as a special issue of TOG) and technical sketches, and for publication and tenure purposes will not be thought of as a separate conference--rather, SIGGRAPH simply takes place twice a year now. (We would have to stop using names like "SIGGRAPH 2006" and start calling it something like "SIGGRAPH 43" or "The Forty-Third SIGGRAPH Conference", but that's no great loss.)

The twice-annual SIGGRAPH will play the same role as FOCS and STOC do in the theory community, ensuring that every exceptional paper (rather than a random selection) gets into a top conference, and that researchers don't wait up to a year to submit their best work.

As an added advantage, it would become feasible to have a "committee members cannot submit" rule, because the summer and winter Papers Committees would be disjoint.

The twice-annual SIGGRAPH would not require much more reviewer effort than the once-annual SIGGRAPH does now, but the acceptance rate would double.

Ideally, TOG would publish two additional issues per year, or expand the size of its existing issues, to accommodate both the twice-annual SIGGRAPH proceedings and a good number of full-length articles.

The problem with SIGGRAPH is that computer graphics is too big a field to have only one top-tier conference each year. But as you can see, this problem has a (relatively) easy fix--if the SIGGRAPH organization is willing to take the obvious step.

I have organized a birds-of-a-feather session at SIGGRAPH 2006, with the informal title, Town Hall Meeting about the Siggraph Papers Review Process. Anyone is welcome to attend. The meeting will be Wednesday evening, 6:15 - 7:30pm, in the conference center (BCEC), room 213. That is after the end of the late afternoon Papers session, but before the conference reception. Please pass this information on to others who might not see this posting. I will also attempt to publicize this BOF at the conference itself.

Bring your constructive ideas and your civility.
Remember, being Papers Chair is a volunteer job.

-Marc Levoy
 SIGGRAPH 2007 Papers Chair

Michael,

Let me begin by sharing John Hart's disappointment that you have decided to leave the field of graphics. I have enjoyed reading your papers over the years. It will be a loss to the field if you leave.

Regarding your specific list of complaints, I think that there is truth in almost every point you make. However, most people could come up with a similar list. The harder problem is to prioritize them, to choose a subset that are solvable, and to find workable solutions to them. Although at this point I can only make minor changes for 2007, I am open to all constructive suggestions.

As it happens, I have plans in place to address some of your points. For example, I plan to send paper reviews and rebuttals to tertiaries as well as primaries and secondaries, thereby giving tertiaries more say in the ensuing accept/reject discussion. Also, during the committee meeting I plan to request additional readers from the committee on all papers that have a high variance in their scores, even if the primary and secondary reviewers are unanimous in their proposed accept/reject decision. The goals of these two changes are to reduce bias and noise, respectively. I have also (with input from many people) revised the Call for Participation (CFP) to clarify our policy on double submissions, our policy on what constitutes prior publication, and our guidelines for ethical behavior in the review process. The CFP will be online in a few days.

Regarding your other suggestions, I support the idea of creating alternatives to Siggraph. Indeed, a trend towards co-locating workshops with Siggraph - a trend which Joe Marks (the S2007 General Chair), the Siggraph Executive Committee (EC), and I support - is an attempt to move in this direction. I also support expanding the role of TOG as an alternative publication venue. However, we shouldn't deconstruct the Siggraph Papers program unless and until we have a better replacement.

Unfortunately, some of your other suggestions may be impractical, like prohibiting committee members from submitting papers in any year they serve; doing so would make it difficult for me to staff my committee. I considered switching to a small committee of area chairs (essentially equivalent to your "editors"), and even making public the mapping from papers to area chairs. However, a conference review system based on area chairs seems to be a mixed blessing. I also cannot singlehandedly raise the acceptance rate of Siggraph papers. Although this rate has been remarkably consistent over the years, it is not predetermined; it arises organically by the actions of each year's committee.

Finally, I would be remiss if I did not register polite disagreement with your comments about Columbia and Stanford. Columbia has many papers in Siggraph this year because (in my personal view) they submitted many good papers. Stanford will probably have fewer papers next year (not more), partly because I plan not to submit any papers with my name on them. No rule requires this, but I am committed to avoiding the appearance of bias.

In closing, let me say that I don't think the Siggraph Papers program or its review process are broken. Indeed, my colleagues outside graphics who have had dealings with Siggraph say it is among the most thorough and fairest review processes in the field of computer science. However, any highly competitive peer review system is subject to flaws, and Siggraph's is no exception. These flaws may include the existence of subfields that are viewed (rightly or wrongly) by their participants as being uncollegial. Siggraph is a community of people, and the character of a community changes only by a concerted and gradual effort of the entire community; it cannot be dictated by a program chair or committee.

That said, I am very concerned that the Siggraph Papers program *appear* fair to all its participants. In this regard, the suggestion has been made to hold a birds-of-a-feather session next week about the Siggraph Papers program and its review process. If I could schedule such a session at this late date, would you attend? I would value your input.

-Marc Levoy
 SIGGRAPH 2007 Papers Chair

I was unwillng to post here because of the recording of the IP.

If you are as well, I found "JAP" is the solution.
It's easy to download and setup.

http://anon.inf.tu-dresden.de/index_en.html

ha ha we should get used to it, or get out of graphics.
It is probably the same as any other field, just more insecure.
Graphics has is mostly just importing things from elsewher
(computer vision). There is little which is fundamentally
invented here, and most of it was a long time ago.
And now that most of the core problems have been already solved,
what we have is:

  More and more people, arguing about less and less!

So of course you will see condescending postures
of "not enough contribution" in your rejection.
All the while at the same time SIGGRAPH is advertising
itself as showing the "provocative and significant" and
"seminal" works. Sure, we pretend to be important.
As the importance goes down, the pomp and splendor go up.


Graphics is a fun but derivative field.

First, this frustration is fairly common and we've all vented frustration with the funding and publication system in graphics from time to time. I'm sorry it has become so severe in Michael Ashikhmin's case and hope our community doesn't actually lose him and his expertise.

The problems with SIGGRAPH paper reviewing and citation are well known in our community at all levels. It does surprise me that we all readily admit to the noise in the selection process, yet I still see recommendations (even tenure decisions) based on counting SIGGRAPH papers. To paraphrase Churchill, SIGGRAPH (like democracy) is a horribly bad process, but its better than anything else. In any case, its the system we work in and, flawed as it is, it must be overcome for success in this field.

It is ironic that Michael's SIGGRAPH 2000 paper launched his career in graphics. While SIGGRAPH rejections make it tough, and papers not published in SIGGRAPH can be overlooked, the benefit of the system is that once a paper is published in SIGGRAPH it becomes well known and that's been a difficult, hard-won battle fought by our field's highest impact papers choosing SIGGRAPH as a publication venue. And it's also been a battle to get SIGGRAPH papers properly considered for tenure decisions.

So how about some solutions? We can overcome the SIGGRAPH-or-nothing mentality by paying more attention to other publications, both in citation and in submission. I've been considering forming a list of the top-10 non-SIGGRAPH high-impact graphics papers to accentuate the fact that many of the most important advances in our field were not SIGGRAPH papers.

It is unhealthy for our community when authors choose to resubmit their SIGGRAPH-rejected papers to the following year's SIGGRAPH because it further delays new ideas from publication. (Though I must confess to that sin on several occasions --- as a graphics researcher I have to work the system too.) There is also no guarantee (as Michael shows) that making all of a SIGGRAPH-reviewer's requested revisions guarantees acceptance. This is not the case with journals, which can *accept* a paper *with revisions* that becomes a contact stating if these, *and only these*, revisions are made to the satisfaction of the referees, then the paper will be published. We've been using this as a way to publish SIGGRAPH papers that were rejected because they needed 6-month-or-less changes in TOG. Plus we're always looking for good manuscripts that were mistakenly rejected by SIGGRAPH.

To a young researcher such as myself this discussion is a valuable admonition - thank you to those who have spoken out. Unfortunately it's not the first time I've heard people express effusively negative sentiments about the SIGGRAPH process. Though I don't have as much perspective as others in this discussion, I do agree that the value placed on papers published at this conference is fairly absurd.

Unfortunately, an researcher cannot opt to reject the process, because tenure committees, faculty hiring, and now even admissions commitees use SIGGRAPH publications as a measuring stick. I expect that if less importance is placed on this distinction, corruption of the review process will naturally diminish. It is in the interest of the community to invoke this change.

As for the commonly-held yet ill-supported belief that the problem is less significant because most people find papers via web searches, this couldn't be further from the truth. People still know what SIGGRAPH is and what publication at the conference (supposedly) means. Many people I talk to in industry (those who care about academic publications at all) are skeptical of anything not published in SIGGRAPH, going so far as to use the term "sketchy conferences" to refer to anything non-SIGGRAPH. Given that level of trust, a broken process is a disservice to the larger community.

Finally, I bet that the people for whom the process works honestly and truly do not believe or see that it's broken, and we should not chastise them. It is, however, our responsibility to raise their awareness of the problem.

I agree to the insightful comments from "anonymous".

I can imagine that making the two primary PC members (for one submission) known to its authors can take effects immediately and it shows *real respect* to the hard working of the authors.

If anyone can get siggraph steering committees and these siggraph paper chairs to know this dicussion, it may have some effects.

First let me note that I have published over a dozen papers at Siggraph, and had more than a dozen rejected, so I've had significant experience with Siggraph in the last few years.

Second, I'd like to point out that although many Siggraph reviewers deliver very sophisticated reviews compared to other conferences and other fields, there is also a significiant number of reviews delivered that are uninformed and all too often negatively biased on purpose (while this is often attributed to jealously, competition, etc., it's actually more often linked to plain old survival in a field with limited funding and only one conference that really counts - don't blame the cornered rat for biting - i.e. poeple that usually do this are doing it in defense, e.g. because it happens to them).

Third, Siggraph touts itself as an anonymous conference in order to shield itself from attack, and the establishment does a good job at dismissing the losers and rewarding the winners thus perpetuating this unfair cycle (and most would count me as a winner!). Well, this is absurd. The committee has access to all of the authors names, two of the five reviews come from these committee members, and for papers on the fence even more reviews (sometimes 2-3 more) are solicited from committee members. And many tertiary reviews come from the students and colleagues of the committee members. Even worse, the tertiary reviews typically bare little weight as committee memebers only use them to enforce their own opinions.

Fourth, the only real anonymous thing about Siggraph is that the authors have no idea who is refereeing and making decisions on their paper. That is, if referee A says something completely provably untrue, and the paper is rejected because of that, all an author can do is blame the entire field or all of Siggraph (which is how this whole thread was started). And fighting a whole field or conference is a untenable situation for an author. Of course the committee is publicly known, but with over 50 pretigious people on it and no knowledge of who was handling the paper, complaints are still basically thrown at the conference and fiedl itself.

Finally, my suggestion.

It's obvious that anyone who referees a paper should stay anonymous, especially to protect the academic speech of young faculty looking for tenure or senior faculty looking for grant money. But in many other fields there is an "impartial" editor who is supposed to read the reviews, read the authors rebuttal, and make a decision. And their name and reputation is attached to this decision.

I serve as an editor myself, and before I write a letter to an author accepting or rejecting their paper, I make certain that I stand by my decision based on real facts. If the authors question aspects of a review, I look into in great detail, beacuse I am responsible.

This automatically removes sloppy reviewing, bias, etc. because those reviews and comments are ignored to protect my name as an editor. The conference chair cannot serve this role, because there are hundreds of submissions, rather the primary person on each paper should *not* remain anonymous. They should be an "impartial" judge who does not review the paper, but rather decides on its outcome by weighing and analyzing the factual content of the reviews and authors rebuttal.

I suspect that very very quickly, the complaints about Siggraph and graphics in general will dissappear, and instead there will be a focusing of attention on a few individuals who have been hiding behind a smoke screen for some time. In fact in my area the results will be shocking, because the savvy characters who everyone likes are the ones being sloppy or playing games behind the scenes - which makes sense because people with a reputation for being noisy and grumpy are can't really get away with it. Of course being such great politicians, they'll simply squirm out of it and become the leaders and role models in making good decisions. And that's fine, because we get a better conference, we get a better field, and no one loses.

Surprisingly, getting an impartial editor who takes responsibility in judging each paper and making sure the reviews are accurate seems to be an uphill battle. I was recently shocked to discover that ACM TOG keeps the editors secret. Although IEEE TVCG does not. We might follow their lead...

 I think as long as siggraph keeps its overblown status, only cosmetic changes are possible. Here is a basic problem: 1. There is a limited resource that everybody wants access to (publication or funding) 2. Access is controlled via what is essentially a subjective opinion of some relatively small group of people who, in turn, are interested in obtaining this resource as well. This is basically a Soviet-style economy and while this can certainly work for a while, we all know what eventually happened.

 There is really only two solutions: 1. Have people making decision under some serious pressure to be more objective and resist temptations (do you know that they had death penalty for high ranking officials in cases of "aggravated bribes" in Soviet Union?) There certainly will still be complaints from those who did not get the resource 2. Go to "market", i.e. remove the bottleneck and publish everything which is not complete rubbish in a VISIBLE place. This is what hard sciences essentially arrived at - there are typically two or three THICK journals with quick review cycle which everyone follows, not to mention online publications. At the end, papers which are marginal naturally go below radar very soon while those of some value to others get "market vote". This does happen in graphics and it is true that useful non-siggraph work eventually gets noticed but the time frame is much, much longer while siggraph papers get (often undeserved) automatic spotlite regardless of content. I do acknowledge that this is much harder to implement for funding which is by definition always limited.

 Of many good ideas suggested, I think prohibiting committee members from submitting to siggraph would be a very positive (and extremely easy logistically) development. Not going to happen though and even if it does, will not fully solve the problem since the correlation between committee composition and papers accepted is not only on personal (although there is a lot of it too) but on organization/connections level. Just look at, for example, the number of papers accepted from Columbia this year compared with, say, traditional powerhouse Stanford. I bet Stanford will return next year - guess why? Not necessarily saying that any specific paper was accepted just because of commitee composition, there is a natural doubt that work coming from a specific place just happened to be superior to everyone else's in the particular year their friends were well represented on the committee. I heard far too many stories of people "with access" actively lobbying for their papers and getting the results they want through channels inaccessible to mere mortals...
M.A.

 P.S. At most few dozens of people viewed this forum and less than 10 people posted here so far. Those of you who want this discussion to be of any consequences might think about telling other people (say, at upcoming siggraph) about this place.

I think your observations about acceptance rate and noise, as well as the seriousness of the funding situation (which has a closely related problem because of high threshold) are dead on. I think it would be hard for SIGGRAPH to get a competitor at this stage, so just raising the acceptance rate to an appropriate level to reduce noise is an excellent idea.

I also agree most siggraph reviews are good, although they are better in some areas than others. But there are two ways they have issues in my experience: 1) a cluster of like-minded people dominate the process (this does not have to be a conspiracy, but does slow progress) and reject ideas that compete with that community; 2) when the committee members override the reviewers. Note 2 in this thread has good partial solutions to both of these problems. Add to that your idea of finding the "natural" threshold, and I think there would be huge improvements.

As to funding, just keep telling your NSF PD what a great thing the CAREER program is and that funding should be diverted there. I have been for years-- if enough of us do that and NSF listens, we can help the must vulnerable age group even with a fixed pie.

I only have limited experience with SIGGRAPH and I do not know much about backroom politics. Most arguments that I have heard against Siggraph are anecdotes. My personal experience is that the reviews I got from SIGGRAPH are a lot more detailed and a lot more knowledgeable then reviews I got from other conferences. Therefore, I am still willing to believe that most reviewers invest a lot of time and are honest when they tell me that they do not like my papers.
However, I also believe that the problem of selecting top 20% papers is inherently difficult. It is probably much easier to select the top 35%. If the task is to select top 20% the process gets more unstable. If the selection process has to select top 5-10% it becomes almost random (NSF). I do believe that a change in the selection process, such as mechanisms for reviewer selection, reviewer exclusion, reviewer punishment, ... might bring an improvement, but I do not think that the improvement will be significant enough. I believe the selection of top 20% will still be fairly noisy.
I would therefore suggest a solution that gives the remaining papers in the top 35% a chance to get published, e.g. create a second top venue to publish graphics work.

Still, I am not sure if any of that will help graphics as a field. I would rather suggest the funding situation in computer science as the main problem for many young researchers.

Edited by author 07-18-2006 08:58 PM
The advantage of SIGGRAPH is that a pub means a lot? Means a lot to one's career I grant. But does it mean a lot to advancing our field? With a high threshold and high noise I don't think much is being measured. Dice and backroom politics are not a good way to measure the work being done in our community. Even if we leave the threshold high (which I would argue against but it is a debatable point) the process needs some serious bias and noise reduction for the measurement to be meaningful.

I think that SIGGRAPH is a very competitive conference. It takes a lot of work to publish a paper. This has advantages and disadvantages.

I agree that there are some problems, but I do not think the problems are as excessive as described here. The idea of SIGGRAPH is that you write three very good papers and then you get one accepted. This works well for a professor who leads a larger lab. You submit 9 papers a year and get three in. If you are a student or a new assistant professor with little funding, the pressure to publish is extreme and some unjust borderline decisions are very hurtful to your career. I think the advantage of SIGGRAPH is that it is very difficult to publish and being able to publish means a lot.
The disadvantage is that it is not easily possible to show productivity in the field. You end up with a very low paper count compared to other fields.

to jplewis,

  yeah, I agree to your point. In particular, setting a rule that prevents PC members to submit papers is feasible and useful. I heard in top CS theory conferences (e.g. STOC and FOCS), this rule has been used for quite a while. I didn't see any problem why it can not be used in siggraph.

In response to the previous suggestions from 'graphics fan':

I agree that suggestion (2) (an "exclude these reviewers" list) would help.
I think some journals allow this already.

Part of suggestion (1) is more difficult because it would
reject the papers whose reviews have high variance. The more
interesting or risky ideas would tend to get this reception.
But if (2) were implemented, perhaps (1) would not be necessary.

Additional thoughts:

(3) Bar those people submitting papers from being on the
committee that same year. This would certainly help
with community perception of the process, and it's
easy to implement.

The last time I asked why this was allowed (~15 years ago),
the justification was that there were too few qualified people
in graphics to exclude those who are submitting papers.
That is certainly not the case now.

(4) Consider the creation of a site "badreviews.org".
People would post their favorite review, and the community would
vote on the global worst list.

The intended effect would be that particularly bad reviews
would become known, and the person who assigned these reviewers
might thus become aware that their (the reviewer's) work is not
well regarded and would become more hesitant to assign to them again
(or, the reviewer might be shamed into better behavior,
out of fear that somehow their association with the #1 worst
review might leak out somehow).

I mention this idea only for discussion purposes. I'm sure no one
has time to create such a website, and I suspect that few would visit it,
just because we're all busy.

---

Overall I think the relative importance of Siggraph has diminished a lot
in recent years. In my experience people are exposed to new work
though web searches, and they do not care whether the results
were published in Siggraph or elsewhere.


John Lewis
(I don't submit to Siggraph very often, so not overly worried
about attaching my name).

The SIGGRAPH system has served me well. However, it is now a poor one. Some history is in order-- graphics was once not viewed as a field, had little funding, and was on hard times. SIGGRAPH worked, and soon became a victim of its own success. People that made their career in
SIGGRAPH hesitate to change it for various reasons good and bad. New people have had SIGGRAPH touted so much they sometimes don't really ask if SIGGRAPH makes sense. The question now is whether we in the field actively strive to improve things, and how that might be done. The first step is to admit we have a problem.

  I was on the papers committee in one recent year.
We rejected one paper that got desirable 3/4 ratings
when the other reviewer said something such as
"we have much better work going on back in our labs".
This was my first time on the committee. I was learning
how things work, so I was hesitant to make a point of it,
but later it has bothered me that I did not fight this. The other
reviewer he was papers chair in another year recently.

So yes, the abuse is there, sometimes.

Edited by author 07-07-2006 04:09 PM
Michael,

Your manifesto speaks to some things I have also thought myself. I too very recently made the decision to get out of CS-oriented graphics and move into more applied graphics and visualization by working with people in other fields - people who actually use the techniques we work so hard to develop, not people who sit around navel-gazing and thinking up new ways to demolish young(ish) computer graphics researchers like you and me. I feel if I can advance the state of mathematics and science (REAL science, not computer "science") in some small way through collaboration, then fantastic. I don't feel the need to be "a little famous" in computer graphics, as a senior researcher once told me is good to strive for. Folks who go around talking about SIGGRAPH or IEEE Vis or some other conference as "the only place" to publish need to get a life. It never ceases to amaze me the high premium people in research put on their reputations. How shallow and self-centered it is to go around worrying about what other people think about you.

I have also said to more than a few people that the real scientists (i.e., chemists, physicists, biologists, etc.) have the right model - a conference is basically for discussion and presentation of immature results, and journals are for the final product. That seems to be a pipe-dream for graphics. Even now I find that many graphics journals are basically as bad as SIGGRAPH. Have you tried submitting anything to a graphics journal lately? I once waited a full year to hear back from one of them.

Computer graphics as a field has become overcrowded - too many people, too few venues for publishing. Something will eventually have to give. Bill Lorensen wrote an interesting article a few years ago called "On the Death of Visualization" (http://visual.nlm.nih.gov/evc/meetings/vrc...papers/lorensen.pdf) We may very well see something similar in graphics. Just look at the list of papers at http://www.cs.brown.edu/~tor/sig2006.html. That is a bizarre hodge-podge of topics, many of which I would not consider graphics (image processing, image manipulation, numerical algorithms, motion capture, etc.) The low point was a couple years ago when they had a session on audio. People are clambering for ideas to publish at SIGGRAPH, including ones having almost nothing to do with graphics, just for the sake of sticking another SIGGRAPH papers on their CVs. Ridiculous!

I know two other people who have decided to leave graphics in the
last year. One of them, you know too (or know of, certainly).

The other one is a PhD student who had is work killed by a competing
group once too many times. Every submission would get good scores,
along with one or two really bad scores - and sometimes statements that
the work was a trivial extension of (given citation). The
competing group did not feel any need to disguise itself.

In case you question the quality of his work, his >first< paper
after switching fields received scores of 5 and comments like
"this is a textbook example of a good paper".
(Sorry *, I feel compelled to tell this story).

I strongly agree with you that "Siggraph superconference status" is hurting the health of the whole graphics community, although I admit that in the early stages of graphics, siggraph really help to attract the top graphics researchers together to make it into a premier forum. But now it seems it is time to look at it again. Building several (close) tier-one graphics conferences probably need a long time. Here are my two cents for siggraph.

(1) Add more weights for external referees' review comments. Two primary reviewers (siggraph PC members) now have questionable (too much!) powers on a submission. Hence, I heard that final siggraph paper selection process is fully mingled with a lot of politics. A better way is to reduce the size of discussed papers/submissions strictly based on average scores of submissions. For example, if this year, only 70 papers are expected to accept in siggraph, then only top 100 papers (average scores) can get into final discussions. It means the final 70 papers can ONLY come out of these papers with top 100 average scores! There is no perfect system of course, but I feel it will be more fair and acceptable than current siggraph paper committee systems.
 (2) For each submission, authors can optionally specify up to three referees as non-preferred reviewers (the paper committees should strictly follow this and not assign the submission to them). There are many top researchers in any sub-field of graphics community, it will not degrade the review quality of submissions (also, 5 reviewers are unncessary, 3-4 reviewer are just enough). Furthermore, authors can specify one or two institutions or research Labs as non-preferred reviewer organization (all reviewers in that organization will not be qualified). Some places are notorious for review quality (by intentionally killing other people's work) and authors should have the right to make this kind of request.

   Again, being fair and creative is vital to a healthy research community. If graphics and siggraph community can not maintain this spirit, it is not surprise to see it will go down...

Edited by author 07-01-2006 02:01 AM
 Dear all,

 I would like to welcome you to the "State of Computer Graphics Research" board. This place is intended for an open discussion of the current situation within CG research community and what, if anything, can/should be changed. To start things up, an (incomplete) exposition of my own views on the subject can be found here: http://ash.5000megs.com/leaving.html
 If fact, this board was created in response to a comment on this document from a very well-known and highly respected researcher who thought there is enough interest to this subject out there but no appropriate place to safely express one's views. Although I am personally less convinced that there will be any significant discussion in this forum, I am willing to try starting one.

 Although I do have administrative priviledges over this board, I do not expect to ever exercise them - please post whatever you feel is relevant to the subject hopefully observing some basic courtesy towards other people. Anonymous posts are certainly welcome since openly discussing, for example, paper review process, is "equivalent to professional suicide" (here I quote the senior researcher mentioned above). Remember, however, that signing with your real name will most likely make the post more valuable to the reader.

 Finally, I would like to apologise for any advertisement you might see here - I have absolutely no control over it. For obvious reasons I am using a free service and this is the way it is kept free.

 So, got anything to say about CG research ? Jump in ...

 M.A. (Michael Ashikhmin)