Edited by author 07-01-2006 02:01 AM
 Dear all,

 I would like to welcome you to the "State of Computer Graphics Research" board. This place is intended for an open discussion of the current situation within CG research community and what, if anything, can/should be changed. To start things up, an (incomplete) exposition of my own views on the subject can be found here: http://ash.5000megs.com/leaving.html
 If fact, this board was created in response to a comment on this document from a very well-known and highly respected researcher who thought there is enough interest to this subject out there but no appropriate place to safely express one's views. Although I am personally less convinced that there will be any significant discussion in this forum, I am willing to try starting one.

 Although I do have administrative priviledges over this board, I do not expect to ever exercise them - please post whatever you feel is relevant to the subject hopefully observing some basic courtesy towards other people. Anonymous posts are certainly welcome since openly discussing, for example, paper review process, is "equivalent to professional suicide" (here I quote the senior researcher mentioned above). Remember, however, that signing with your real name will most likely make the post more valuable to the reader.

 Finally, I would like to apologise for any advertisement you might see here - I have absolutely no control over it. For obvious reasons I am using a free service and this is the way it is kept free.

 So, got anything to say about CG research ? Jump in ...

 M.A. (Michael Ashikhmin)

I strongly agree with you that "Siggraph superconference status" is hurting the health of the whole graphics community, although I admit that in the early stages of graphics, siggraph really help to attract the top graphics researchers together to make it into a premier forum. But now it seems it is time to look at it again. Building several (close) tier-one graphics conferences probably need a long time. Here are my two cents for siggraph.

(1) Add more weights for external referees' review comments. Two primary reviewers (siggraph PC members) now have questionable (too much!) powers on a submission. Hence, I heard that final siggraph paper selection process is fully mingled with a lot of politics. A better way is to reduce the size of discussed papers/submissions strictly based on average scores of submissions. For example, if this year, only 70 papers are expected to accept in siggraph, then only top 100 papers (average scores) can get into final discussions. It means the final 70 papers can ONLY come out of these papers with top 100 average scores! There is no perfect system of course, but I feel it will be more fair and acceptable than current siggraph paper committee systems.
 (2) For each submission, authors can optionally specify up to three referees as non-preferred reviewers (the paper committees should strictly follow this and not assign the submission to them). There are many top researchers in any sub-field of graphics community, it will not degrade the review quality of submissions (also, 5 reviewers are unncessary, 3-4 reviewer are just enough). Furthermore, authors can specify one or two institutions or research Labs as non-preferred reviewer organization (all reviewers in that organization will not be qualified). Some places are notorious for review quality (by intentionally killing other people's work) and authors should have the right to make this kind of request.

   Again, being fair and creative is vital to a healthy research community. If graphics and siggraph community can not maintain this spirit, it is not surprise to see it will go down...

I know two other people who have decided to leave graphics in the
last year. One of them, you know too (or know of, certainly).

The other one is a PhD student who had is work killed by a competing
group once too many times. Every submission would get good scores,
along with one or two really bad scores - and sometimes statements that
the work was a trivial extension of (given citation). The
competing group did not feel any need to disguise itself.

In case you question the quality of his work, his >first< paper
after switching fields received scores of 5 and comments like
"this is a textbook example of a good paper".
(Sorry *, I feel compelled to tell this story).

Edited by author 07-07-2006 04:09 PM
Michael,

Your manifesto speaks to some things I have also thought myself. I too very recently made the decision to get out of CS-oriented graphics and move into more applied graphics and visualization by working with people in other fields - people who actually use the techniques we work so hard to develop, not people who sit around navel-gazing and thinking up new ways to demolish young(ish) computer graphics researchers like you and me. I feel if I can advance the state of mathematics and science (REAL science, not computer "science") in some small way through collaboration, then fantastic. I don't feel the need to be "a little famous" in computer graphics, as a senior researcher once told me is good to strive for. Folks who go around talking about SIGGRAPH or IEEE Vis or some other conference as "the only place" to publish need to get a life. It never ceases to amaze me the high premium people in research put on their reputations. How shallow and self-centered it is to go around worrying about what other people think about you.

I have also said to more than a few people that the real scientists (i.e., chemists, physicists, biologists, etc.) have the right model - a conference is basically for discussion and presentation of immature results, and journals are for the final product. That seems to be a pipe-dream for graphics. Even now I find that many graphics journals are basically as bad as SIGGRAPH. Have you tried submitting anything to a graphics journal lately? I once waited a full year to hear back from one of them.

Computer graphics as a field has become overcrowded - too many people, too few venues for publishing. Something will eventually have to give. Bill Lorensen wrote an interesting article a few years ago called "On the Death of Visualization" (http://visual.nlm.nih.gov/evc/meetings/vrc...papers/lorensen.pdf) We may very well see something similar in graphics. Just look at the list of papers at http://www.cs.brown.edu/~tor/sig2006.html. That is a bizarre hodge-podge of topics, many of which I would not consider graphics (image processing, image manipulation, numerical algorithms, motion capture, etc.) The low point was a couple years ago when they had a session on audio. People are clambering for ideas to publish at SIGGRAPH, including ones having almost nothing to do with graphics, just for the sake of sticking another SIGGRAPH papers on their CVs. Ridiculous!

  I was on the papers committee in one recent year.
We rejected one paper that got desirable 3/4 ratings
when the other reviewer said something such as
"we have much better work going on back in our labs".
This was my first time on the committee. I was learning
how things work, so I was hesitant to make a point of it,
but later it has bothered me that I did not fight this. The other
reviewer he was papers chair in another year recently.

So yes, the abuse is there, sometimes.

The SIGGRAPH system has served me well. However, it is now a poor one. Some history is in order-- graphics was once not viewed as a field, had little funding, and was on hard times. SIGGRAPH worked, and soon became a victim of its own success. People that made their career in
SIGGRAPH hesitate to change it for various reasons good and bad. New people have had SIGGRAPH touted so much they sometimes don't really ask if SIGGRAPH makes sense. The question now is whether we in the field actively strive to improve things, and how that might be done. The first step is to admit we have a problem.

In response to the previous suggestions from 'graphics fan':

I agree that suggestion (2) (an "exclude these reviewers" list) would help.
I think some journals allow this already.

Part of suggestion (1) is more difficult because it would
reject the papers whose reviews have high variance. The more
interesting or risky ideas would tend to get this reception.
But if (2) were implemented, perhaps (1) would not be necessary.

Additional thoughts:

(3) Bar those people submitting papers from being on the
committee that same year. This would certainly help
with community perception of the process, and it's
easy to implement.

The last time I asked why this was allowed (~15 years ago),
the justification was that there were too few qualified people
in graphics to exclude those who are submitting papers.
That is certainly not the case now.

(4) Consider the creation of a site "badreviews.org".
People would post their favorite review, and the community would
vote on the global worst list.

The intended effect would be that particularly bad reviews
would become known, and the person who assigned these reviewers
might thus become aware that their (the reviewer's) work is not
well regarded and would become more hesitant to assign to them again
(or, the reviewer might be shamed into better behavior,
out of fear that somehow their association with the #1 worst
review might leak out somehow).

I mention this idea only for discussion purposes. I'm sure no one
has time to create such a website, and I suspect that few would visit it,
just because we're all busy.

---

Overall I think the relative importance of Siggraph has diminished a lot
in recent years. In my experience people are exposed to new work
though web searches, and they do not care whether the results
were published in Siggraph or elsewhere.


John Lewis
(I don't submit to Siggraph very often, so not overly worried
about attaching my name).

to jplewis,

  yeah, I agree to your point. In particular, setting a rule that prevents PC members to submit papers is feasible and useful. I heard in top CS theory conferences (e.g. STOC and FOCS), this rule has been used for quite a while. I didn't see any problem why it can not be used in siggraph.

I think that SIGGRAPH is a very competitive conference. It takes a lot of work to publish a paper. This has advantages and disadvantages.

I agree that there are some problems, but I do not think the problems are as excessive as described here. The idea of SIGGRAPH is that you write three very good papers and then you get one accepted. This works well for a professor who leads a larger lab. You submit 9 papers a year and get three in. If you are a student or a new assistant professor with little funding, the pressure to publish is extreme and some unjust borderline decisions are very hurtful to your career. I think the advantage of SIGGRAPH is that it is very difficult to publish and being able to publish means a lot.
The disadvantage is that it is not easily possible to show productivity in the field. You end up with a very low paper count compared to other fields.

Edited by author 07-18-2006 08:58 PM
The advantage of SIGGRAPH is that a pub means a lot? Means a lot to one's career I grant. But does it mean a lot to advancing our field? With a high threshold and high noise I don't think much is being measured. Dice and backroom politics are not a good way to measure the work being done in our community. Even if we leave the threshold high (which I would argue against but it is a debatable point) the process needs some serious bias and noise reduction for the measurement to be meaningful.

I only have limited experience with SIGGRAPH and I do not know much about backroom politics. Most arguments that I have heard against Siggraph are anecdotes. My personal experience is that the reviews I got from SIGGRAPH are a lot more detailed and a lot more knowledgeable then reviews I got from other conferences. Therefore, I am still willing to believe that most reviewers invest a lot of time and are honest when they tell me that they do not like my papers.
However, I also believe that the problem of selecting top 20% papers is inherently difficult. It is probably much easier to select the top 35%. If the task is to select top 20% the process gets more unstable. If the selection process has to select top 5-10% it becomes almost random (NSF). I do believe that a change in the selection process, such as mechanisms for reviewer selection, reviewer exclusion, reviewer punishment, ... might bring an improvement, but I do not think that the improvement will be significant enough. I believe the selection of top 20% will still be fairly noisy.
I would therefore suggest a solution that gives the remaining papers in the top 35% a chance to get published, e.g. create a second top venue to publish graphics work.

Still, I am not sure if any of that will help graphics as a field. I would rather suggest the funding situation in computer science as the main problem for many young researchers.

I think your observations about acceptance rate and noise, as well as the seriousness of the funding situation (which has a closely related problem because of high threshold) are dead on. I think it would be hard for SIGGRAPH to get a competitor at this stage, so just raising the acceptance rate to an appropriate level to reduce noise is an excellent idea.

I also agree most siggraph reviews are good, although they are better in some areas than others. But there are two ways they have issues in my experience: 1) a cluster of like-minded people dominate the process (this does not have to be a conspiracy, but does slow progress) and reject ideas that compete with that community; 2) when the committee members override the reviewers. Note 2 in this thread has good partial solutions to both of these problems. Add to that your idea of finding the "natural" threshold, and I think there would be huge improvements.

As to funding, just keep telling your NSF PD what a great thing the CAREER program is and that funding should be diverted there. I have been for years-- if enough of us do that and NSF listens, we can help the must vulnerable age group even with a fixed pie.

 I think as long as siggraph keeps its overblown status, only cosmetic changes are possible. Here is a basic problem: 1. There is a limited resource that everybody wants access to (publication or funding) 2. Access is controlled via what is essentially a subjective opinion of some relatively small group of people who, in turn, are interested in obtaining this resource as well. This is basically a Soviet-style economy and while this can certainly work for a while, we all know what eventually happened.

 There is really only two solutions: 1. Have people making decision under some serious pressure to be more objective and resist temptations (do you know that they had death penalty for high ranking officials in cases of "aggravated bribes" in Soviet Union?) There certainly will still be complaints from those who did not get the resource 2. Go to "market", i.e. remove the bottleneck and publish everything which is not complete rubbish in a VISIBLE place. This is what hard sciences essentially arrived at - there are typically two or three THICK journals with quick review cycle which everyone follows, not to mention online publications. At the end, papers which are marginal naturally go below radar very soon while those of some value to others get "market vote". This does happen in graphics and it is true that useful non-siggraph work eventually gets noticed but the time frame is much, much longer while siggraph papers get (often undeserved) automatic spotlite regardless of content. I do acknowledge that this is much harder to implement for funding which is by definition always limited.

 Of many good ideas suggested, I think prohibiting committee members from submitting to siggraph would be a very positive (and extremely easy logistically) development. Not going to happen though and even if it does, will not fully solve the problem since the correlation between committee composition and papers accepted is not only on personal (although there is a lot of it too) but on organization/connections level. Just look at, for example, the number of papers accepted from Columbia this year compared with, say, traditional powerhouse Stanford. I bet Stanford will return next year - guess why? Not necessarily saying that any specific paper was accepted just because of commitee composition, there is a natural doubt that work coming from a specific place just happened to be superior to everyone else's in the particular year their friends were well represented on the committee. I heard far too many stories of people "with access" actively lobbying for their papers and getting the results they want through channels inaccessible to mere mortals...
M.A.

 P.S. At most few dozens of people viewed this forum and less than 10 people posted here so far. Those of you who want this discussion to be of any consequences might think about telling other people (say, at upcoming siggraph) about this place.

First let me note that I have published over a dozen papers at Siggraph, and had more than a dozen rejected, so I've had significant experience with Siggraph in the last few years.

Second, I'd like to point out that although many Siggraph reviewers deliver very sophisticated reviews compared to other conferences and other fields, there is also a significiant number of reviews delivered that are uninformed and all too often negatively biased on purpose (while this is often attributed to jealously, competition, etc., it's actually more often linked to plain old survival in a field with limited funding and only one conference that really counts - don't blame the cornered rat for biting - i.e. poeple that usually do this are doing it in defense, e.g. because it happens to them).

Third, Siggraph touts itself as an anonymous conference in order to shield itself from attack, and the establishment does a good job at dismissing the losers and rewarding the winners thus perpetuating this unfair cycle (and most would count me as a winner!). Well, this is absurd. The committee has access to all of the authors names, two of the five reviews come from these committee members, and for papers on the fence even more reviews (sometimes 2-3 more) are solicited from committee members. And many tertiary reviews come from the students and colleagues of the committee members. Even worse, the tertiary reviews typically bare little weight as committee memebers only use them to enforce their own opinions.

Fourth, the only real anonymous thing about Siggraph is that the authors have no idea who is refereeing and making decisions on their paper. That is, if referee A says something completely provably untrue, and the paper is rejected because of that, all an author can do is blame the entire field or all of Siggraph (which is how this whole thread was started). And fighting a whole field or conference is a untenable situation for an author. Of course the committee is publicly known, but with over 50 pretigious people on it and no knowledge of who was handling the paper, complaints are still basically thrown at the conference and fiedl itself.

Finally, my suggestion.

It's obvious that anyone who referees a paper should stay anonymous, especially to protect the academic speech of young faculty looking for tenure or senior faculty looking for grant money. But in many other fields there is an "impartial" editor who is supposed to read the reviews, read the authors rebuttal, and make a decision. And their name and reputation is attached to this decision.

I serve as an editor myself, and before I write a letter to an author accepting or rejecting their paper, I make certain that I stand by my decision based on real facts. If the authors question aspects of a review, I look into in great detail, beacuse I am responsible.

This automatically removes sloppy reviewing, bias, etc. because those reviews and comments are ignored to protect my name as an editor. The conference chair cannot serve this role, because there are hundreds of submissions, rather the primary person on each paper should *not* remain anonymous. They should be an "impartial" judge who does not review the paper, but rather decides on its outcome by weighing and analyzing the factual content of the reviews and authors rebuttal.

I suspect that very very quickly, the complaints about Siggraph and graphics in general will dissappear, and instead there will be a focusing of attention on a few individuals who have been hiding behind a smoke screen for some time. In fact in my area the results will be shocking, because the savvy characters who everyone likes are the ones being sloppy or playing games behind the scenes - which makes sense because people with a reputation for being noisy and grumpy are can't really get away with it. Of course being such great politicians, they'll simply squirm out of it and become the leaders and role models in making good decisions. And that's fine, because we get a better conference, we get a better field, and no one loses.

Surprisingly, getting an impartial editor who takes responsibility in judging each paper and making sure the reviews are accurate seems to be an uphill battle. I was recently shocked to discover that ACM TOG keeps the editors secret. Although IEEE TVCG does not. We might follow their lead...

I agree to the insightful comments from "anonymous".

I can imagine that making the two primary PC members (for one submission) known to its authors can take effects immediately and it shows *real respect* to the hard working of the authors.

If anyone can get siggraph steering committees and these siggraph paper chairs to know this dicussion, it may have some effects.

To a young researcher such as myself this discussion is a valuable admonition - thank you to those who have spoken out. Unfortunately it's not the first time I've heard people express effusively negative sentiments about the SIGGRAPH process. Though I don't have as much perspective as others in this discussion, I do agree that the value placed on papers published at this conference is fairly absurd.

Unfortunately, an researcher cannot opt to reject the process, because tenure committees, faculty hiring, and now even admissions commitees use SIGGRAPH publications as a measuring stick. I expect that if less importance is placed on this distinction, corruption of the review process will naturally diminish. It is in the interest of the community to invoke this change.

As for the commonly-held yet ill-supported belief that the problem is less significant because most people find papers via web searches, this couldn't be further from the truth. People still know what SIGGRAPH is and what publication at the conference (supposedly) means. Many people I talk to in industry (those who care about academic publications at all) are skeptical of anything not published in SIGGRAPH, going so far as to use the term "sketchy conferences" to refer to anything non-SIGGRAPH. Given that level of trust, a broken process is a disservice to the larger community.

Finally, I bet that the people for whom the process works honestly and truly do not believe or see that it's broken, and we should not chastise them. It is, however, our responsibility to raise their awareness of the problem.

First, this frustration is fairly common and we've all vented frustration with the funding and publication system in graphics from time to time. I'm sorry it has become so severe in Michael Ashikhmin's case and hope our community doesn't actually lose him and his expertise.

The problems with SIGGRAPH paper reviewing and citation are well known in our community at all levels. It does surprise me that we all readily admit to the noise in the selection process, yet I still see recommendations (even tenure decisions) based on counting SIGGRAPH papers. To paraphrase Churchill, SIGGRAPH (like democracy) is a horribly bad process, but its better than anything else. In any case, its the system we work in and, flawed as it is, it must be overcome for success in this field.

It is ironic that Michael's SIGGRAPH 2000 paper launched his career in graphics. While SIGGRAPH rejections make it tough, and papers not published in SIGGRAPH can be overlooked, the benefit of the system is that once a paper is published in SIGGRAPH it becomes well known and that's been a difficult, hard-won battle fought by our field's highest impact papers choosing SIGGRAPH as a publication venue. And it's also been a battle to get SIGGRAPH papers properly considered for tenure decisions.

So how about some solutions? We can overcome the SIGGRAPH-or-nothing mentality by paying more attention to other publications, both in citation and in submission. I've been considering forming a list of the top-10 non-SIGGRAPH high-impact graphics papers to accentuate the fact that many of the most important advances in our field were not SIGGRAPH papers.

It is unhealthy for our community when authors choose to resubmit their SIGGRAPH-rejected papers to the following year's SIGGRAPH because it further delays new ideas from publication. (Though I must confess to that sin on several occasions --- as a graphics researcher I have to work the system too.) There is also no guarantee (as Michael shows) that making all of a SIGGRAPH-reviewer's requested revisions guarantees acceptance. This is not the case with journals, which can *accept* a paper *with revisions* that becomes a contact stating if these, *and only these*, revisions are made to the satisfaction of the referees, then the paper will be published. We've been using this as a way to publish SIGGRAPH papers that were rejected because they needed 6-month-or-less changes in TOG. Plus we're always looking for good manuscripts that were mistakenly rejected by SIGGRAPH.

ha ha we should get used to it, or get out of graphics.
It is probably the same as any other field, just more insecure.
Graphics has is mostly just importing things from elsewher
(computer vision). There is little which is fundamentally
invented here, and most of it was a long time ago.
And now that most of the core problems have been already solved,
what we have is:

  More and more people, arguing about less and less!

So of course you will see condescending postures
of "not enough contribution" in your rejection.
All the while at the same time SIGGRAPH is advertising
itself as showing the "provocative and significant" and
"seminal" works. Sure, we pretend to be important.
As the importance goes down, the pomp and splendor go up.


Graphics is a fun but derivative field.

I was unwillng to post here because of the recording of the IP.

If you are as well, I found "JAP" is the solution.
It's easy to download and setup.

http://anon.inf.tu-dresden.de/index_en.html

Michael,

Let me begin by sharing John Hart's disappointment that you have decided to leave the field of graphics. I have enjoyed reading your papers over the years. It will be a loss to the field if you leave.

Regarding your specific list of complaints, I think that there is truth in almost every point you make. However, most people could come up with a similar list. The harder problem is to prioritize them, to choose a subset that are solvable, and to find workable solutions to them. Although at this point I can only make minor changes for 2007, I am open to all constructive suggestions.

As it happens, I have plans in place to address some of your points. For example, I plan to send paper reviews and rebuttals to tertiaries as well as primaries and secondaries, thereby giving tertiaries more say in the ensuing accept/reject discussion. Also, during the committee meeting I plan to request additional readers from the committee on all papers that have a high variance in their scores, even if the primary and secondary reviewers are unanimous in their proposed accept/reject decision. The goals of these two changes are to reduce bias and noise, respectively. I have also (with input from many people) revised the Call for Participation (CFP) to clarify our policy on double submissions, our policy on what constitutes prior publication, and our guidelines for ethical behavior in the review process. The CFP will be online in a few days.

Regarding your other suggestions, I support the idea of creating alternatives to Siggraph. Indeed, a trend towards co-locating workshops with Siggraph - a trend which Joe Marks (the S2007 General Chair), the Siggraph Executive Committee (EC), and I support - is an attempt to move in this direction. I also support expanding the role of TOG as an alternative publication venue. However, we shouldn't deconstruct the Siggraph Papers program unless and until we have a better replacement.

Unfortunately, some of your other suggestions may be impractical, like prohibiting committee members from submitting papers in any year they serve; doing so would make it difficult for me to staff my committee. I considered switching to a small committee of area chairs (essentially equivalent to your "editors"), and even making public the mapping from papers to area chairs. However, a conference review system based on area chairs seems to be a mixed blessing. I also cannot singlehandedly raise the acceptance rate of Siggraph papers. Although this rate has been remarkably consistent over the years, it is not predetermined; it arises organically by the actions of each year's committee.

Finally, I would be remiss if I did not register polite disagreement with your comments about Columbia and Stanford. Columbia has many papers in Siggraph this year because (in my personal view) they submitted many good papers. Stanford will probably have fewer papers next year (not more), partly because I plan not to submit any papers with my name on them. No rule requires this, but I am committed to avoiding the appearance of bias.

In closing, let me say that I don't think the Siggraph Papers program or its review process are broken. Indeed, my colleagues outside graphics who have had dealings with Siggraph say it is among the most thorough and fairest review processes in the field of computer science. However, any highly competitive peer review system is subject to flaws, and Siggraph's is no exception. These flaws may include the existence of subfields that are viewed (rightly or wrongly) by their participants as being uncollegial. Siggraph is a community of people, and the character of a community changes only by a concerted and gradual effort of the entire community; it cannot be dictated by a program chair or committee.

That said, I am very concerned that the Siggraph Papers program *appear* fair to all its participants. In this regard, the suggestion has been made to hold a birds-of-a-feather session next week about the Siggraph Papers program and its review process. If I could schedule such a session at this late date, would you attend? I would value your input.

-Marc Levoy
 SIGGRAPH 2007 Papers Chair

I have organized a birds-of-a-feather session at SIGGRAPH 2006, with the informal title, Town Hall Meeting about the Siggraph Papers Review Process. Anyone is welcome to attend. The meeting will be Wednesday evening, 6:15 - 7:30pm, in the conference center (BCEC), room 213. That is after the end of the late afternoon Papers session, but before the conference reception. Please pass this information on to others who might not see this posting. I will also attempt to publicize this BOF at the conference itself.

Bring your constructive ideas and your civility.
Remember, being Papers Chair is a volunteer job.

-Marc Levoy
 SIGGRAPH 2007 Papers Chair

I have a suggestion that I think will solve many of these problems. The SIGGRAPH board should start a new annual conference that takes place every winter. The new conference will be called "SIGGRAPH".

Although the winter SIGGRAPH will be smaller and more intimate than the summer SIGGRAPH (as it consists primarily of a Papers Program), it will publish the same number of papers (also as a special issue of TOG) and technical sketches, and for publication and tenure purposes will not be thought of as a separate conference--rather, SIGGRAPH simply takes place twice a year now. (We would have to stop using names like "SIGGRAPH 2006" and start calling it something like "SIGGRAPH 43" or "The Forty-Third SIGGRAPH Conference", but that's no great loss.)

The twice-annual SIGGRAPH will play the same role as FOCS and STOC do in the theory community, ensuring that every exceptional paper (rather than a random selection) gets into a top conference, and that researchers don't wait up to a year to submit their best work.

As an added advantage, it would become feasible to have a "committee members cannot submit" rule, because the summer and winter Papers Committees would be disjoint.

The twice-annual SIGGRAPH would not require much more reviewer effort than the once-annual SIGGRAPH does now, but the acceptance rate would double.

Ideally, TOG would publish two additional issues per year, or expand the size of its existing issues, to accommodate both the twice-annual SIGGRAPH proceedings and a good number of full-length articles.

The problem with SIGGRAPH is that computer graphics is too big a field to have only one top-tier conference each year. But as you can see, this problem has a (relatively) easy fix--if the SIGGRAPH organization is willing to take the obvious step.

First let me note that I am a relatively new researcher and only worked in the field for about ten years.

I agree to Jonathan's suggestion. SIGGRAPH is a big brand that was built by graphics community for many years, it is not a good idea to give it up. Twice-annual siggraph will make everything easier.

Actually, as pointed out by Jonathan, besides theory (FOCS/STOC), other CS subfields also have two top-tier conferences each year and it works well. For example, computer vision community has two top-tier conferences each year (ICCV/CVPR, or ECCV/CVPR), and Database has even more (SIGMOD/PODS/VLDB/ICDE). So, if one paper is improperly rejected by SIGMOD/PODS, it still have chance to be published in VLDB or ICDE. Authors don't need to hold a submission for one year long.

to 2007 SIGGRAPH paper chair, Prof. Levoy,

Thanks for your great post. You mentioned the system of area chairs that is being used at least in Computer Vision community. Now I try to argue that
the area chair system for SIGGRAPH could be much better than current siggraph paper committee system, if it is properly implemented.

(1) First, significantly increase the number of SIGGRAPH Committees, and set area chairs for each graphics subfield (or set two area co-chairs for each subfield). Both names of siggraph committees and area chairs should be publicly available on siggraph website on submission time. Right now, only paper chair is public, other paper committees are *hidden* super men. It is very weird for me.

(2) For each submission, one or two area chair (or co-chairs) are the primary reviewers for the submission, and the other three tertiary reviewers for this submission ONLY can be selected from paper committees (this is different from current system). It can avoid assigning review jobs to non-committee students or even research interns at XXX research center.
In current siggraph review system, it is not a secret that many paper reviews were done by junior graduate students or even research interns! That is one of the sources why many complaints come up. One big problem of current siggraph review system is that the quality of siggraph paper reviews has unbearable variance and noise. This rule will decrease a lot of noises. Because I assume siggraph paper committees (even significantly increased) are delicately selected based on their expertise and research/publication record.

(3) After reviews are done and authors submit their rebuttal, SIGGRAPH should provide an online (anonymous) dicussion forum for all reviewers of one paper (they can see all review comments). At this stage, area co-chairs
should participate in the discussion too. By interactively discussing this paper/submission online, they can form a semi-final decision on the acceptance/rejection of this work. This semi-final decision is important and it should not be changed generally.

The reason for this design has two important advantages that are ABSENT in current siggraph review systems. The first advantage is if some *evil* reviewer intentionally wrote an unreasonable review, it will make him/her bad in discussion (although still nobody know who is him/her). Also, since this reviewer is intentionally *bad*, he/she will not be confident enough to argue strongly for his/her stupid comments, so the *fair* reviewers (if any) for this submissions still have a chance to rescue this work (if this work is indeed good) and decrease the damage caused by the *bad* reviewer.
It will help to make a more *fair* decision for each submission.

The second advantage is that it can *really* exploit the expertise of ordinary paper committee (tertiary reviewers). Current paper review system really piss tertiary reviewers off, because they are essentially neglectible. I believe most of current siggraph paper committees (or area co-chairs in the proposed system) are *famous* and *ethical* people in the field, but it does not mean in each topic/subsubfield they are the top-level (not even talking about the best) expert, because the field is so broad! and essentially they are not super men either. For example, although I never served in any siggraph program committee, I can safely say that I am not worse than (if not better) most of siggraph program committees in recent years in some specific topics I had worked on for years. In this way, ordinary program committees could say louder on specific topic/paper than area co-chairs.

(4) After semi-final decisions are made, area co-chairs (tens of them) attend a siggraph paper final meeting in person. They checked these semi-final decisions. If there are some wrong decisions or
too many (few) papers are accepted in semi-final decisions, they still can fix it. But in general, they should follow the semi-final decisions which are *truly* peer-reviewed.
     
Finally, I sincerely wish SIGGRAPH could be better. Current situation is not good.

I am a new researcher in the field with only less than 5 years of experiences. I appreciate Michael Ashikhmin who initiated this discussion. I admire your courage. I appreciate everyone who post here. Here is my two cents:

1. The junior reviewer.
Many tertiary reviewers are graduate students. Totally eliminate all student reviewer is not realistic. However, something we could do: student reviewers must have a good publication record; faculty advisors should take the responsibility to teach their students how to be a good reviewers; SIGGRAPH could have a short tutorial for students on how to be a responsible reviewers for SIGGRAPH; a student can be selected as a paper reviewer unless his/her current research is very very relevant to the paper to be reviewed.

2. A feedback system
I hope the conference could have a feedback system from author side. If they feel the review is “unfair”, they can file a complaint. For most case, no action needs to take. However, if a committee member or reviewer consistently received significant amount of complaint, he could be removed committee next year. This will not fix the problem for every paper. It does help on eliminate some “bad people”.

3. Now SIGGRAPH reviewers care too many superficial things. Perfect rendering, professional level video, etc. They are good, but not essential. At least for non-rendering paper. Those should not be used to judge the technical soundness of a paper. I remember one year I attended Visualization conference, in a talk to all conference attendees, the paper chair pledged the reviewers not reject a paper based on the limitations the authors discussed in the paper. Unfortunately, many SIGGRAPH reviewers use that as an excuse to kill papers. Indeed, a paper clearly states its limitation should be encouraged.

4. A discussion forum between reviewers before making acceptance/rejection recommendation is very helpful. Many other graphics conferences have this feature. This should be adapted by SIGGRAPH.

The graphics field is not perfect, but we could improve it. I see the hope. I wish SIGGRAPH 2007 a great success.

Michael, you have a lot courage to do what you did - although you probably shouldn't have. After all, you apparently are/were one of those researchers in the field who actually cared about *content* (as opposed to getting tenure as fast as possible with pretty "designer" papers), and who genuinely tried to advance the field by making contributions that were actually useable.

Most of the comments here on this board are focused on Michael's comments on the SIGGRAPH review process, which IMHO just underlines the absurd level of importance attributed to this one conference. In my opinion the current problems of graphics run deeper than just the way SIGGRAPH is done, though.

The field I'm working in - the problems of which are probably fairly typical within the graphics realm - is physically-based rendering and global illumination.

This is an area that has ceased to be considered "hot" quite some time ago; after all, we have long had ray-tracers and (bidirectional) path tracers (and we were able to do those pretty pictures with glass spheres floating over a checkerboard *ages* ago), so the problems associated with computing arbitrarily accurate images are largely considered to be solved by the community.

There is only one small problem with this, though: if everything in the field of non-interactive global illumination rendering has been solved - where is the software that implements these fancy methods we supposedly have at our disposal now?

If you look into this more you'll find that all we really have are some pretty slick-looking academic papers (a large number of them of course published at SIGGRAPH), and very few books on the topic.

And as far as useable open-source GI software you can download and use for research purposes (i.e. the "live" version of said research results) is concerned: discounting PBRT (which is really nice but still incomplete, and most importantly lacks some more advanced methods) and RenderPark (which was unfortunately abandoned years ago in an incomplete state) the only software out there that is capable of reliably computing global illumination is still Radiance.

Which was written by a guy with a B.S. in electrical engineering in his spare time over 15 years ago.

<cynicism> <slight exaggeration>

Way to go, CG academia - all that those top labs have produced over the past 25 years or so is tons of exquisitely crafted SIGGRAPH papers, but the only working example of all this was done by someone who - at the time he did it - was outside the field.

</cynicism> </slight exaggeration>

(And yes, I know that the actual rendering algorithms in Radiance are somewhat dated, and that Greg now belongs to the community. My point is that practically nothing done since Radiance comes close in terms of robustness and reliability. Which should make anyone in the field think twice, especially given the boisterous claims made in some SIGGRAPH papers published since then.)

To make matters worse, quite a lot of the top publications on global illumination do not really contain all the information needed to duplicate the work supposedly done by the authors - quite a lot of them actually not by a wide margin.

You only notice that the really tricky aspects of the problem have been omitted if you try to implement the stuff yourself, though - a well-written paper with cool result images can hide a lot of missing information quite effectively.

Also, quite a number of these papers fail to discuss the relative merits and shortcomings of the various techniques in an even remotely fair way.

But both these problems make sense given the way current CG academia works: your goal when publishing a paper is to

a) convince the reviewers that you did what you claim (you'll of course have to brag, and show flashy images), and

b) give the other top labs in the field as little information as possible on *how* you did it in order to maintain your lead over them.

Item b) is of course exactly the opposite of what a scientific paper should be about, but given the competitiveness of achieving tenure and getting funding, this nasty tendency is hardly surprising.

And you of course should never mention if your method has any shortcomings at all - you might as well not bother to submit it to SIGGRAPH if your new approach is anything but the absolutely best thing since sliced bread was invented.

(The last paragraphs are of course also slightly exaggerated. But I guess you know what I mean - and unfortunately there really is some truth to this.)

How could this not-so-optimal state of the GI Community be improved, then?

Apart from the suggestions made in other posts (most importantly, improvements to the review processes; this is crucial, since the general quality of reviews is really exceedingly poor these days) my suggestion would be to start to demand source code along with submissions.

If someone cannot submit a working prototype that can be compiled, run and inspected by the reviewers along with his/her paper, then the claimed contribution should be taken with a sizeable grain of salt - at least in the area of global illumination.

Given the complexity of modern GI algorithms it very often pretty absurd for a reviewer (even an expert in the field) to be expected to competently judge the merits of a proposed improvement based on 8 pages of text, and probably half a dozen result images which were mostly picked for their good looks. And in some rare cases, maybe slightly "improved" before submission as well - which is often impossible to tell without access to the renderer that allegedly generated them.

Of course source code submission is not always possible or practical, but my two cents is that we'll have to go in this direction sooner or later, and that the conferences and journals out there should perhaps start to encourage such behaviour in the future. Or at least start to think about how this could be done effectively, since there are of course a number of problems associated with this.

At the moment Graphics in general terms still is great fun - but Graphics Academia definitely is not.

Hopefully we can also manage that academia eventually becomes fun (again), and that people like Michael are not compelled to leave out of disgust any more.

A.W.

P.S. as someone who has never had a paper accepted at SIGGRAPH I probably should have kept my mouth shut about all this, or perhaps have posted anonymously. But somehow I did not feel like hiding behind a nickname today... :-)

Michael, thank you for creating this forum.
Here are my two cents.

(1) Submitting source codes is not quite practical. But submitting screen-captured video is feasible and helpful (not crafted video by premier software), e.g. if it is 5-minute siggraph video, 3-minute portion of the video must be screen-caputred real-time video that shows how the system is doing real work. Submissions with real-time captured video should have higher priority to be accepted. Manipulating the 3-minute video is much harder than pictures.

(2) My last suggestion is that when authors are presenting siggraph papers, audience has the right to request them to show their live system, and the authors have responsibility to prepare and demonstrate their system on the spot. (Maybe for some rendering papers, this suggestion is not practical, but for many other papers, it is reasonable.)

Marc's small adjustments for SIGGRAPH certainly sound like good ones. He cannot do a lot in the time he has but doing what is possible is good. Some other suggestions possible this year:

1) eliminate the in-person PC meeting. Having more latency and a good bboard system gives the tertiaries more of a voice and eliminates some of the "we have to decide by Sunday mistakes". This is just a stronger (and more resource efficient) version of what Marc already plans.

2) avoid "paper killers" on the PC committee. I believe much of the variance in the decisions is caused by whether you get a person with a strong personality that wears down the other people until a perfectly good paper is rejected. We all know who these people are (if we have served with them). Just don't put them on the committee.

>>4. A discussion forum between reviewers before making acceptance/rejection recommendation is very helpful. Many other graphics conferences have this feature. This should be adapted by SIGGRAPH.

I'm in favor of communication among reviewers for resolving technical issues, but *dislike* review processes which let reviewers see others' actual reviews because it encourages conformity.

I'm more interested in increasing the amount of communication back and forth between reviewers and authors. It's difficult and frustrating to have a single rebuttal and hope the reviewer properly decodes what you're saying. Something like an anonymous message board (or even chat) between reviewers and the author would be great.

 Dear all,
 First of all, message #28 in this thread has a note that it was "Deleted by topic administrator 07-26-2006 11:12 AM". I would like to assure you that it was not me. I have NOT deleted any messages and have no idea who deleted it and why this happened.

 Second, I certainly wish Prof. Levoy best of luck with organizing next paper revision process as well as a session at siggraph-2006 (information about which is, by the way, is currently not present on the web page where BOFs are listed, i.e. http://www.siggraph.org/s2006/main.php?f=conference&p=birds ). I do have some doubts that people, especially younger ones, will be willing to speak openly when everyone sees who they are. Still, I wish I could attend what might end up being an interesting discussion but have already started at my new job.

 Finally, the document which initiated all this, i.e. http://ash.5000megs.com/leaving.html
has disappeared for some reason along with the whole site. I was NOT planning to remove it and hope it is just some temporary technical difficulties - you get what you pay for and I paid exactly nothing. However, given the upcoming siggraph session, I think it is useful to have it available. Plus, it contains probably the only public link to this discussion forum. I will therefore put this document to my very old Utah web page for now. See http://www.cs.utah.edu/~michael/leaving.html

 I hope you understand that U. of U. CS Department by no means endorses it. Therefore, I would appreciate if those who disagree with my views do not ask Department officials to remove it, directly or indirectly (as some of you apparently did a year ago concerning my Stony Brook "siggraph broken" note).

M.A.

 P.S. Dear Alex and others. Thanks for nice words, but inability to shut up for one's own good is hardly courage, it's borderline stupidity. It should be highly discouraged if one can help it. It my case, however, "the leopard cannot change his spots" as they say although I personally prefer the more "optimistic" Russian analog which in exact translation sounds "Grave will fix the hunchback" ...

Just a note on how things get decided in SIGGRAPH:

Key conference positions (chair, papers chair, courses chair...) are approved and policy decisions are made by members of the SIGGRAPH Executive Committee. Members of the Executive Committee are elected by popular vote by the SIGGRAPH membership. One way to influence what happens in SIGGRAPH is to volunteer or nominate someone else for an Executive Committee post :
http://www.siggraph.org/newsfeed/call-for-nominees/
(yes the date posted there is passed, but it is not too late to contact Alain Chesnais and express and interest.) If you or your nominee is not selected by the nominating committee, it is possible to petition to put someone on the ballot
http://www.siggraph.org/gen-info/bylaws.html#article_10
This has been a successful approach in the recent past. Note that you have to join SIGGRAPH to vote.Finally, when it is time to vote, please consider who you think will make the best decisions about how the conference is conducted.

The EC does not dictate to the papers chair how to run the review process, but clearly approving who the papers chair is and determining when and where SIGGRAPH is held and what other events SIGGRAPH sponsors are important influences in determining the opportunities for all of us to publish and present our work.

Edited by author 07-27-2006 11:06 PM
A few comments and a proposal:

1. The reviewing situation is bad in other fields --- the number of submissions has been growing exponentially, while the number of "good reviewers" (i.e., the number of people you feel comfortable assigning reviews/editorial positions to) grows linearly. Although it seems to have stabilized. The reviews I've seen in vision and learning conferences have been very mixed in quality, but the average vision conference review is often much worse than the average SIGGRAPH review. On the other hand, the average vision *journal* review is of *much* higher quality than the average SIGGRAPH review.

2. It is a very human process (as, I think, Marc said), and most people I've seen really try to do their best. I think the right decision is being made in the majority of cases.

3. There's a big cultural problem. I "grew up" as a graphics researcher, and learned that all that matters in a paper is how good it looks; I didn't learn much about science. Reviews are often about feelings; "I do/don't like this paper," is a common sentiment. As I've been exposed to other fields, SIGGRAPH sometimes seems more like an art gallery with equations than a scientific conference. As a student, I never submitted a journal paper, and nobody ever explained to me the importance of journals in other fields. I never read really scientific papers (except as my interests grew to other areas), and never considered myself a scientist. I suspect that a lot of other people in graphics are in a similar situation.


4. Here's my proposal for how to fix the system:

* eliminate the main SIGGRAPH papers program

* add a collection of collocated conferences on topics that roughly span the field: SCA, NPAR, Hardware, Games, SGP, rendering, vision/graphics, etc. Combine logistics, so that attendees pay one fee to attend all conferences, but keep reviewing separate. Create a process that makes it relatively easy to add conferences as necessary.

* optionally, have another set of collocated conferences in the Winter.

There are many advantages: the quasi-journal SIGGRAPH track is eliminated, at which point the conferences will become more selective, but - we hope - not SIGGRAPH selective. There will be some incentive to write journal papers. The committee meetings become smaller and more focused. There is less concern about one community being less fair than another (they may remain different from each other, but I don't think that's as bad as having different SIGGRAPH areas seen as having different standards).

The disadvantage, obviously, is that we lose the showcase papers track, and that there's more possibility for good - but uncategorizable - work to get lost between the cracks. As long as everything is flexibile and co-located, I'm not too concerned about these things.

Unlike Aaron I "grew up" outside of graphics, and so mainly read a lot of papers from other fields (mainly engineering.) A lot of my work has been with people from outside of computer graphics. Papers and reviews in other fields are all over the map in quality. I find people from other fields have a lot of respect and enthusiasm for the algorithms we develop, and the new ways to visualize data and solve problems with images. I see graphics papers referenced in journals as diverse as the "Journal of the Optical Society", the "Engineering Analysis with Boundary Elements", the "International Journal of Heavy Vehicle Systems" and the "International Journal of Diabetes and Metabolism" (just a quick check of some of my own papers on Scopus -- other people have been much more influential.) Early papers in radiosity used to always reference texts like Siegel and Howell, now Siegel and Howell reference us back. A lot of excellent contributions have been made by computer graphics researchers, assisted by the reviewers that help get their manuscripts prepared for publication. I think it is great to discuss how we can do things better, but I think we should remember that we have a lot of good work to build on.

It's encouraging to know that there are people like Michael who are willing to say something about problems like this. I'm an undergrad CS student who's dream is to someday become a big graphics researcher. I've been writing graphics code since I was 14 and I wrote my first GI ray tracer just two years later (man, those noisy caustics look so cool when you're 16!). Ever since then, I have always wanted to make a serious contribution to the field. After reading what Michael has to say about the current state of CG research, the future looks bleak for my dreams of becomming more than an amateur researcher. I can only hope that in the years to come this can be changed.

Michael: Rock, rock on.

Being someone totally outside this field (from computer networks), it was very interesting to read Michael's writeup on the state of SIGGRAPH. I couldn't help but wonder how replacing the word SIGGRAPH with SIGCOMM in Michael's writeup would give an (almost) accurate description of the state of affairs in networking community - one of old boys exclusive club. Perhaps the same applies to other SIG*s or their equivalents in other CS areas. Perhaps this is now a wide enough problem in Computer Science community that any solutions you SIGGRAPH folks come up with would greatly benefit the entire CS community.

I urge Michael to leave this message board up (and also admire his courage for doing
this). Or transition the board to someone else if he receives too much pressure.

This is the first time we have had an open, public discussion about the review process in
our field. Note that, it is an important statement.

These are things that have been said time and again at dinner meetings around the world
every March or April, but with the common understanding that none of it can be said in public.
Let it be said in public. SIGGRAPH will only improve from some self criticism.

And the discussion is clearly having some impact, with the response from Levoy and others.
Unfortunately, I have to make note of Levoy's response. He says he thinks that the review
system "is not broken".

Although it may be no more broken than some other conferences, this is a disappointing response
from him. The impact of its existing problems can be larger, and they impact us.

So despite whatever the intent is, Levoy's statement just makes me believe that the actual effect
of his BOF will be to pull the discussion away from this open public forum to a quiet room
where it will not be recorded, and were people cannot speak freely, and where the discussion
can be generally ignored.

So again, this message board has a major value for the community.

You are making this harder than it needs to be. There is one simple secret to
getting in to the papers program, and that is to pick a topic that no one has
interested in, and then make a really polished paper and video about it.

Here is a practical step-by-step formula for easy paper acceptance:

1. Scan CVPR, ECCV/ICCV, or NIPS for the algorithms they use. Pick one that
the SIGGRAPH audience has not heard of yet. This is not too difficult, CVPR
is huge.

2. (This is the creative step)
INVENT A "CUTE PROBLEM" THAT IS HAS NO PRACTICAL VALUE. It does not even need
to be a problem in graphics. But if you choose an area in graphics that has
possible practical value, the reviewers will kill it in a second. We're smart
and can easily detect any true relevance, and we want to save those problems
for ourself.

Example: do texture synthesis on some sort of spline or manifold that is
little used. Apply someone's vision segmentation code to make a system that
would allow even your grandmother to make a video of a virtual fishtank from
clips downloaded from the web. Build a clever device for capturing the BRDF
of some phenomenon that is of little interest.

I'm not going to give you any better examples, because I have to admit that
this step is not trivial. But by definition it is not as difficult as getting
a relevant paper accepted to SIGGRAPH, since that is impossible.

3. Download the Matlab code from the identified CVPR or NIPS papers.

4. Add a scale parameter. Now you have an innovative new algorithm.

5. Apply it to the bunny and the dragon.

6. Put ATTITUDE into the video narration. The tone of voice in the video
should convey how both clever you were to download that Matlab code, and how
effortless it was. Make it seem so simple that anyone can do it, yet so
clever that only a SIGGRAPH author could have thought of it.

I have had papers rejected, and one accepted. Trust me, it does work.

As someone who has had one siggraph paper accepted and several rejected, I've read this forum with great interest. I have a couple of things to offer to what's already been said.

The biggest problem with siggraph is that given the amount of material submitted, the time alloted for the review process is disproportionate to the reputation of the conference, and encourages careless reviewing. A reviewer often reviews more than one paper, and there's simply not time to try to test any of the ideas to see if they work. There are several notable examples of siggraph papers that don't work, as anyone who's ever assigned a new graduate student the task of implementing a siggraph paper can attest. These "flukes" are more common than the siggraph community likes to admit. The harsh truth is that most siggraph authors probably don't want their results reproduced.

As an aside, if one thinks that testing any of the methods or algorithms in the paper is impractical, then they should be advocating for the removal of the review criterion "is this work implementable by a competent graduate student?". Why does anyone care? Most graphics is engineering, some is science. A good engineering job may not be implementable by a competent graduate student. A good engineering job may require a team.

Prof. Levoy remarks that he intends to bring the tertiary reviewers in on the rebuttal process. I think that's a good suggestion. Another good suggestion is that if the committee asks for additional reviews, as often seems to be the case among the program committee, then those additional reviews need to write a review that the author can see. The primary/secondary paragraph summarizing the findings of the committee is insufficient to give exposure to the process that ultimately accepted or (more commonly) rejected a paper. Also, Prof. Levoy remarks that he will implement new measures regarding papers with high variances in their scores. High variances are not always bad, and in fact often indicate that there is disagreement in the community on some issue. Those papers ought to be published, and squashing dissenting opinions doesn't do any community any good. Prof. Rushmeier remarks that outsiders think that siggraph reviews are often more thorough than in other areas of computer science. That may be true for conferences but is not true for many journals in other areas of computer science. Such reviews are often more insightful than any siggraph review I've received, simply because (I think) that the time pressure is not as constraining.

Heard about the forum on the floor today. Levoy's session might help but I expect it'll mostly be people who've succeeded in the system protesting that minor changs might be good but the system is sound.

The posts so far have ignored the "Glam" factor in siggraph, a thing I think broken. Papers are accepted/rejected on their images or videos. I heard a senior member of the papers committee say one year "the video is good so we have to figure out a way to get the paper in." The paper, needless to say, has never been reproduced becaus the intense amount of handholding and manual labour required to get the technique to work. I've also seen papers that represented outstanding engineering efforts rejected because while the authors did a reasonable job with their video, they didn't spend a couple of weeks tweaking Renderman to get the best possible results out. "Find a reason to reject" prevailing in these cases. I don't expect that will change about siggraph but they ought to be upfront about it. Requiring more disclosure of how the results are generated would be good. Special evaluation of the video could be included in the reviews. And its daft to penalize those authors who hold their papers so that they can polish their videos as some hints in this forum have suggested.

John Hart's comments seem off the mark, too. Graphics is not healthy, siggraph is not healthy, and supporting elitism and not repairing siggraph means that in the long run you'll only be tenuring 18% percent of your academic-minded graduate students. We already see that in the last few years.

Edited by author 08-02-2006 01:14 AM
I have some suggestions to address some of the current problems with the SIGGRAPH review process.

1) Eliminate the physical meeting of the SIGGRAPH papers committee.
2) Replace the committee meeting with an online discussion among all the reviewers of the paper (not just the committee members).
3) The job of the senior reviewer on a paper is try to generate consensus. That may ultimately require more reviews. Because there is not the intense time pressure of an in-person committee meeting, these reviewers should be best experts on the subject matter, not necessarily other committee members.
4) A job of the papers chair is ultimately to make sure that each decision is made in a reasonable fashion, as indicated by the reviews and subsequent discussion. It's a big responsibility, but it is essential. At least she should have sound information to either approve of the senior reviewer's recommendation or the necessary ammunition in writing to overrule.

This process has a number of desireable properties.

A) All discussion is in writing and subject to the scutiny of a number of people.
B) All reviewers can see and participate in the whole review process.
C) We are now free to turn the often-present criticality on each other as reviewers and not just as paper authors. So if I see that reviewer #2 wrote a really sketchy review, I can say so, and suggest either that he go into more depth or perhaps that we need another review.

This process has worked well for at least the past 2 meetings of the ACM Symposium on Interactive 3D Graphics and Games (http://www.i3dsymposium.org -- sorry for the shameless plug). And what's more, saves travel time and money for the committee. It may seem a large departure from the historical SIGGRAPH review process, but I don't think the change is prohibitively large.

This will not fix all problems of how papers in our field are reviewed and what the standards should be, but at least it puts it all more in the open. Also note that property C has the flaw that if I am Joe Schmo assistant professor and reviewer #2 is graphics pioneer god-incarnate, I may be reluctant to speak up.

Well, that's all I've got tonight.

I knew this forum from siggraph yesterday. I just read most of existing posts here, and want to add two points (could be wrong, just for discussions).

(1) Some posts mentioned "siggraph put too much weight on superficial video/images". I agree! It is true for recent siggraph, but not true for siggraph long time ago. There are various possible reasons. The first possible reason is that traditional graphics runs out of topics and need to bring new things from other fields. Vision certainly is one option, But i feel it is going too far now. You have to twist video to get siggraph papers in. Siggraph is quickly becoming just another CVPR/ICCV style conference which publishes hundreds of papers but the generality of most of the works are weak (i.e. only work on several intentionally selected video/images/models).

The second possible reason is that siggraph paper committees who have super power on paper selections are over visionized. Just check the composition of recent siggraph paper committees, and you will find the percentage of vision experts in siggraph paper committees. So, it is not a surprise that more and more boring vision/image papers got in.

(2) Online discussion among reviewers are certainly better than in-person PC meeting, which has been successfully used for many other graphics conferences (except siggraph) for years. I am very surprised to see the siggraph is just indifferent to it. As pointed out by other posts, the similar thing happens to TOG that does not show the name of assigned editor to authors, while TVCG certainly is taking the lead. I wish it is not because some powered people want to keep their hidden (thus safe) super power on paper selection of top confs/journals. If so, it is really bad to the whole community.

I believe the review problem in our field is predictable.

In a mature field,

  for any topic, there are several groups working on it, and

  every new development is necessarily "incremental", because
  the big new ideas were discovered long ago.

This means that:

  your paper will be sent to your competitors,

  who can trivially reject it because it is indeed incremental.

This also means that papers that are well constructed but
do not mean much to anyone (as a previous post indicated)
will have more success. And the opposite, Papers that
have a bit of new idea will have a particularly difficult time.

Despite all this, papers do get accepted. Unless there
is this body of great unpublished papers out there?

In response to the claim that acceptance depends largely on visual appearance and less on scientific content, I propose a control experiments: In parallel to the committee, form a committee of visual arts people (from academia and graduate students), tell them how many papers to accept from each category and watch which papers they “accept”. These people should have internet access like the reviewers, but be people who don’t know computer graphics.

Another idea: do the review process in two phases. In the first phase, reviewers get the paper with all figures removed (without videos), and review based the text only (it might be needed to ask authors to submit text-only version). Papers that survive this phase go through a second phase which considers the graphical content. The idea is that a text-only version requires a more scientific job both from authors and reviewers.

I am sorry to have missed the BOF on this topic - I heard about it too late and had other committments.

I should note that, as someone who works in the visual effects industry, I have no career-oriented need to publish anything I do. However, I do want to give papers/sketches at siggraph because two-way sharing of technical information improves what I do, and because it is intellectually satisfying when my research grows with the help of others.

The early note in this forum about the early history of siggraph was very helpful. For a fledgling field of study, it makes sense to build up a forum with a rigorous process that can build respect from the larger academic community. That approach has been wildly successful. The graphics community is now a large academic field, but also drives several large commercial industries (e.g. visual effects and graphics cards).

And now the community is VERY different from the audience it began with. On the technical side, my experience is that visual effects companies are conducting basic graphics research that is at least as good as the academic community. And although that research is closely held for a period of time, it eventually can be published and/or openly discussed. Siggraph is the natural forum, but we have all been reading about the barrier to entry. And since publishing has a more altruistic drive than in academia, going through this broken review process is not worth the effort.

While siggraph is very aware and sensitive to the non-academic community members when it comes to accepting registration money, we are very underrepresented in the review and organizational process. I have reviewed 2-3 papers every year for the past 5 years, and have asked to be on the review committee so that I could better understand the baffling results I have witnessed, but there does not seem to be any interest from papers chairs. This lack of representation substantially limits siggraph's actual value and potential.

I beileive the goal siggraph should adopt is to provide a forum for presenting as many ideas and concepts as possible. This not only better serves the rapided improvement of graphics research, it also is more in line with the needs of the larger (non-academic) graphics community that supports it. I am not sure why, but the focus in this forum has been on the details of how to implement a better review process. No one has addressed the idea suggested twice by Michael to operate the conference more like scientific conferences: completely eliminate the review process. I used to attend quite a few engineering and scientific conferences, some smaller and some much larger than siggraph, in which the process required you to send in an abstract, and that assured that you would recieve 15-20 minutes to present your ideas. Yes, some turkeys do get in, but the community is capable of deciding for itself - a review committee is unneeded for this. And I dont think the current system in entirely turkey-free anyway. So longer term, I think siggraph and graphics in general will be more healthy eliminating the review process entirely. Michael was correct in his original note, the physics community uses the arXive for distributing papers, leaving reviews for publications that take place well after the community sees the work and comments on it. That is a much more sane and efficient process.

Conversations I had after the BOF confirm what I was afraid of: there is strong resistance to changing the process because tenure positons are on the line. That rationale is illogical from my point of view. For one thing, that means that all of the graphics community must be suppressed by the needs of the few people who have tenure issues to attend to. The tenure process functions just fine in other academic communities that dont have a peer-reviewed conference. For them, peer-review happens in journals. Conferences are an opportunity to learn about the ideas of others and share your ideas. And that is an important contributor to the health of those communities. It would be a shame if, in a misguided interest in a flawed form of tenure seeking, graphics is held back. And it is being held back by the current process.

Certainly we cannot switch to a non-reviewed system overnight, but it should be entirely workable to decide to do so in 2010, for instance. That would give the academic community time to switch over to a journal based (or some other mechanism) approach for their needs.

I certainly look forward to seeing what comes from this discussion and any changes that might be made to the system. I am always optomistic that things will get better. I am writing this on the last day of siggraph, and I was quite discouraged after seeing a lot of talks with severe flaws in their reasoning and/or technical implementation, because we are told that they are the best the graphics community has to offer. I know that is not true because I have seen much better, so I am optomistic that changes will make siggraph better.

There have been very good suggestions in this discussion board.

1. Have online discussion system for even tertiary reviewers to participate. Primary reviewers should take this discussion seriously and respect the consensus among the reviewers that is evolving in the discussion board.

2. There were suggestions to avoid submissions from the program committee, and understandable reluctance (difficult to staff) to do that. There was also a suggestion to have two SIGGRAPH conferences per year. These are all good suggestions and reasonable concerns. I heard that Eurographics is going to shift its submission date to October 1, and move the conference early (During this transition, there are going to be two Eurographics deadlines in 2007). If we are reluctant to have two Siggraph conferences and to prevent the PC to submit papers, we can make use of this opportunity to encourage Eurographics to become a quality alternate venue, so that PC memebers need not wait for a year to submit their work, but just submit it to Eurographics. If possible, we can encourage EG to time their conference appropriately to have close to 6 months spacing from Siggraph.

3. Third suggestion is to make the author names blind even to primary reviewers. It is almost a mockery fo the "blind" reviewing process if the most important person who makes the final decision about the paper knows the author names. This can possibly be done by pseudo randomization of the paper numbering, mapping of which is known only to the Papers Chair (or only to Price Water House Cooper ;) )

4. Have no limit on acceptance rate. If a paper is good, it should be accepted. Although it is made to believe that there has been no fixed "acceptance rate" enforced during PC meeting, it is hard to believe that there can be so many rejects especially when our community knows the standard of SIGGRAPH and makes all effort to submit only highly polished papers. My guess would be around 30% of the submitted papers would be of very good quality.

If these suggestions are implemented, I strongly believe that the hard work put in by the PC, tertiary reviewers, and not to mention the authors will be well-rewarded.

Having said that, I would say around 50% of papers in this siggraph were of very high quality, and the rest were mostly pure hacks and problem specific fine-tuned implementation. Unfortunately, this is the randomness in the selection process that has creeped in due to low acceptance rates, and also our fatal attraction to flashy images and videos.

Looking forward to Siggraph 2007! Good luck Marc! You are in an unenviable position that invites too much scrutiny.

3. Third suggestion is to make the author names blind even to primary reviewers. It is almost a mockery fo the "blind" reviewing process if the most important person who makes the final decision about the paper knows the author names. This can possibly be done by pseudo randomization of the paper numbering, mapping of which is known only to the Papers Chair (or only to Price Water House Cooper ;) )

It wouldn't be too hard to increase the anonymity of the process with a little cleverness and a couple simple black boxes. For instance, suppose the authors' names were unknown to the primary reviewers. They would choose a reviewer and give their name to a black box, which would complain only if the name matched one of the authors' names. Otherwise the black box would tell the primary that the proposed reviewer name is acceptable. Of course, they might get (un)lucky and pick an author as a reviewer.

A more robust scheme would be: a primary reviewer selects a large number (~5-10) of potential reviewers, and the black box chooses a subset of reviewers which are not authors. There's some small chance that all of the proposed reviewers are authors, but on average this procedure would improve anonymity.

These black boxes could be implemented with a simple perl script or even a human moderator.

Is it viable for Siggraph to publish the reviews, as they do in other venues? Maybe not in print, but certainly online. This may induce some extra responsibility when submitting reviews and, in the long run, it should improve the review process. Any thoughts on this?

I attended the Town Hall meeting in SIGGRAPH. Most discussion there were about reviewing processing. However, all technical measures to improve the review process will not solve the problem. The fundermental problem we are facing is that SIGGRAPH is THE ONLY dominate conference in computer graphics. That is the most dangerous thing.

I was also at the Town Hall meeting, and thought the idea of making SIGGRAPH more like CVPR made a lot of sense. Accept more papers, but only orally present the best papers (present the rest as posters), hopefully this way papers with less polished results but with interesting ideas can get in more often. If these papers lead to polished work, they could then be published in TOG or other journals. Perhaps the orally presented work at SIGGRAPH could also get automatic invitations to be published in TOG, let TOG be the end of a research project instead of SIGGRAPH. The longer review process of TOG should be able to reduce much of the noise generated at the SIGGRAPH review process.

After very hard work, I got a paper in this year. The natural thing was to work VERY hard on the talk because so many important people would see it. AFTER my talk (which went well due to A LOT OF WORK), I found out that almost nobody saw it from the audience I had aimed for. Ironically, few relevant people saw it because SIGGRAPH is so big (other concurrent sessions). So at some point, a bigger conference means a smaller relevant audience. :(

I am very glad to see all of the postings on this topic. I find that the timing of this discussion is absolutely wonderful - as Siggraph 2006 seems to be the apogee of the problem. I attended the conference, hoping for the usual burst of inspiration. Unfortunately, I was sorely disappointed. And even though what I say here is just my opinion, everyone who I have talked with after the conference seemed to think that unfortunately this year's Siggraph was paltry in comparison in terms of papers (and even sketches, to some degree).

Is this the graphics conference that it claimed to be? It's hard to imagine. Now there are papers on sound, perception, human interfaces, haptics (which were present for many years), image processing. Look at the percentage of actual graphics papers that are currently published at Siggraph? This year it seems to be an image processing conference. And how many papers make a claim to be "real-time" and applicable to interactive applications while in reality using "practical" amounts of memory such as 4GB of textures or a 600MB database of small data to lerp from for real-time performance?! It's absolutely ridiculous. To a degree,I have to tip my hat to Jonathan Cohen - because in the last two years I3D has really become the conference to watch for releveant _graphics_ papers. I find that if one is looking for good, applicable, implementable rendering techniques, then one shouldn't look at Siggraph papers. Instead, we turn to I3D or Eurographics (which has its own set of problems).

Siggraph has been a great source of inspiration for many years. But having been through the process myself and having witnessed people submitting their work it seems that the process is indeed very broken. I personally like the suggestion of building up other conferences meanwhile. And the suggestion of refusing paper submissions from committee members seems reasonable - specifically as a method to build up other conferences.