Please assign each paper you have read two values:  a "score" and a
"confidence."

The scores are INTEGERS (no fractions, please) in the range 1 to 10
(inclusive), and have the following interpretations:

     1     -- Absolutely unacceptable, incorrect, etc.
     2     -- Very weak, definitely below conference quality.
     3     -- Paper may have some merits,
              but doesn't meet conference standards of quality.
     4     -- Paper has some merits, suitable for a journal
              or specialized conference/workshop, but probably
              doesn't meet POPL conference standards of quality
     5     -- Neutral score, could be accepted
     6     -- Certainly above threshold, but only a marginal acceptance
     7     -- Typical POPL paper.
     8     -- Very good, among the 25 best in my memory.
     9     -- Outstanding paper, among the 10 best in my memory.
    10     -- Breakthrough, best paper.

Since you will each review only about 3/13th's of the papers you should
expect to award relatively few 8's and 9's.  If you are like me, you will
have a great temptation to give a lot of 5 and 6 scores.  This is OK, but
in such cases please take a bit of extra time and see if you can make a
case for a higher or lower score.  Having a large number of papers with 5
and 6 scores will likely lead to a long and difficult PC meeting.

The confidence scores are also INTEGERS in the range 0 to 4 (inclusive),
and have the following interpretations:

     0     -- I didn't read the paper, or I have no opinion, or I don't
	      have a clue.  Pass.
     1     -- I have some idea of what this paper is about, but I'm
              not all that confident of my judgment on it.  Don't quote me.
     2     -- I am reasonably well-informed about the subject of this
	      paper, and have read the paper closely enough to have some
	      confidence in my judgment.
     3     -- I fully understand the paper, but I don't know all of the
	      relevant literature; it's possible this is not a new idea.
     4     -- Consider me an "expert" on this paper.  I understand it in
              detail.


For each paper you review, I would like you to fill out the following
electronic score sheet:


*
Paper #:
Title:
Authors:
Overall Evaluation:
Confidence:
PC Member Name:
Referee Name:

Justification:

Comments to Authors:

Comments to PC:
*

^
|
-------- important separator between each review


I will attempt to process the reviews automatically.  (Note that the "*" is
an important separator between reviews.)  Each section label (e.g.
"Comments to Authors:") should appear literally at the beginning of a line.
Sections may be empty or extend across multiple lines.  Please don't omit
sections.  You need to fill out the referee name field only if you use an
external referee.  I intend to send the Justification and Comments to
Authors to the authors.  The Comments to the PC will be for use by the PC
only.

What follows now is fairly standard advice on reviewing conference papers.
I've modified this from recent advice given to POPL and LICS program
committees:

When grading a paper, I ask you to address the following (interrelated)
issues.  A paper with high grade should score high on several of the
parameters below.  Please think of what (in terms of these parameters)
makes a paper, say below POPL quality, average/excellent POPL paper.  As
you evaluate the criteria below, remember that POPL stands for *Principles*
of Programming Languages.  You should give preference to papers who
contribute to our understanding of fundamental principles of programming
and programming languages over papers with greater technical depth but less
relevance to basic principles.


FOUNDATIONAL/CONCEPTUAL CONTRIBUTION:  Note things like a new model,
new notion, new definition, new approach.  Note the significance and
reasons for this novelty (and note the absence of such a novelty!).

TECHNICAL DEVELOPMENT: Does the paper make a
- Introduction of a new idea
- Novel use of known ideas
- Talented use of known ideas
- Traditional use of known ideas
- Trivial use of technical ideas

RELATION TO IMPORTANT PROBLEMS: Does the paper solve completely/partially
address an open question? or provide a theoretically sound approach to
solving it? or a demonstration of a practical solution?  How important is
this problem?  (Central/ important/interesting/legitimate/stupid).  How
much effort has been invested in solving it and by whom?

SOCIAL INTEREST IN PAPER: Is it potentially interesting to the whole
community, to a major field, to everyone in a restricted area, interesting
only to the authors.

QUALITY OF PRESENTATION: Is the paper well-written, and likely to form the
basis of an interesting presentation which will enhance the conference?

HOW WILL IT CONTRIBUTE: Fertilization, satisfy curiosity, who knows?

PAPER TYPE: Is it a
- First step (opens a new area)
- Last step (closes an important area)
- Giant step (makes essential progress)
- None of the above.

WILL I CHAMPION THE PAPER?: Are you willing to fight for the acceptance.
You need to have at least confidence 3 in your opinion for that.  It will
be nice if every paper we eventually accept has at least one champion.
Note that a paper with just a single champion will likely be accepted, even
if the rest of committee disagrees.