Judging Guide

This page is for judges and administrators, not for contestants. It is on the public contest site, so it describes how the system works and how to grade; it holds nothing a team could use to get ahead. The rules and instructions are what teams see, and the Scoring section of the rules is the policy this page puts into practice.

Logging in

• Your username is your email address. You are emailed a link to set your password along with a code to type at /verify if your mail system mangles the link. Both work once. The invite expires in 2 weeks or at the end of the contest; contact the head judge for a reset.
• If the invite doesn’t arrive (check spam first), ask the administrator to resend the invite.

The Submissions page

Two tabs:

• Ready for Evaluation: submissions no one has scored, newest first.
• Evaluated Submissions: scored ones, along with time and who did the scoring.

Both lists can be grouped By Problem or By Team. They can filtered by problem, team and language. The lists update on their own when teams submit or other judges finish; if the page ever says it couldn’t load the submissions, reload it.

Teams submit .py, .cpp, .c or .java files of up to 1 MB. The practice problem, typically Hello World, is graded automatically by the system. You will never see those submissions.

Grading a submission

1. Press Score on a submission. This checks it out to you: other judges now see it as Checked Out and are steered away from it.
2. In the window that opens, press Download to save the source file under the name the team gave it. Code Preview shows the code in the window for a quick look; run the downloaded file to judge it.
3. Run it against the test cases for the problem and decide on a verdict (see How to decide). It can be helpful to set up a easily-accessible folder containing the test files so you can drop them into the evaluation folder when needed. It is fine to add additional test cases; you can be useful to share them with other judges.
4. Write feedback in the box (an editor with a Preview tab; Markdown works). If the submission has been graded before, Feedback History shows the earlier feedback. Teams do appreciate including givs to inject a bit of humor or excitement, but use these only if you are confident in how they will be interpreted.
5. Press one of:
   • Pass: the solution is correct
   • Fail: the solution is not correct; include a brief note explaining why or giving describing a failed test case
   • Ignore: the submission shouldn’t count either way (see below); the team is told it was ignored
   • Cancel (or the X): closes the window without grading, releasing the checkout for others

The team sees a green check or a red circle next to the submission, and can open your feedback. They see feedback only once you press a verdict button.

Whatever you’ve typed is kept in your browser if you close the window before choosing a verdict, so you won’t lose it by accident.

Checkouts

A checkout is a courtesy so two judges don’t grade the same submission. It is not a lock.

• A checkout you never finish is released automatically after 30 minutes. There is a warning when there is just 10 minutes left in the checkout period.
• If you reload the page or get pulled away mid-grade, the submission shows Continue on your list; press it to pick up where you were, with your feedback still in the box.
• If someone else gets there first, you’ll see “This submission is already being graded.” That is normal at busy times; pick another. If two judges do finish the same submission at the same moment, the system accepts the first verdict and refuses the second, so a submission can’t be scored twice.
• If a submission is stuck on someone who walked away, ask an administrator to press Free next to it.
• If saving your verdict fails (a network problem, say), a red message says so and the submission stays checked out to you. Press Continue and try again; your feedback is put back in the box.

Changing a verdict

On the Evaluated Submissions tab, Re-Score reopens a graded submission and lets you pick a new verdict. The team’s points are corrected automatically. The penalty is returned or applied, and points for a problem are added or removed as needed. Earlier feedback stays in the history.

How to decide

The rules the teams read say:

• Run the submitted solution with various test cases. If it produces the expected results, mark it Pass; otherwise Fail with a brief description of why it failed.
• A problem may state a time limit. Judges may terminate a solution that looks very inefficient or stuck in an infinite loop; mark it Fail.
• Teams are told that Ignore is for when there is a problem with the application, or a duplicated submission: the team is not penalized. Teams are told this, so keep it for those cases rather than using it to postpone a hard call.
• Keep feedback to what the team needs to fix the problem. Teams may resubmit as often as they like, and a clear but brief note is the most useful thing you can give them.
• Questions from teams reach you through floor assistants, who message the head judge. Answers that apply to everyone are announced on the contest site.

The usual practice is to use whatever tools you are most familiar with to build and run the programs. If a solution does not build, check that you do not have a simple incompatibility. If it is simple to get a program to build, strongly consider just making the change and rebuilding. For example, BlueJ does not require Java programs to contain static void main; you may have to provide those. Discuss other exceptions with other judges.

Another common problem is that teams often hard-code filenames and paths to open test files. Modify the code (or the test file names) to ensure the programs do run. Being more restrictve is useful in many contexts, but here.

Scoring

• A Pass earns the problem’s points, once. If the team then resubmits the problem, additonal pass verdicts do not add to the score.
• Every Fail costs 3 points, whether or not the team has since solved the problem. So a wrong call hurts: test carefully, and use Re-Score if you find you got one wrong.
• Ignore has no impact on the score.
• The scoreboard breaks ties by the time of each team’s last correct submission
  • the time the team submitted it, not when it was graded - so a slow queue doesn’t cost anyone. If there is still a tie after considering submission time, the head judge will organize a tie-breaker based on solution elegance.
• Standings read “Scoring in Progress” while submissions are waiting, then “Unofficial Scores” once everything is graded. The administrator marks them Final afterwards.

Data files

Some problems come with data files that teams download. What teams see and experience:

• On the problems page, students see a title for each problem, the description if it’s provided, and a submit box to submit solutions. If there are data files for a problem, they appear in an expandable box.
• Each data file has a download button that will download it using CRLF line endings. (That’s an improvement over the old way; the software enforces uniformity.) There’s also a view button that could be used to copy/paste the text, but students are advised to use the download. Copying from the view button gives the line endings of the team’s own computer, which can differ from the downloaded file.
• Data files are restricted to ASCII text.
• Data file names are enforced to be unique across all problems. My assumption is that they would involve a problem number like “prob3a.txt” or similar. There is no requirement that they have particular filename extensions.
• A data file can be shared across problems. This is essentially built for supplying a dictionary that might be used by multiple problems.
• We will be able to update data files during the contest and even provide new ones. We wouldn’t want to do that often, but it’ll help if students are making common mistakes and we decide to give them more help. Teams aren’t notified when a file changes, so the administrator posts an announcement saying which one did.
• They will be able to see the data files when the contest opens, at the same time they can see the problems themselves.

If you find a data file is wrong, tell the administrator right away, as a team that already downloaded the old file has to download it again.

Talking to other staff

Chat (top menu) has three channels: Everyone (all staff), Judges (judges and administrators) and Logistics (floor assistants and administrators - judges don’t see it). Each post starts a conversation; use Respond to reply. A badge on Chat shows unread messages. Floor assistants forward anything that needs a judge’s call to the Judges channel, so keep an eye on it.

Administrators can send an Announcement to every team; ask one if an answer applies to everyone.

When something looks wrong