I teach courses involving programming, but I don’t have a good way of collecting and evaluating student submissions. I think a web application would be a great way of doing this. I’m inspired by the Project Euler and UVa Online Judge systems, which allow users to submit short programs, which are then tested and scored automatically by the system. Those sites don’t work in the educational setting, though, where instructors need to evaluate submissions by their own criteria, enforce deadlines and plagiarism policies, and protect student privacy.
Implementing and supporting a complete system with every feature on my wishlist is far more than a one-semester project. I’d expect that a student could implement a prototype, with a minimal set of essential features, as a project. Then other students could add features and improvements in the future.
I think the minimal requirements for a one-semester prototype are
- Represent the following as first-class objects: user, class, class section, assignment, submission attempt, instructor assessment.
- Roles: site administrator, instructor, student.
- Instructors create, edit, delete, and administer classes, sections, and assignments.
- Students can associate with sections, view assignments, and submit assignment attempts.
- An assignment attempt is a single source code file for the C++ language.
- Assignments may have deadlines, after which submissions may not be made.
- The system validates the source code, to sanity check that it resembles C++ source, is of an appropriate size, and is not obviously malicious.
- Code that passes validation is compiled with g++.
- Code that compiles without errors is run, and tested for correctness against a set of instructor-provided input/output text files.
- Submitted code is run in some kind of sandbox to prevent attackers from doing something like deleting the entire filesystem. A chroot or FreeBSD jail is probably the easiest way of doing this.
- Code that runs too long must be killed to prevent a denial-of-service attack from infinite loops.
- The verdict of each submission is emailed to the student and visible to the student and instructor on the website. Possible verdicts are: upload error, failed validation, did not compile, exceeded timeout, incorrect output.
- Instructors may convert verdicts to numeric scores and export them to CSV files for the purposes of grading.
- All activity should be logged to a text file.
- Data should be stored in a form (e.g. database) that can be backed up, migrated to a different host, and exported to a human-readable form.
- Language must be something that I, and a critical mass of other CSUF students, know. Preferably Python, but I’m open to Java, C++, C#, Scheme, or Haskell if there’s a compelling reason. No PHP.
- Host code in a public source code control system (e.g. SourceForge or Google Code) under an open source license.
Future projects could add other important features:
- Support for Python, and possibly other, submission languages.
- More elaborate security precautions, such as running each submission in its own virtual machine, and reporting suspected malicious submissions to the site admin.
- Support multiple C++ compilation environments (Visual Studio, Xcode, *nix g++, *nix clang).
- Synchronizing user lists with Moodle.
- Synchronizing grades with Moodle.
- Automated plagiarism detection, perhaps using MOSS.
- Use CSUF usernames/passwords for authentication.
- More elaborate submission correctness tests. Perhaps incorporate ideas from unit testing. Or allow the instructor to provide a reference implementation.
- Fuzzy output correctness testing; ignore whitespace and capitalization differences, interpret “1” and “01” as the same thing, be robust to floating point rounding errors, etc.
- Runtime correctness testing; allow the instructor to provide a timeout value for specific queries. Even better, allow them to provide a big-O time bound.
- Group submissions.
- Additional roles: grader, guest instructor, guest student.
- A failure viewer that shows students the discrepancies in their program’s output that is causing it to fail. Incorporate ideas from diff viewers such as highlighting differences and collapsing identical sections.
- Multiple-file submissions.
- Support instructor-supplied data files that are moved into each submission’s working directory.
- Output file formats other than text; especially media such as images or sound files.
- Allow assignments to require supporting materials, such as a lab report in PDF format.
- Allow instructors to clone classes or assignments.
- Allow instructors to edit assignments offline.
There are a lot of ways to go with this. But right now I have nothing, so I’m more interested in getting the ball rolling than any of the particular requirements listed above.
UPDATE (10/16/2012): I’ve learned about two similar projects:
Rather than writing something from scratch, it may make more sense to build on one of those projects.