A new online grading tool is now available to simplify the process of marking paper assignments at the university level. The idea for the tool, named Crowdmark, came to founder James Colliander in 2011 in the wake of the Canadian Open Mathematics Challenge. Four thousand eight hundred forty nine people wrote the 16-page exam, which arrived from couriers all over the world.
“It was a logistical nightmare. During the process, I saw faculty waiting for exams, while others were marking sections A and sections B. It was a serialization bottleneck in the workflow. And I had this very simple idea. If I could upload every page into a database, I could show the right page to a person at the right time, and I could do so in a parallel fashion,” said Colliander.
Since then, Colliander has taken the simple idea and greatly expanded it. One of the benefits of the program is that it allows markers to work from remote locations. “I can recruit someone from Indiana, who’s really good at particle physics, to assess a physics question on a big physics exam,” said Colliander. This creates an environment where merit is valued over proximity, and the most skilled markers can be assigned to work that they would normally not have access to.
Having people not work together in person can present the problem of poor communication between markers. To combat this issue, Crowdmark features “evaluation multiplicity,” which allows for any number of markers to evaluate the exact same assignment. “Suppose me and you are marking the same paper, you gave it a four, and I gave it a seven. That’s too far apart and it generates a flag. A supervisor marks this exact same paper and gives it a three. You’re within tolerance, so you get a +1 in your trust, and I don’t,” explains Colliander. Over time, the evaluation multiplicity method will generate a badging system on a marker-by-marker basis.
Crowdmark also attempts to ease the process of running massive online courses. Of the around 1000 public high schools in Ontario, roughly 300 of them allow twelfth grade students to earn university credits through the Advanced Placement (AP) or the International Baccalaureate (IB) programs. Crowdmark aims to streamline the marking of exams in these programs, allowing instructors to design fair and appropriate courses that reflect what is expected at the post-secondary level.
At the end of AP or IB courses, students take an exam that must be paid for; by having Crowdmark keep these papers together, a fair price can be determined. Colliander hopes to see a system whereby high school students can earn credits that have been verified through Crowdmark; he believes that such a system would ultimately save studets money. If this system can be applied to more disciplines, it is possible students can be prepared to graduate out of university in three years, instead of four.
To date, U of T has invested $115,000 in the project, which is also funded by the province of Ontario, and MaRS Innovation.
