Computational Wisdom of Crowds: Algorithm Development & Evaluation

This is an ongoing project.

We are now reviewing applications received for the Spring 2015 position; no further applications are requested at this time.

Work with us to build and benchmark effective learning algorithms to advance the state-of-the-art for crowdsourcing and "human computation". You will have the opportunity to participate in our active research and contribute to our ongoing efforts. The project typically has 4-5 graduate students to interact with, and a shared lab space with some seating available.

Crowdsourcing involves outsourcing of tasks to a large group of people instead of assigning such tasks to an in-house employee or contractor. While we have successfully automated many routine tasks, human competency still exceeds automated algorithms for many other, more complex processing tasks, such as analyzing text or imagery. Today’s Internet-based access to 24/7 online human crowds has led to the advent of crowdsourcing and a renaissance of research in using human computers once more. These new opportunities have brought a disruptive shift to research and practice for how we build intelligent systems today. On one hand, labeled data for training and evaluation can be collected faster, cheaper, and easier than ever before. While traditional scarcity of labeled data has helped to drive research on unsupervised and semi-supervised methods, strategic use of crowdsourcing now allows us to collect the labels needed on demand and at scale. With access to a human crowd “on-call” whenever the system has a question, there is tremendous potential for intelligent systems to enjoy constant, lifetime learning.

In addition to collecting labeled data, human computation is also being increasingly integrated into intelligent systems themselves, operating in concert with AI. While AI accuracies will certainly continue to improve, use of human computation in concert with automation lets us achieve greater capabilities today using hybrid systems. For example, better assistance can be provided to visually and aurally impaired persons by leveraging superior capabilities of humans to make sense of images and sound.


Must be able to program, preferably in Java, scala, or clojure, and be comfortable with basic math and statistics.

Project Timeline

Project running through Spring 2018 at least.


Work with graduate students and/or professor to implement and benchmark statistical algorithms. Expected to maintain weekly contact. Hours can be varied, depending on time available, but must commit to at least one semester of work to justify our time teaching and training you. Depending on prior skill and experience, degree of supervision and sophistication tasks will be tailored accordingly. You will be expected to document your research activities in writing, with the opportunity to publish a technical report or research at the end.


The Office of Undergraduate Research recommends that you attend an info session or advising before contacting faculty members or project contacts about research opportunities. We'll cover the steps to get involved, tips for contacting faculty, funding possibilities, and options for course credit. Once you have attended an Office of Undergraduate Research info session or spoken to an advisor, you can use the "Who to contact" details for this project to get in touch with the project leader and express your interest in getting involved.