CS 5306 / INFO 5306: Crowdsourcing and Human Computation

• Instructor: Prof. Haym Hirsh, haym.hirsh@cornell.edu, Gates 352
• TA: Eric Wang, ericwang0701@gmail.com, Gates 323
• Office Hours: TBA
• Textbook: Infotopia: How Many Minds Produce Knowledge, by Cass Sunstein
• Prerequisites: knowledge of basic computer science principles and skills (such as CS 1110, CS 1114, CS 2110, CS 3110, or equivalent)
• Class meeting time: Tuesday/Thursday 2:55pm - 4:10pm, Gates G01
• Online:
  • cs.cornell.edu/courses/cs5306
  • piazza.com/cornell/fall2017/cs5306
  • gradescope.com (entry code 9ZRR3V)
• Grade:
  • 15-20%: Prelim (tentatively Oct 17, in class)
  • 15-20%: Final exam (Dec 12, 9:00am - 11:30am)
  • 10-20%: Assignments (~5)
  • 45-55%: Course projects (2)
  • Extra credit: used if you are borderline between two grades
    • Class participation
    • Piazza participation
    • Optional lectures/activities (list will be update as the semester progresses)
      • Thursday, Aug 31, 4:15, Gates G01: Louis Hyman. The Return of The Independent Workforce: The History and The Future of Work
      • Thursday, Sep 7, 4:15, Gates G01: Henry Kautz, Mining Social Media to Improve Public Health
      • Thursday, Sep 28, 4:15, Gates G01: Michael Bernstein, TBA

• 8/22/17: Lecture 1
• 8/24/17: Lecture 2
  • Readings for next time:
    • "Judgment under Uncertainty: Heuristics and Biases”, Amos Tversky and Daniel Kahneman, Science  27 Sep 1974
    • "They saw a game; a case study." Albert H. Hastorf and Hadley Cantril, The Journal of Abnormal and Social Psychology 49, no. 1, 1954
    • "Opinions and social pressure”, Solomon E. Asch, Scientific American Nov 1955

• 8/29/17: Lecture 3
  • Homework 0: Due Thursday, 8/30
  • Readings for next time:
    • “Games with a Purpose”, Luis von Ahn, IEEE Computer, June 2006
    • “A game-theoretic analysis of games with a purpose”, S. Jain and D. Parkes, 2008.  Internet and Network Economics, pp.342-350.
• 8/31/17: Lecture 4
  • Readings for next time:
    • "A comparison of social, learning, and financial strategies on crowd engagement and output quality", Yu, L., André, P., Kittur, A. and Kraut, R., 2014, February. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing (pp. 967-978). ACM.
    • "Financial incentives and the performance of crowds", Mason, Winter, and Duncan J. Watts.  In Proceedings HComp 2009.
• 9/5/17: Lecture 5
  • Readings for next time:
    • "Turkit: human computation algorithms on mechanical turk", Little, G., Chilton, L.B., Goldman, M. and Miller, R.C., In Proceedings of the 23nd annual ACM symposium on User interface software and technology, (pp. 57-66), 2010.
    • "VoxPL: Programming with the Wisdom of the Crowd", Barowy, D.W., Berger, E.D., Goldstein, D.G. and Suri, S., In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 2347-2358).
• 9/7/17: Lecture 6
  • Readings for next time:
    • "The sum of all human knowledge”: A systematic review of scholarly research on the content of Wikipedia", Mesgari, M., Okoli, C., Mehdi, M., Nielsen, F.Å. and Lanamäki, A., 2015. Journal of the Association for Information Science and Technology, 66(2), pp.219-245.
      The goal of this reading is to get a sense of both what we know about Wikipedia (as of 2015), and also how people go about studying this.
    • Infotopia, Introduction
  • Assignment 1, Due Thursday, September 14:
    
    • Complete Cornell’s human subjects certification
      Submit a copy of the human subjects certificate (the system provides you with a pdf of this) via Gradescope.
      (If you haven't signed up for the course on Gradescope as yet, please do so using the code 9ZRR3V.)
• 9/12/17: Lecture 7: Guest lecture by Prof. Serge Belongie
  • Readings for next time:
    • Infotopia, Chapter 1
• 9/14/17: Lecture 8
  • Readings for next time:
    • Infotopia, Chapter 2.
    • "Soylent: A Word Processor with a Crowd Inside", Bernstein, M., Little, G., Miller, R.C., Hartmann, B., Ackerman, M., Karger, D.R., Crowell, D., and Panovich, K., In Proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST) 2010. 
    • "VizWiz: nearly real-time answers to visual questions.", Bigham, J.P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R.C., Miller, R., Tatarowicz, A., White, B., White, S. and Yeh, T., In Proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST) 2010. 
  • Assignment 2, Due Thursday, October 5
    
• 9/19/17: Lecture 9
  • Readings for next time:
    • Infotopia, Chapter 3.
    • "Embracing Error to Enable Rapid Crowdsourcing", Ranjay Krishna, Kenji Hata, Stephanie Chen, Joshua Kravitz, David A. Shamma, Li FeiFei, Michael Bernstein, CHI 2016.
    • "Interactive Consensus Agreement Games For Labeling Images, Upchurch, P., Sedra, D., Mullen, A., Hirsh, H. and Bala, K., Proceedings HComp, 2016.
• 9/26/17: Lecture 10
  • Readings for next time:
    • Infotopia, Chapter 4.
    • "The power of prediction markets", Mann, A., 2016. . Nature, 538(7625), p.308.
• 9/28/17: Lecture 11
  • Readings for next time:
    • Infotopia, Chapter 5.
    • "CrowdDB: answering queries with crowdsourcing", Franklin, M.J., Kossmann, D., Kraska, T., Ramesh, S. and Xin, R., In Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data (pp. 61-72). ACM.
• 10/3/17: Lecture 12
  • Readings for next time:
    • Infotopia, Chapter 6.
    • "Personalized online educational crowdsourcing challenge”, Daniel S Weld, Eytan Adar, Lydia Chilton, Raphael Hoffmann, Eric Horvitz, Mitchell Koch, James Landay, Christopher H Lin, and Mausam. In Proceedings HComp 2012.
• 10/5/17: Lecture 13
• 10/10/17: No class
• 10/12/17: Exam review
• 10/17/17: Prelim
  • Readings for next time
    • "CrowdVerge: Predicting If People Will Agree on the Answer to a Visual Question", Gurari, D. and Grauman, K., In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems.
    • "Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets", Chang JC, Amershi S, and Kamar E. In Proceedings of the 2017 CHI Conference on Human Factors.
• 10/19/17: Lecture 14
  • Readings for next time:
    • "Crowd Diversity and Performance in Wikipedia: The Mediating Effects of Task Conflict and Communication", Ren R, Yan B. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems.
      (You will need to be inside the Cornell firewall, or otherwise go to library.cornell.edu, navigate to the ACM Digital Library, then search for the paper title.)
• 10/24/17: Lecture 15
  • Project 1
  • Readings for next time:
    • "The Effect of Peripheral Micro-tasks on Crowd Ideation", Girotto V, Walker E, Burleson W. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems 2017 May 2 (pp. 1843-1854). ACM.
• 10/26/17: Lecture 16
  • Readings for next time:
    • "Predicting the future with social media", Asur S, Huberman BA. In Proceedings 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), 2010 Aug 31 (Vol. 1, pp. 492-499). IEEE.
• 10/31/17: Lecture 17
  • Readings for next time:
    • "Next steps for citizen science". Bonney R, Shirk JL, Phillips TB, Wiggins A, Ballard HL, Miller-Rushing AJ, Parrish JK. Science. 2014 Mar 28;343(6178):1436-7.
    • "To help or hinder: Real-time chat in citizen science". Tinati, Ramine, Simperl, Elena and Luczak-Roesch, Markus. In Proceedings of the 11th International Conference on Web and Social Media.
• 11/2/17: Lecture 18
  • Readings for next time:
    • "The language that gets people to give: Phrases that predict success on Kickstarter". Mitra T, Gilbert E. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing 2014.
    • "Crowdfunding platforms and the design of paying publics. Light A, Briggs J. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems.
• 11/7/17: Lecture 19
  • Readings for next time:
    • Wikipedia's description of the Netflix Prize.
    • Wikipedia's description of the DARPA "Red Baloon" Network Challenge.
    • "Money, glory and cheap talk: analyzing strategic behavior of contestants in simultaneous crowdsourcing contests on TopCoder, Archak N. In Proceedings of the 19th International Conference on World Wide Web, 2010 Apr 26 (pp. 21-30). ACM.
• 11/9/17: Lecture 20
  • Readings for next time:
    • "Artificial intelligence and collective intelligence", Weld, D.S., Lin, C.H. and Bragg, J., 2015. In Handbook of collective intelligence. Malone, T.W. and Bernstein, M.S. (eds). MIT Press.
• 11/14/17: Lecture 21
  • Readings for next time:
    • "Generalized Task Markets for Human and Machine Computation", Shahaf, D. and Horvitz, E., 2010. In Proceedings AAAI
• 11/16/17: Lecture 22
  • Readings for next time:
    • "The crowd is a collaboration network", Gray, M.L., Suri, S., Ali, S.S. and Kulkarni, D., in Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing.
    • "Taking a HIT: Designing around rejection, mistrust, risk, and workers' experiences in Amazon Mechanical Turk", McInnis, B., Cosley, D., Nam, C. and Leshed, G., in Proceedings of the 2016 CHI conference on human factors in computing systems.
• 11/21/17: Lecture 23
  • Project 2
• 11/28/17: Lecture 24
  • Readings for next time:
    • "Human computers: the first pioneers of the information age", Grier, D.A., 2001, Endeavour, 25(1), pp.28-32.
• 12/1/17: Just for fun, "Hidden Figures", tonight at 8:30pm in Gates G01.
  • Homework 3

• Evolving reading list