CS 6742 Fall 2014: Natural Language Processing and Social Interaction

Image source: http://en.wikipedia.org/wiki/The_School_of_Athens. Some people are speaking to each other; some are reading and perhaps being influenced by that text; some are writing text, perhaps hoping to have an effect on others; some texts are being read by several people simulataneously.

Scan of lecture notes

Images and webpages displayed in class:

• An annotated Wikipedian vote page with comment threads (comments can be seen by clicking on the "1" speech balloons) and notabilia.net visualization of vote dynamics on such pages
• A Slashdot page (try out the various filtering/visualizations) and an interactive visualization of the corresponding conversation tree by the ConVis project, Hoque, Enamul, Giuseppe Carenini, and Shafiq Joty. 2014. Interactive exploration of asynchronous conversations: Applying a user-centered approach to design a visual text analytic system. Proceedings of the ACL Workshop on Interactive Language Learning, Visualization, and Interfaces (ILLVI).
• A screenshot of another Slashdot conversation- tree visualizer taken from Pascual-Cid, Victor and Andreas Kaltenbrunner. 2009. Exploring asynchronous online discussions through hierarchical visualisation. Information Visualisation.

References

Bryan, Christopher J, Gregory M Walton, Todd Rogers, and Carol S Dweck. 2 August 2011. Motivating voter turnout by invoking the self. Proceedings of the National Academy of Sciences 108 (31): 12653-12656.

Chong, Dennis and James N. Druckman. 2007. Framing theory. Annual Review of Political Science 10:103--126.

Image source: Randall Munroe, xkcd (click on image for original link). Expletive obscured in this presentation.

Scan of lecture notes

Images and handouts from class

• outline handout
• helpfulness features handout
• SGJ, WWW'14, fig 3a
• MAT, Science '13, fig 1c
• Slides on DNM/K/K/L, WWW'09
• C/DNM/L, ICWSM '14, fig 4

References on lecture topics

Cheng, Justin, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec. 2014. How community feedback shapes user behavior. Proceedings of ICWSM.

Danescu-Niculescu-Mizil, Cristian, Gueorgi Kossinets, Jon Kleinberg, and Lillian Lee. 2009. How opinions are received by online communities: A case study on Amazon.com helpfulness votes. Proceedings of WWW: 141—150. [alt link]

Ghose, Anindya and Panagiotis Ipeirotis. 2011. Estimating the helpfulness and economic impact of product reviews: Mining text and reviewer characteristics. IEEE Transactions on Knowledge and Data Engineering 23(10): 1498—1512. Official link can be found through Worldcat, e.g., here.

Muchnik, Lev, Sinan Aral, and Sean Taylor. 2013. Social influence bias: A randomized experiment. Science 341.

Otterbacher, Jahna. 2009. 'Helpfulness' in online communities: a measure of message quality. Proceedings of CHI, 955-964.

Sipos, Ruben, Arpita Ghosh, and Thorsten Joachims. 2014. Was this review helpful to you? It depends! Context and voting patterns in online content. Proceeedings of WWW.

Wang, R.Y. and Strong, D.M. Beyond accuracy: what data quality means to data consumers. Journal of Management Information Systems 12, 4 (1996), 5-34.

Representative additional references on "unconventional" text classification, by popular demand

Davidov, Dmitry, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in Twitter and Amazon. Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pp. 107--116. http://aclweb.org/anthology/W10-2914

Kiddon, Chloé and Yuriy Brun. That's what she said: Double entendre classification. Proceedings of the ACL (short papers), 89--94.

Li, Jiwei, Myle Ott, Claire Cardie, and Eduard Hovy. 2014. Towards a general rule for identifying deceptive opinion spam. Proceedings of the ACL. The paper showing a learned classifier outperforming humans on Tripadvisor-style reviews is Ott, M, Y Choi, C Cardie, and J T Hancock. 2011. Finding deceptive opinion spam by any stretch of the imagination. Proceedings of the ACL, pp. 309--319.

Mihalcea, Rada and Carlo Strapparava. 2006. Learning to laugh (automatically): Computational models for humor recognition. Computational Intelligence 22(2).

Image source: http://www.metmuseum.org/toah/works-of-art/49.70.33. "The image is one for which Picasso did a number of variations in Paris during the autumn–winter of 1912; in each version, a tall bottle and goblet are set out on a small round table."

Scan of lecture notes and the handout

References related to the A1 project discussions

• On code switching: Auer, Peter. 2013. Code-switching in Conversation: Language, Interaction and Identity. Routledge. Sample "ACL-style" paper: Elfardy, Heba and Mona Diab. 2012. Token level identification of linguistic code switching, Proceedings of COLING.
• The LIWC lexicon (categorized word lists): Tausczik, Y R and J W Pennebaker. 2010. The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology 29(1). Technically, single licenses are available for purchase from this site. A freely available similar type of lexicon is the Harvard General Inquirer lexicon.
• Duplicate detection: Sorokina, Daria, Johannes Gehrke, Simeon Warner, and Paul Ginsparg. 2006. Plagiarism detection in arxiv. In Proceedings of the Sixth International Conference on Data Mining, 1070-1075. The code may be available upon request; there's a Java port of the original C code, but available only as a web service, I think, here.
• Humorous reviews subreddit, Funny reviews on Amazon subreddit, Funniest Amazon review I've ever read thread
• Subcommunity review sites: DPRreview, RateMyProfessors

 

References from discourse lecture

Grice, H.P. 1975. Logic and Conversation. In Syntax and semantics 3: Speech Acts, pp. 41-58.

Jurafsky, Dan, and Martin, James H. 2009. Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition Second edition. Chapter 21 covers discourse.

Moser, Megan and Johanna Moore. Toward a synthesis of two accounts of discourse structure. Computational Linguistics 22(3):409--419.

Rogers, Todd and Michael I Norton. June 2011. The artful dodger: Answering the wrong question the right way. Journal of Experimental Psychology: Applied 17 (2).

References for the examples on the handout:

Jordan Boyd-Graber Google+ post

Allen, James. 1995. Natural Language Understanding. Benjamin/Cummings Pub Co. Second ed.

Hirst, Graeme. 1981. Anaphora in Natural Language Understanding: A Survey. Lecture Notes in Computer Science. Springer, Berlin.

Sidner, Candace Lee. 1979. Towards a computational theory of definite anaphora comprehension in English discourse. MIT AITR-537.

Wilks, Yorick. 1975. An intelligent analyzer and understander of English. Communications of the ACM 18 (5): 264-274.

 

"Stacking", by Alastair Hesletine. Image source: http://thumbpress.com/the-art-of-stacking-wood/

Scan of discussion notes

References see also the previous discourse lectures

Wikipedia entry on Deep Blue vs. Garry Kasparov (pronunciation)

Stolcke, Andreas Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech. Computational Linguistics 26(3): 339--373.

Taboada, Maite and William C. Mann. 2006. Rhetorical structure theory: Looking back and moving ahead. Discourse Studies 8(3): 423-459. Gives an overview of many issues in analyzing discourse structure.

Walker, Marilyn A. 1996. Limited attention and discourse structure. Computational Linguistics 22(2): 255-264.

Image source: http://xkcd.com/1055/

The assigned reading: one of:

1. Glasgow, Kimberly, Clayton Fink, and Jordan Boyd-Graber. 2014. Our grief is unspeakable: Automatically measuring the community impact of a tragedy. Proceedings of ICWSM.
2. Mitra, Tanushree and Eric Gilbert. 2014. The language that gets people to give: Phrases that predict success on Kickstarter. Proceedings of CSCW.

Sites examined or mentioned during class

• A sample www.debate.org "duel"; notice the plethora of evaluative explicit or implicit annotations. Other debate sites include createdebate, www.forandagainst.com, www.convinceme.net, idebate.org See also the Internet Argument Corpus, described in A corpus for research on deliberation and debate. Marilyn A. Walker, Pranav Anand, Jean E. Fox Tree, Rob Abbott, Joseph King. LREC 2012
• Scrape of 87K Kickstarter projects. Discovered by looking at the Greenberg et al. 2013 paper mentioned below, who linked to thekickbackmachine.com by neight-allen. That site itself is no longer maintained, and there may be a problem with the file, but "dm" at thekickbackmachine.com recommends Walter Haas' Kickspy.
  • Potato-salad kickstarter project.
• Data on 45K Kickstarter projects, by "Jeanne"
• Memetracker website, including data: http://www.memetracker.org/index.html. "Palling around with terrorists" quote graph (links = share many words). "Heartbeat" figure for when blogs vs. the mainstream media discuss an event. See publication info below.
• Note also the ICWSM 2011 Spinn3r dataset: over 386 million blog posts, news articles, classifieds, forum posts and social media content between January 13th and February 14th. It spans events such as the Tunisian revolution and the Egyptian protests (see http://en.wikipedia.org/wiki/January_2011 for a more detailed list of events spanning the dataset's time period).
• A sample Quirky appeal for help in pricing, tagline selection, etc.
• randomactsofpizza.com. Note the need for guarding against scam attempts, described in the "Best Practices" section. The actual subreddit is here. Example of reciprocity offer. Note annotation of who actually received pizza. See publication info below.
• TREC 2011 microblog-track tweet dataset and link to twitter-tools by Jimmy Lin ("lintool"), which "provides support for removing deleted tweets from your copy of the corpus", as is important to be compliant with Twitter's policies.

References (including some that came up during class)

Althoff, Tim, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2014. How to ask for a favor: A case study on the success of altruistic requests. Proceedings of ICWSM.

Bailey, Michael, Daniel J Hopkins, and Todd Rogers. 2013. Unresponsive and unpersuaded: The unintended consequences of voter persuasion efforts. Working paper on SSRN.

Bamman, David, Brendan O'Connor, and Noah Smith. 2012. Censorship and deletion practices in Chinese social media. First Monday 17(3).

Bell, Brad E and Elizabeth F Loftus. May 1989. Trivial persuasion in the courtroom: The power of (a few) minor details Journal of Personality and Social Psychology 56(5):669-679.

Danescu-Niculescu-Mizil, Cristian, Justin Cheng, Jon Kleinberg, and Lillian Lee. 2012. You had me at hello: How phrasing affects memorability. Proceedings of the ACL, pp. 892--901.

Gayo-Avello, Daniel. December 2013. A meta-analysis of state-of-the-art electoral prediction from Twitter data. Social Science Computer Review 31(6): 649-679. Hat tip to Brendan O'Connor; I saw this on his 2013 blog post Some analysis of tweet shares and “predicting” election outcomes. Also of interest, for the title alone: Gayo-Avello's "I wanted to predict elections with twitter and all I got was this lousy paper" -- A balanced survey on election prediction using twitter data, Eprint ArXiv:1204.6441 and On Twitter and Elections, catchy paper titles, press releases and telling scientist's opinions from facts: A brief comment to DiGrazia et al. 2013 and to Fabio Rojas Op-Ed in Washington Post.

Greenberg, Michael D, Bryan Pardo, Karthic Hariharan, and Elizabeth Gerber. 2013. Crowdfunding support tools: Predicting success & failure. Proceedings of CHI: Extended Abstracts, pp. 1815--1820.

Guerini, Marco, Carlo Strapparava, and Oliverio Stock. 2010. Evaluation metrics for persuasive NLP with Google adwords. Proceedings of LREC.

Hannak, Aniko, Drew Margolin, Brian Keegan, and Ingmar Weber. 2014. Get back! You don't know me like that: The social mediation of fact checking interventions in Twitter conversations. Proceedings of ICWSM.

King, Gary, Jennifer Pan, and Margaret E Roberts. 2013. How censorship in China allows government criticism but silences collective expression. American Political Science Review 107(02): 326-343.

Leskovec, Jure, Lars Backstrom, and Jon Kleinberg. 2009. Meme-tracking and the dynamics of the news cycle. In Proceedings of KDD, 497-506.

Petrovic, Sasa, Miles Osborne, and Victor Lavrenko. 2013. I wish I didn't say that! Analyzing and predicting deleted messages in Twitter. eprint arXiv:1305.3107.

Qazvinian, Vahed, Emily Rosengren, Dragomir R. Radev, and Qiaozhu Mei. 2011. Rumor has it: Identifying misinformation in microblogs. Proceedings of EMNLP, 1589--1599.

Thelwall, Mike, Kevan Buckley, and Georgios Paltoglou. 2011. Sentiment in Twitter events. Journal of the American Society for Information Science and Technology 62(2): 406-418.

Image source: www.catandgirl.com/?p=2105

1. Garley, Matt and Julia Hockenmaier. 2012. Beefmoves: Dissemination, diversity, and dynamics of English borrowings in a German hip hop forum. Proceedings of ACL.
2. Vasilescu, Bogdan, Alexander Serebrenik, Prem Devanbu, and Vladimir Filkov. 2014. How social Q&A sites are changing knowledge sharing in open source software communities. Proceedings of CSCW, pp. 342--354.

Some things discussed or tried in class

• About the Cornell Institutional Review Board for Human Participants. Flowchart: how to decide if your activity is covered by Cornell's Human Research Protection Program.
• Link generated for sharing a StackOverflow question for Twitter: http://stackoverflow.com/q/23639039?stw=2 ; for email: http://stackoverflow.com/q/23639039?sem=2 , etc. Note that we can search for such things; for example: https://twitter.com/search?q=stackoverflow.com%20stw%3D2 (this is a query for stackoverflow.com and stw=2).
  • About collecting data from Twitter:
    • Tutorial on using twitteR (R) to mine Twitter attitudes towards airlines
    • REST API documentation and Streaming API
    • Cristian Danescu-Niculescu-Mizil's advice:

      (a) Rate limitations: http://dev.twitter.com/pages/rate-limiting
      Not complying with the rate limitations will result in your IP getting blacklisted. For the REST API there is a clear limit and an easy way to track your limit status: https://api.twitter.com/1.1/application/rate_limit_status.json (which should be called frequently by your code).

      (b)  When interacting with the API, use exception clauses. Many things can go wrong. When handling the exceptions, keep (a) in mind.

      (c)  Design your data gathering so that it can be easily restarted, without losing what was already collected.

      (d)  For the most popular programing languages there are many Twitter Libraries that can be used to send requests to the API: https://dev.twitter.com/overview/api/twitter-libraries

• Query for code-switching examples in Thai: https://twitter.com/search?q=%22code-switching%22%20lang%3Ath .
  • 55555, or, How to laugh online in other languages. Megan Garber, The Atlantic, 2012.

References

Anderson, Ashton, Daniel Huttenlocher, Jon Kleinberg, and Jure Leskovec. 2013. Steering user behavior with badges. Proceedings of WWW.

Farshad Kooti, Haeryun Yang, Meeyoung Cha, Krishna Gummadi, and Winter Mason. The emergence of conventions in online social networks. Proceedings of ICWSM. Best paper award.

Vasilescu, Bogdan, Andrea Capiluppi, and Alexander Serebrenik. 2012. Gender, representation and online participation: A quantitative study of stackoverflow. Proceedings of Social Informatics, pp. 332--338.

Upcoming submission deadlines: NAACL long & short papers: sub Dec 4, notification Feb 20, multiple submissions OK. ICWSM full and poster: abstracts Jan 18, papers Jan 23, notification March 9. WWW "Web Science track": abstracts Jan 19, papers Jan 23, notification Feb 27.

Scan of lecture notes

Deferred, for the most part, until next class meeting:

Louis, Annie and Ani Nenkova. 2013. What makes writing great? First experiments on article quality prediction in the science journalism domain. Transactions of the Association for Computational Linguistics 1:341-352.

Resources

MRC Psycholinguistic database. Citation: Wilson, M.D. (1988) The MRC Psycholinguistic Database: Machine Readable Dictionary, Version 2. Behavioural Research Methods, Instruments and Computers, 20(1), 6-11. A search interface makes it clear what kind of features (annotations) are identified for the lexicon items.

Image source: http://popchartlab.com/products/a-diagrammatical-dissertation-on-opening-lines-of-notable-novels, available for purchase; image crop by Popular Science

Scan of lecture notes

Our starting point: Louis, Annie and Ani Nenkova. 2013. What makes writing great? First experiments on article quality prediction in the science journalism domain. Transactions of the Association for Computational Linguistics 1:341-352.

Jurafsky, Daniel and James H. Martin. 2009. Speech and Language Processing: An Introduction to Natural Language Processing,
Computational Linguistics, and Speech Recognition. 2nd edition. Slides by Kathy McCoy corresponding to chapter 14 - slide 7 (Fig 14.12) is what I showed in class, and slides 17 and 18 (need to split categories), 20 and 22 (importance of lexical information) were ones I contemplated showing. Section 12.7 discussed dependency parsing. I also displayed figure 12.14, a dependency parse, from chapter 12.

Notes for a lecture I gave on context-free grammars in 2007, scribed by Cristian Danescu-Niculescu-Mizil, Nam Nguyen, and Myle Ott.

References

Abeillé, Anne and Yves Schabes. 1989. Parsing idioms in lexicalized TAGs. EACL, pp. 1–9.

Jäger, Gerhard and James Rogers. 2012. Formal language theory: refining the Chomsky hierarchy. Philosophical Transactions of the Royal Society B: Biological Sciences. I picked this for being brief, hitting the whole Chomsky hierarchy, and mentioning the mildly-context-sensitive languages and their relation to natural language; but one would not necessarily argue that this is an easy introduction.

Kroch, Anthony and Aravind Joshi. 1985. The linguistic relevance of tree adjoining grammar. UPenn Technical Report MS-CIS-85-16.

Joshi, Aravind K. and Yves Schabes. 1997. Tree-adjoining grammars. In Grzegorz Rozenberg and Arto Salomaa, editors, Handbook of Formal Languages, volume 3 (Beyond words), pp. 69–12.

Joshi, Aravind, K. Vijay-Shanker, and David Weir. 1991. The convergence of mildly context-sensitive grammar formalisms. In Peter Sells, Stuart Shieber and Thomas Wasow, Eds., Foundational Issues in Natural Language Processing. Link is to a technical-report version.

Nadeau, David and Sekine, Satoshi. 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes 30(1): 3-26. Alternative link.

Pullum, Geoffrey K. 1986. Topic ... Comment: Footloose and context-free. Natural Language & Linguistic Theory 4(3): 409-414. Comments on attempts to prove that natural languages are not context-free.

The MaltParser dependency parser. An early paper, mentioning that "The runtime of the algoerithm is linear in the length of the input string, and the dependency graph is guaranteed to be projective and acyclic": Nivre, Joakim. 2003. An efficient algorithm for projective dependency parsing. In Proceedings of the 8th International Workshop on Parsing Technologies (IWPT 03), pp. 149-160.

The XTAG project.