Lectures

• Lecture 1, Thursday Jan 22: course introduction, beginning of dimensionality reduction.
• Lecture 2, Thursday Jan 27: Principal Component Analysis (PCA), additional lecture notes lecture notes, matlab code for the smilie example for you guys to play with code .
• Lecture 3, Thursday Jan 29: Geometric intuitions regarding Principal Component Analysis (PCA).
  • lecture notes
  • handout, with corrections (missing transpose indicators in the data matrix's rows, covariance matrix mistakenly talked about the rows instead of the columns)
  • PCA example in Python from class, slightly extended version lec03PCAExample2.py
  • Java version of the PCA example done in class, by TA Jack Hessel
  • C++ version of a PCA example, by TA Gaurav Aggarwal
• Lecture 4, Tuesday Feb 3: Introduction to canonical correlation analysis (CCA).
  • lecture notes
  • handouts: announcements, outline and cribsheet, Penn State example
  • covariance vs. correlation for the pens-and-pencils example: R program
  • Penn State Stats 505 CCA example: detailed writeup, R code, data . Very accessible.
  • UCLA IDRE CCA example: detailed writeup, R code, data . Rather accessible.
  • Warning regarding CCA implementation in scikit-learn (python)
  • Canonical correlation: A tutorial, by Magnus Borga, dated Jan 2001. Not too gentle, but useful.
  • Analysis of factors and canonical correlations, Mans Thulin, dated 2011. Not too gentle, but gives a different perspective and an example.
• Lecture 5, Thursday Feb 05: More on Canonical Correlation Analysis.
• Lecture 6, Tuesday Feb 10: Random projections.
  • handout: updated cheatsheet on projections
• Lecture 7, Thursday Feb 19: review/A1 overview.
  • lecture notes
  • handout: updated cheatsheet on projections (links to same file as mentioned in lecture 6)
  • handout: selected review questions
• Lecture 8, Tuesday Feb 24: Compressed Sensing and Sparse Recovery.
  • lecture notes
• Lecture 9, Thursday Feb 26: Introduction (transition) to clustering
  • updated lecture handout
  • lecture notes
  • References:
    • Section 10.7 of Richard O. Duda, Peter E. Hart, David G. Stork, 2001, Pattern Classification (2nd ed)
    • Sections 9.3-9.4 of David Hand, Heikki Mannila, and Padhraic Smyth, 2001, Principles of Data Mining (link gives access to Cornellians; you may need to be coming from a Cornell IP address). Official book link at MIT Press here
    • Sections 14.3.5-14.3.6 of Trevor Hastie, Robert Tibshirani and Jerome Friedman, 2009 The Elements of Statistical Learning (2nd ed)
• Lecture 10, Tuesday Mar 3: k-means clustering
  • lecture notes
  • updated lecture handout
  • k-means visualization by Naftali Harris
  • References:
    • Section 8.2 ("A k-means clustering algorithm") of John Hopcroft and Ravindran Kannan, draft of November 2014, Foundations of Data Science
    • Section 4.7 ("Clustering") of Jon Kleinberg and Éva Tardos, 2005, Algorithm Design. Slides by Kevin Wayne here.
• Lecture 11, Thursday Mar 05: Single-link clustering's optimality; spectral clustering
  • lecture notes on single-link clustering; lecture notes on spectral clustering
• Lecture 12, Tuesday Mar 10: Spectral Clustering continued
• Lecture 13, Thursday Mar 12: Kleinberg's impossibility theorem for clustering
  • lecture notes
  • lecture handout
  • iClicker results analysis
  • Reference: Jon Kleinberg. An impossibility theorem for clustering. NIPS 2003.
• Lecture 14, Tuesday Mar 17: More on the impossibility theorem; intro to ((Gaussian) mixture) models
  • lecture notes
  • lecture handout, with minor updates
  • Reference: Sections 10.1-10.3 (ignore the material on identifiability in 10.2) of Richard O. Duda, Peter E. Hart, David G. Stork, 2001, Pattern Classification (2nd ed)
• Lecture 15, Thursday Mar 19: MLE vs. MAP principles, probabilistic modeling, towards EM.
• lecture slides, with minor updates
• Lecture 16, Tuesday Mar 24: EM Algorithm
• lecture slides
• for now, until the typeset notes come out: handwritten notes scan
• Lecture 17, Thursday Mar 26: Mixtures of multinomials and EM
• Lecture 18, Tuesday Apr 7: Development of Latent Dirichlet Allocation (LDA)
  • lecture notes
  • lecture handout
  • summary/comparison of three generative stories
  • References:
    • Percy Liang and Dan Klein (2007): Structured Bayesian Nonparametric Models with Variational Inference
    • David Blei's group's topic modeling software (C, C++. R, Python)
• Lecture 19, Thursday April 9th: Graphical models
  • announcement slides
  • handwritten lecture transcript (starts late because Prof Lee was uploading the announcement slides at the beginning)
• Lecture 20, Tuesday April 14th: Graphical models
  • handwritten lecture transcript (starts late because Prof Lee was handling some course administrative stuff at the beginning)
• Lecture 21, Thursday April 16th: Graphical models
  • announcements
  • handwritten lecture transcript (typo fix Fri May 1 9:25am)
• Lecture 22, Tuesday April 21: More intuition building for reasoning with graphical models and HMMs
  • lecture notes
  • handout
  • Reference: Sections 20.2 ("Belief propagation for trees") and 20.3 ("The variable elimination algorithm") of Kevin Murphy, 2012. Machine Learning: A Probabilistic Perspective. MIT Press. (link gives access to Cornellians; you may need to be coming from a Cornell IP address).
• Lecture 23, Thursday April 23: Learning parameters for a Bayes net: the case of EM on HMMs
  • lecture notes
  • lecture handout
  • Reference: Chapter 9.3 of Frederick Jelinek (1997), Statistical Methods for Speech Recognition. Google books link
• Lecture 24, Sunday April 26th, 5-6pm, Guest lecture by Lars Backstrom, G01 Gates Hall. No notes available, by lecturer request.
• Lecture 25, Thursday April 30th: Graphical models and Wrapping up
• Lecture 26, Tuesday May 5: Valedictory: Lessons learned