Instructor: Anil Damle

Contact: damle@cornell.edu

Office hours: Tuesdays 3:15 - 4:30 pm (Booking Link) in 423 Gates Hall

Instructor: Wen Sun

Contact: ws455@cornell.edu

Office hours: Thursdays 2 - 3 pm in Gates Hall 416b

Lectures: Tuesday and Thursday from 9:40 am to 10:55 am in Bailey Hall 101.

Course staff office hours: Calendar link

Course overview: The course provides an introduction to machine learning, focusing on supervised learning and its theoretical foundations. Topics include regularized linear models, boosting, kernels, deep networks, generative models, online learning, and ethical questions arising in ML applications.

Prerequisites: probability theory (e.g. BTRY 3080, ECON 3130, MATH 4710, ENGRD 2700), linear algebra (e.g. MATH 2940), calculus (e.g. MATH 1920), and programming proficiency (e.g. CS 2110).

Course logistics: For enrolled students the companion Canvas page serves as a hub for access to Ed Discussions (the course forum), Vocareum (for course projects), Gradescope (for HWs), and paper comprehension quizzes. If you are enrolled in the course you should automatically have access to the site. Please let us know if you are unable to access it.

Homework, projects, and exams

---

Your grade in this course is comprised of three components: homework, exams, and projects. Please also read through the given references in concert with the lectures.

• Homework:

  There will be a number of homework assignments throughout the course, typically made available roughly one to two weeks before the due date. The homework primarily focuses on theoretical aspects of the material and is intended to provide preparation for the exams. Homework may be completed in groups of up to three. The assignments themselves will be made available via Gradescope (through Canvas). You are allowed two slip days per homework.

• Projects

  To provide hands on learning with the methods we will discuss in class there are a number of programming projects throughout the course. The projects may be completed solo or in a group of two. They are accessed, submitted, and graded using Vocareum. You are allowed two slip days per project.

• Paper comprehension

  Students enrolled in this course at the graduate level (i.e., enrolled in 5780) are required to read assigned research papers and complete the associated online quiz. Papers will be assigned roughly once every two to three weeks. You are allowed two slip days per quiz.

• Exams:

  There will be two exams for this class, an evening prelim and a final exam.

  • Midterm exam: October 18, 2022 at 7:30 pm in Baker Lab 200, Rockefeller 201, and Rockefeller 203
  • Final exam: December 14, 2022 at 2 pm in Statler 185 and 196

Grading

Final grades are based on homework assignments, programming projects, and the exams. For the 5780 level version of the course the research comprehension quizzes will also factor in.

For CS 4780 your final grade consists of:

• Exams: 50%
• Homework: 10%
• Projects: 40%

For CS 5780 your final grade consists of:

• Exams: 45%
• Homework: 10%
• Projects: 35%
• Paper comprehension: 10%

Undergraduates enrolled in 4780 may choose to do the paper comprehension assignments; if completed you will receive the higher of your two grades between the above schemes.

Schedule

---

A tentative schedule follows, and includes the topics we will be covering, relevant reference material, and assignment information. It is quite possible the specific topics covered on a given day will change slightly. This is particularly true for the lectures in the latter part of the course, and this schedule will be updated as necessary.

Date	Topic	References	Notes/assignments
8/23	Introduction	PML: 1.1; ESL: Ch. 1; and PPA: Ch. 1
8/25	ML Basics	Lecture Notes, Slides, Annotated Slides, PML: 1.2, and ESL: 2.1 and 2.2.
8/30	K Nearest Neighbors and the curse of dimensionality	Lecture Notes, Slides, Annotated Slides, PML: 16.1	5780: Cover and Hart 1967
9/1	Clustering: K-means	Lecture Notes, Slides, Annotated Slides, ESL: 14.3.6 and 14.3.7, and PML: 21.3
9/6	Principal Component Analysis	Lecture Notes, Slides, Annotated Slides, PML: 20.1, ESL: 14.5.1 and 14.5.2
9/8	The Perceptron	Lecture Notes and Wikipedia article
9/13	MLE and MAP	Lecture Notes, Slides, Annotated Slides, Nice Youtube video for MLE and MAP. Ben Taskar's lecture notes. ESL: 8.2.2-8.3	Homework 1 due; Project 1 due
9/15	Naive Bays	Lecture Notes, ESL: 6.6.3, and Tom Mitchell's book chapter	Cover and Hart reading quiz due
9/20	Logistic Regression	Lecture Notes, ESL: 4.4, and PML: 10.1, 0.2, and 10.3
9/22	Gradient descent and Newton's method	Lecture Notes, Alternative Lecture Notes, PML: 8.1, 8.2, and 8.3	Homework 2 due
9/27	Assorted, relevant topics in optimization	Lecture Notes, Alternative Lecture Notes, PML: 8.1, 8.2, 8.3, and 8.4 (specifically, see PML 8.4 for SGD)	Project 2 due
9/29	Linear regression	Lecture Notes;, Slides, Annotated Slides,PML 11.1, 11.2,11.3; Tom Mitchell’s book chapter on Naive Bays and Linear Regression; and ESL 3.2
10/4	Support Vector Machine	Lecture Notes, Slides, Annotated Slides	Project 3 due
10/6	Empirical Risk Minimization	Lecture Notes, Slides, Annotated Slides, PML 4.3, 5.4	Homework 3 due
10/11	Fall break, no class
10/13	Midterm Jeopardy		Homework 4 due 10/14
10/18	Exam Review		Evening prelim
10/20	Bias and Variance Tradeoff	Lecture Notes, Slides, Annotated Slides, Note
10/25	Bias and Variance Tradeoff and Model Selection	Lecture Notes, Slides, Annotated Slides, Note	Project 4 due
10/27	Kernels, part 1	Lecture Notes, PML: 17.1
11/1	Kernels, part 2	Lecture Notes, PML: 17.3
11/3	Classification and regression trees, part 1	Lecture Notes	Homework 5 due; Project 5 due
11/8	Classification and regression trees, part 2	Lecture Notes
11/10	Ensemble Methods: Bagging & random forest	Lecture Notes, Slides, Annotated Slides
11/15	Ensemble Methods: Boosting	Lecture Notes, Slides, Annotated Slides	Project 6 due
11/17	Neural Network	Lecture Notes, Slides, Annotated Slides	Homework 6 due
11/22	Neural Network: backpropagation	Note on Backpropagation, Slides,Annotated Slides, Lecture Notes
11/24	Thanksgiving, no class
11/29	Neural networks: convolutional and residual layers	PML: 14.1, 14.2, 14.3 and Lecture Notes	Project 7 due; Homework 7 due
12/1	Neural networks: transformers	PML: 15.4 and 15.5, and Lecture Notes	Project 8 due and Homework 8 due 12/5
12/14	Final Exam	2 pm in Statler 185 and 196

References

---

While this course does not explicitly follow a specific textbook, there are several that are very useful references to supplement the course.

Books

We will not be explicitly following any single textbook in this course. Nevertheless, the books by Golub and Van Loan, and Trefethen and Bau collectively cover the material for the course and are recommended. Most suggested readings are assigned out of these two texts. Three additional texts are provided that complement these texts and are useful for further study (or to gain another perspective).

• Machine Learning A Probabilistic Perspective by Murphy
  We will provide section numbers to this text alongside many of the lectures (abbreviated as PML in the schedule). This text is available digitally through the Cornell University Library and a draft version is available directly from the author. [Book website]
• The Elements of Statistical Learning by Hastie, Tibshirani, and Friedman
  This text provides a comprehensive introduction to statistical learning and provides in-depth discussion of many of the topics in this course (abbreviated as ESL in the schedule). The book is available directly from the authors. [Book website]

Additional references

• An Introduction to Statistical Learning by James, Witten, Hastie, and Tibshirani
  This book provides a good overview of some methods in statistical learning, some of which we will discuss. The book is available online through the books website and via the Cornell Library. [Book website]
• Patterns, Predictions, and Actions by Hardt and Recht
  A very nice new book that covers many of the topics we do in this class (abbreviated as PPA in the schedule). The book is available directly from the authors. [Book website]
• Fairness and Machine Learning by Barocas, Hardt, and Narayanan
  While a work in progress, this text provides insight into fairness as a central tenet of machine learning. In particular, it highlights ethical challenges that arise in the practice of machine learning. The current version of this book is available directly from the authors. [Book website]

Background references

• Linear Algebra by Kahn Academy
  Relive the basics of linear Algebra. Everybody loves Kahn Academy. [Linear Algebra (Kahn Academy)]
• Linear algebra course by Strang
  Portions of this course will utilize your knowledge of linear algebra. If you feel you need additional preparation, or would like to revisit the topic, you may find Gilbert Strangs linear algebra course quite useful. [MIT Open Courseware]
• Matrix Methods in Data Analysis, Signal Processing, and Machine Learning by Strang
  A subsequent course to the above by Strang covers some of the same topics we will (particularly for the linear algebra part of the course) and you may find the videos a useful additional resource. [MIT Open Courseware]

Software

• Python
  
• NumPy
  
• PyTorch
  

Course policies

---

Inclusiveness

You should expect and demand to be treated by your classmates and the course staff with respect. You belong here, and we are here to help you learn and enjoy this course. If any incident occurs that challenges this commitment to a supportive and inclusive environment, please let the instructors know so that the issue can be addressed. We are personally committed to this, and subscribe to the Computer Science Department’s Values of Inclusion. [Statement reproduced with permission from Dan Grossman.]

Mental health resources

Cornell University provides a comprehensive set of mental health resources and the student group Body Positive Cornell has put together a flyer outlined the resources available.

Participation

You are encouraged to actively participate in class. This can take the form of asking questions in class, responding to questions to the class, and actively asking/answering questions on the online discussion board.

Collaboration policy

Students are free to share code and ideas within their stated project/homework group for a given assignment, but should not discuss details about an assignment with individuals outside their group. The midterm and final exam are individual assignments and must be completed by yourself.

Academic integrity

The Cornell Code of Academic Integrity applies to this course.

Accommodations

In compliance with the Cornell University policy and equal access laws, we are available to discuss appropriate academic accommodations that may be required for student with disabilities. Requests for academic accommodations are to be made during the first three weeks of the semester, except for unusual circumstances, so arrangements can be made. Students are encouraged to register with Student Disability Services to verify their eligibility for appropriate accommodations.

COVID-19 considerations

While many aspects of this course are built with flexibility in mind, if situations arise that may require additional accommodations please reach out to the instructors to discuss potential arrangements.