CS 6784: Advanced Topics in Machine Learning

Course Overview

Instructor: Kilian Weinberger

Contact: kqw4@cornell.edu

Faculty Office Hours: Gates 410, Mondays 9-10am

Course Staff:

TA: Yann Hicke (ylh8@cornell.edu)
Office Hours: 6-8pm, Friday

Credits: 4.0 credits (3.0 for lecture, 1.0 for independent research project)

Prerequisites: CS 4780 or CS 5780 or equivalents or permission of instructor

Time and Location: Tuesdays and Thursdays, 14:55-16:10, for a total of 27 lectures

Course overview: This course covers advanced topics in machine learning, focusing on recent developments in large language models, multimodal models, and their applications. Students will engage with cutting-edge research through presentations, discussions, and a final project.

Course Structure

Weekly Schedule

Tuesdays: Overview of the week's topic. One team gives a 70-minute presentation.
Thursdays:
- The same team from Tuesday presents a paper.
- 16 other students are assigned roles (4 each: Archaeologists, Academic Researchers, Industry Experts, Ethicists).
- Role-assigned students submit 2-3 slides before class and present during small group discussions.

Class Session Structure (Thursdays)

Quiz on the paper and previous class (10 minutes)
30-minute presentation by the team
30-minute group discussion with role presentations
Optional post-class quiz (5 minutes)

Social Reading Groups

Every main presenter will lead a Perusall group for pre-class paper annotation. Groups consist of one Presenter, one Archaeologist, one Academic Researcher, one Industry Expert, one Ethicist, and six other students without specific roles (11 people total).

Course Topics

This course covers the following main topics in advanced machine learning:

LLM Foundations: Tokenization, Positional embeddings, Attention, Causal Language Modeling, In-Context Learning
LLM Extension: Quantization, Adapters, Mixture of experts, Context window, Parallelism, Distillation, Attention free architectures
To be announced

Schedule

The course follows this tentative schedule. Please note that topics and dates may be subject to change.

Date	Style	Topic
Tuesday, August 27, 2024	Lecture	Introduction
Thursday, August 29, 2024	Lecture	Introduction
Tuesday, September 3, 2024	Overview	LLM Foundations
Thursday, September 5, 2024	Paper	LLM Foundations
Tuesday, September 10, 2024	Overview	LLM Extension
Thursday, September 12, 2024	Paper	LLM Extension

Roles

Students will rotate through various roles throughout the semester:

Reviewer (all students): Provide a critical review following NeurIPS reviewer guidelines.
Presenter (4 students): Create the main presentation, covering motivation, problem definition, method, and experimental findings.
Archaeologist (4 students): Contextualize the paper within previous and subsequent work.
Researcher (4 student): Propose a follow-up project building on the current paper.
Industry Expert (4 students): Propose a new application or product and discuss its potential impacts.
Ethicist (4 students): Assess the paper's potential impact on economy, society, and/or environment.
Hacker (optional): Implement a simplified version of the paper and share a Jupyter Notebook.

Each student assigned to a role should prepare 2-3 slides and present for approximately 5-6 minutes, except for the Hacker, who provides a Jupyter Notebook instead of slides.

Weekly Assignment: By 10 PM the day before each Thursday class, all students must submit a review on CMT here.

Role Sign-ups

Students are required to sign up for their roles using this Google Sheets link. Please ensure you sign up for your preferred roles as soon as possible.

Presenter Responsibilities

In addition to creating and delivering the main presentation, students assigned as presenters will be responsible for creating quizzes for the rest of the class. These quizzes will be used to assess the class's understanding of the presented material.

Acknowledgment

The role-playing aspect of this course has evolved from an initial design by Alec and Eitan Grinspun at Columbia University.

Grading

10% Paper reviews: Each week one research paper is assigned to the class, and all students must write and submit a detailed review.
10% In-class quizzes (weekly): An in-class online quiz tests comprehension of recent material and the paper that the class had to read.
40% Independent research project: Groups of 4 students must propose and complete a research project.
40% In-class participation and presentations:
- Performance in assigned roles (Archaeologist, Academic Researcher, Industry Expert, Ethicist)
- Quality of role-based presentations and prepared slides
- Main team presentations (for those assigned as Presenters)
- Active participation in group discussions
- Optional Hacker role contributions

Note: The role-playing aspect of the course is a significant part of the in-class participation and presentations grade. Students are expected to fully engage with their assigned roles, prepare thoroughly for their presentations, and contribute meaningfully to class discussions.

Final Project

The final project aims to engage students in research related to machine learning fields covered in class.

Project Guidelines:

Extend papers from topics covered in class
Present outcomes as a research paper in standard conference paper format
Work in groups of four members
Include a "contributions" section detailing each member's work
Host the project on GitHub
Provide a written report and a descriptive Jupyter Notebook
Present the final project during the last two class sessions
Prepare a PowerPoint or LaTeX final presentation

Students are encouraged to demonstrate limitations of related work and suggest improvements by applying methods to public datasets beyond those used in the original papers.

More details about project proposals and suggested topics will be provided later in the course.

Course Policies

Attendance and Participation

Regular attendance and active participation in class discussions and group activities are essential for success in this course.

Academic Integrity

All students are expected to adhere to the Cornell University Code of Academic Integrity. Any work submitted by a student in this course for academic credit will be the student's own work.

Late Work Policy

Given the collaborative nature of many assignments, late work will generally not be accepted without prior approval from the instructor.

Accommodations for Students with Disabilities

If you have a disability and are registered with the Student Disability Services office, please make an appointment with the instructor as soon as possible to discuss any accommodations needed for the course. All such appointments will be kept confidential.

Diversity and Inclusion

We are committed to creating an inclusive learning environment that values diversity and fosters respect for all individuals. We encourage open and respectful dialogue and expect all participants to contribute to a positive and supportive learning atmosphere.