• 01/08/18 The course website/syllabus is now available in draft form. It will be made official on the first day of class, but it will also continue to evolve throughout the semester. Those who are interested in taking the course should attend the first class, where we will collect information to help us decide who should be force added.

Various forms of optimization play critical roles in machine learning methods. A majority of machine learning algorithms minimize empirical risk by solving a convex or non-convex optimization. Structured predictors solve combinatorial optimizations, and their learning algorithms solve hybrid optimizations. And new approaches for stochastic optimization have become integral in modern deep learning methodology.

Students who take this course will study the latest knowledge and foundational concepts on optimization in machine learning, including theoretical analyses of optimization-based learning algorithms, theoretical bounds of discrete optimization for structured prediction, and recent discoveries about non-convex optimization methods.

Class meets Tuesday and Thursday from 12:30 PM to 1:45 PM in the New Classroom Building (NCB) 110A.

• Instructor:
  Bert Huang, Assistant Professor of Computer Science
  Office hours: Tuesday 3:30pm–4:30pm, Thursday 2:00pm–3:00pm.
  [email protected]

The course homepage (this page) is at http://berthuang.com/courses/opt18/.

The course Canvas site is at https://canvas.vt.edu/courses/66045 and should be visible to all users with a Virginia Tech login.

Prerequisites and Accommodations

Students must have taken CS5824 Advanced Machine Learning, or they must obtain the instructor’s permission if they have an equivalent background.

Please speak with the instructor if you are concerned about your background. Note: If any student needs special accommodations because of any disabilities, please contact the instructor during the first week of classes.

Learning Objectives

A student who successfully completes this class should

• know about the classical optimization techniques used in machine learning throughout its history;
• understand some of the important, known theoretical guarantees about particular optimization techniques in the context of machine learning;
• have read and analyzed the latest optimization concepts in machine learning research;
• and have a foundational understanding of optimization methods to be able to comprehend new research on optimization.

Topics

We will aim to cover the following topics.

• Gradient descent, strong convexity, stochastic gradient descent, back-propagation, Nesterov’s accelerated gradient, stochastic methods and local optima, subgradient method
• Interior point methods, quadratic programming, quadratically-constrained quadratic programming, semi-definite programming
• Newton’s method, coordinate descent, quasi-Newton methods, proximal methods
• Duality, Lagrangian duality, Fenchel duality, alternating direction method of multipliers
• Adaptive gradient methods, adagrad, Adam
• Discrete optimization, linear programs, submodularity, cutting-plane methods, structured prediction, convex-concave procedure
• Natural gradients, Bayesian optimization, meta-learning, parallelization for optimization

(Or click here if the embedding is not working.)

Academic Integrity

The tenets of the Virginia Tech's Honor Codes will be strictly enforced in this course, and all assignments shall be subject to the stipulations of the Undergraduate and Graduate Honor Codes. For more information on the Graduate Honor Code, please refer to the GHS Constitution at http://ghs.graduateschool.vt.edu. The Undergraduate Honor Code pledge that each member of the university community agrees to abide by states: "As a Hokie, I will conduct myself with honor and integrity at all times. I will not lie, cheat, or steal, nor will I accept the actions of those who do." A student who has doubts about how the Undergraduate Honor Code applies to any assignment is responsible for obtaining specific guidance from the course instructor before submitting the assignment for evaluation. Ignorance of the rules does not exclude any member of the University community from the requirements and expectations of the Honor Code. For additional information about the Undergraduate Honor Code, please visit: https://www.honorsystem.vt.edu/

This course will have a zero-tolerance philosophy regarding plagiarism or other forms of cheating. Your homework assignments must be your own work, and any external source of code, ideas, or language must be cited to give credit to the original source. I will not hesitate to report incidents of academic dishonesty to the graduate school or honor system.

Principles of Community

Because the course will include in-class discussions, we will adhere to Virginia Tech's Principles of Community. The first two principles are most relevant:

• We affirm the inherent dignity and value of every person and strive to maintain a climate for work and learning based on mutual respect and understanding.
• We affirm the right of each person to express thoughts and opinions freely. We encourage open expression within a climate of civility, sensitivity, and mutual respect.

The remaining principles are also important and we will take them seriously as a class.

Grading Breakdown

• 5%: Homework 0
• 25%: Discussion leading
• 20%: Takeaway summaries
• 25%: Blog posts
• 25%: Final project

Based on the grading breakdown above, each student's final grade for the course will be determined by the final percentage of points earned. The grade ranges are as follows:

A:

93.3%–100%

A-:

90.0%–93.3%

B+:

86.6%–90.0%

B:

83.3%–86.6%

B-:

80.0%–83.3%

C+:

76.6%–80.0%

C:

73.3%–76.6%

C-:

70.0%–73.3%

D+:

66.6%–70.0%

D:

63.3%–66.6%

D-:

60.0%–63.3%

F:

00.0%–60.0%