CS 4727/5727 – Adversarial Machine Learning

University of Idaho - Department of Computer Science

Instructor: Alex Vakanski (vakanski@uidaho.edu)

Office Hours: Fridays, 12:00 PM – 1:00 PM Pacific (1:00 PM – 2:00 PM Mountain), held on Zoom, link available on Canvas.

Semester: Spring 2026 (January 14 – May 15)

Credit Hours: 3

Instructor website: https://www.idahofallshighered.org/vakanski/Courses/Adversarial_Machine_Learning.html

Course website: https://avakanski.github.io/Spring-2026-Adversarial-Machine-Learning/

GitHub repository: https://github.com/avakanski/Spring-2026-Adversarial-Machine-Learning

Previous Offerings

Course materials from previous years are available on the instructor’s website: Spring 2025, Spring 2024, Spring 2023.

Course Syllabus

View Syllabus

Course Description

The course introduces students to adversarial attacks on machine learning models and defenses against the attacks. The particular focus is on adversarial attacks and adversarial examples in deep learning models, due to their prevalence in modern machine learning applications. Covered topics include evasion attacks targeting white-box and black-box machine learning models, data poisoning attacks, privacy attacks, jailbreak attacks on large language models, defense strategies against adversarial attacks, and robust machine learning models. The course also provides an overview of adversarial attacks on machine learning models used in cybersecurity applications, including malware detection and classification, network intrusion detection, spam filtering, URL detection, cyber-physical systems, and biometric systems. Additional work is required for graduate credit.

Textbooks

There are no required textbooks for this course.

Learning Outcomes

Upon the completion of the course, the students should demonstrate the ability to:

  1. Identify the vulnerabilities of machine learning models to various types of adversarial attacks.
  2. Differentiate between adversarial evasion attacks in white-box and black-box settings and understand the principles of data poisoning attacks.
  3. Explain the fundamentals of adversarial privacy attacks and outline privacy-preserving defense methods.
  4. Describe jailbreak and poisoning attacks on large language models and propose corresponding mitigation methods.
  5. List common defense strategies against adversarial attacks and discuss approaches for improved robustness of machine learning models.
  6. Identify the unique characteristics of adversarial attacks on machine learning models in the cybersecurity domain.
  7. Implement adversarial attacks and defenses against conventional machine learning models and deep learning models.
  8. Evaluate the effectiveness of adversarial attacks on anomaly detection systems for network intrusion detection, machine learning malware classifiers, and anti-spam filtering models.
  9. Analyze the ethical and societal implications of adversarial attacks and defenses.

Prerequisites

CS 2120 Practical Python, or CS 4771 Python for Machine Learning, or Instructor’s Permission.

Students are expected to have a basic understanding of linear algebra, probability and statistics, and machine learning concepts. Familiarity with neural networks and deep learning is recommended, but not required. A working knowledge of Python programming is required to complete the course assignments and the course project. Additionally, it is preferred that the students have a basic level of familiarity with at least one of the following machine learning libraries: TensorFlow, Keras, or PyTorch.

Grading

Student assessment will be based on 5 homework assignments (worth 30 marks), 4 quizzes (worth 40 marks), a course project (worth 10 marks), a lecture presentation (worth 10 marks), and class attendance and participation (worth 10 marks). The assignments and the project involve implementation of the studied adversarial attacks and defense methods on medium-size datasets (e.g., a few thousand images or similar). To complete the assignments and the project, the students can either use the GPUs provided by Google Colab, or they can use other programming environments that provide access to GPUs.

Assessment Component Marks
Assignments (×5) 30
Quizzes (×4) 40
Course Project 10
Presentation 10
Attendance and Participation 10
Total 100

Course Schedule

The schedule below is tentative and lists the topics covered each week. Lecture titles link to the corresponding lecture slides. Each lecture lists a required reading and optional readings that are discussed in class.

Week Topics Due
Week 1
Jan. 12		 Introduction
Lecture 1 - Introduction to Adversarial Machine Learning
Week 2
Jan. 19		 Lecture 1 - Introduction to Adversarial Machine Learning (continued)
Foundations
Lecture 2 - Deep Learning Overview
Week 3
Jan. 26		 Lecture 2 - Deep Learning Overview (continued)
Lecture 3 - Mathematics for Machine Learning
Reading: Goodfellow (2014) Explaining and Harnessing Adversarial Examples		 Assignment 1
Week 4
Feb. 2		 Lecture 3 - Mathematics for Machine Learning (continued)
Evasion Attacks
Lecture 4 - White-box Evasion Attacks
Reading: Carlini (2017) Towards Evaluating the Robustness of Neural Networks
Optional: Papernot (2016) The Limitations of Deep Learning in Adversarial Settings; Xiao (2018) Spatially Transformed Adversarial Examples; Su (2019) One Pixel Attack for Fooling Deep Neural Networks		 Quiz 1
Week 5
Feb. 9		 Lecture 4 - White-box Evasion Attacks (continued)
Lecture 5 - Black-box Evasion Attacks
Reading: Brendel (2017) Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models
Optional: Papernot (2016) Transferability in Machine Learning: From Phenomena to Black-Box Attacks using Adversarial Samples; Chen (2019) HopSkipJumpAttack: A Query-efficient Decision-based Adversarial Attack; Guo (2019) Simple Black-box Adversarial Attacks
Week 6
Feb. 16		 Lecture 5 - Black-box Evasion Attacks (continued)
Large Language Models
Lecture 6 - Introduction to Large Language Models
Week 7
Feb. 23		 Lecture 7 - Adversarial Attacks on Large Language Models
Jailbreak attacks, prompt injection attacks
Reading: Zou (2023) Universal and Transferable Adversarial Attacks on Aligned Language Models
Optional: Paulus (2024) AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs; Wei (2023) Jailbroken: How Does LLM Safety Training Fail?; Qi (2024) Visual Adversarial Examples Jailbreak Aligned Large Language Models; Ding (2023) A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts Can Fool Large Language Models Easily		 Assignment 2
Week 8
Mar. 2		 Defenses and Poisoning Attacks
Lecture 8 - Defenses Against Evasion Attacks
Reading: Cohen (2019) Certified Adversarial Robustness via Randomized Smoothing
Optional: Madry (2017) Towards Deep Learning Models Resistant to Adversarial Attacks; Zhang (2019) Theoretically Principled Trade-off between Robustness and Accuracy; Carlini (2022) (Certified!!) Adversarial Robustness for Free!; Zou (2024) Improving Alignment and Robustness with Circuit Breakers		 Assignment 3
Week 9
Mar. 9		 Quiz 2
Week 10
Mar. 23		 Lecture 9 - Poisoning Attacks and Defenses
Reading: Liu (2018) Trojaning Attack on Neural Networks; Wang (2019) Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks
Optional: Souri (2022) Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch; Souly (2025) Poisoning Attacks on LLMs Require a Near-Constant Number of Poison Samples; Wan (2023) Poisoning Language Models During Instruction Tuning; Huang (2019) NeuronInspect: Detecting Backdoors in Neural Networks via Output Explanations; Steinhardt (2017) Certified Defenses for Data Poisoning Attacks
Week 11
Mar. 30		 Adversarial ML in Cybersecurity
Lecture 10 - Network Intrusion Detection
Lecture 11 - Malware Detection and Classification
Reading: Severi (2021) Explanation-Guided Backdoor Poisoning Attacks Against Malware Classifiers
Optional: Rosenberg (2021) Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain; Kuppa (2019) Black Box Attacks on Deep Anomaly Detectors; Demetrio (2019) Explaining Vulnerabilities of Deep Learning to Adversarial Malware Binaries		 Assignment 4
Week 12
Apr. 6		 Lecture 12 - Spam Filtering, URL Detection, and Cyber-physical Systems
Spam filtering and URL detection; cyber-physical and biometric systems
Reading: Erba (2019) Constrained Concealment Attacks against Reconstruction-based Anomaly Detectors in Industrial Control Systems
Optional: Shirazi (2019) Adversarial Sampling Attacks Against Phishing Detection; Anderson (2016) DeepDGA: Adversarially-Tuned Domain Generation and Detection; Wang (2018) With Great Training Comes Great Vulnerability: Practical Attacks against Transfer Learning
Week 13
Apr. 13		 Privacy, Explainability, Fairness
Lecture 13 - Privacy Attacks and Defenses
Privacy attacks in machine learning
Reading: Shokri (2017) Membership Inference Attacks Against Machine Learning Models
Optional: Rigaki (2021) A Survey of Privacy Attacks in Machine Learning; Fredrikson (2015) Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures		 Quiz 3
Week 14
Apr. 20		 Lecture 13 - Privacy Attacks and Defenses (continued)
Defenses against privacy attacks
Reading: Abadi (2016) Deep Learning with Differential Privacy
Optional: Nasr (2018) Machine Learning with Membership Privacy using Adversarial Regularization; Papernot (2018) Scalable Private Learning with PATE; McKenna (2025) Scaling Laws for Differentially Private Language Models		 Assignment 5
Project proposal
Week 15
Apr. 27		 Lecture 14 - Explainability in Machine Learning
Reading: Templeton (2024) Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
Optional: Cunningham (2023) Sparse Autoencoders Find Highly Interpretable Features in Language Models; Sundararajan (2017) Axiomatic Attribution for Deep Networks; Montavon (2017) Explaining Nonlinear Classification Decisions with Deep Taylor Decomposition
Week 16
May 4		 Lecture 15 - Bias and Fairness in Machine Learning
Reading: Navigli (2023) Biases in Large Language Models: Origins, Inventory, and Discussion
Optional: Gallegos (2023) Bias and Fairness in Large Language Models: A Survey; Dai (2024) Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era; Anthis (2024) The Impossibility of Fair LLMs; Ye (2024) Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge		 Project report
Week 17
May 11		 Quiz 4