Responsible Artificial Intelligence

COMPSCI 690F, University of Massachusetts Amherst, 2023

The real-world deployment of machine learning models faces a series of lateral challenges affecting model trustworthiness, such as domain generalization, dataset shifts, causal validity, explainability, fairness, representativeness, and transparency. These challenges become increasingly important in techno-social systems affecting human high-stake decision making, which is often regulated by law. In this course, students will learn techniques for robust model evaluation, model selection, causal discovery, explainable and fair artificial intelligence, and interpretable models. In addition, students will reason about representativeness, transparency, and legal aspects of techno-social systems. The course will review both cutting-edge research and relevant portions of recent open-access textbooks. Coursework includes reading recent research papers, programming assignments, and a final group project. After completing the course, students should be able to develop, investigate, evaluate, and deploy artificial intelligence systems more responsibly.

image Image generously contributed by Mohamed Hassan.

Course Objectives

Through this course students will:

  • become familiar with outstanding challenges in artificial intelligence and techno-social systems,
  • gain an understanding of various ways in which machine learning can be evaluated,
  • practice the usage of tools explaining black-box models,
  • learn to develop models meeting various fairness criteria,
  • learn to develop interpretable machine learning models,
  • learn to reason about trustworthiness of techno-social systems.

Prerequisites

There are not official prerequisites for graduate students of computer science, but the following would be useful:

  • comfort with programming in Python (libraries: numpy, pandas, matplotlib, sklearn),
  • comfort with introductory data science, introductory machine learning, basic statistics.

Grading Criteria

Breakdown:

  • Quizzes – 25%
    • Ten weekly quizzes
  • Homework assignments – 35%
    • Five assignments
  • Final group project – 35%
    • Proposal – 7%
    • Progress report – 14%
    • Final report and poster – 14%
  • Course participation and reflection – 5%

Scale

FCC+B-BB+A-A
<70%70-73%74-77%78-81%82-85%86-89%90-92%>93%

Fractional scores will be rounded down for the purpose of establishing the final grade.

Quizzes and Assignments

There will be a weekly quiz. It will cover the material from most recent two lectures and corresponding readings. If you attend the classes and read the assigned texts, it should be easy for you to answer the quiz. Homework assignments will be more intense in the first half od the semester, so that you have more time to focus on the final project in the second half of the semester.

Final Group Project

You will complete a final project on a topic and environment/dataset of your choosing. This project will be broken into three graded components: project proposal, progress report with a presentation, and final report. Students will work in groups of 3-4. Each team is expected to meet the instructor to consult their project progress individually at least twice throughout the semester. The work on final project will begin at the end of first half of the semester and last until semester’s end.

Schedule

ClassAreaTopicReadings
1CourseIntroduction 
2CourseFinal projects introduction 
3Model evaluationOverparametrization, model stability and confidence, underspecification and colinearityD’Amour et al.
4Model evaluationCovariate shiftsSagawa et al.
5Model evaluationModel evaluation and selection , model calibration 
6Model evaluationBayesian statstics and model selection/checking, structured models, mixture models 
7Interpretable structured modelsGeneralized linear models, mixture models, mixed effect models 
8Open-world learningOpen-world learning, novelty, and concept driftsLu et al.
9Techno-social systemsHeavy-tailed distributions, Pareto distribution, Simon model, Polya urn 
10Techno-social systemsSocial networks, demographic inference, non-representativeness, post-stratificationVosoughi et al.
11Techno-social systemsDisinformation and polarization, echo chambers, biases in social media, information diffusionHuszar et al.
12Techno-social systemsQuality vs popularity: click models and heterogenous preferential growth model. Social media audit and differential privacy.Michnik et al.
13CourseFinal project proposal discussions 
14Legal notions of fairnessFATE. Introduction to fairness and legal perspective, demographic parity and disparate treatment 
15Fairness vs causalityBusiness necessity. Simpson paradox, causal graphical models introductionBarocas et al.
16Fairness vs causalityFrom legal notions to interventional mixturesBarocas et al.
17Fairness vs causalityCausal vs statistical fairness notions. Algorithmic discrimination more broadly.Obermeyer et al.
18Causality and explainabilityRandomized experiments, d-separation, interventions, ATE, potential outcomes notation, multi-stage marginal interventional mixture, direct and indirect effectsAI Bill of Rights
19Counterfactual fairnessCounterfactuals, nested counterfactuals, causal fairness, path-specific counterfactual fairness, marginal counterfactual mixtureAthey
20Statistical fairnessMixture of discriminating and non-discriminating individuals vs human-in-the-loop. Other fairness criteria: sufficiency and separation, recall parity, calibration. Impossiblity of achieving multiple fairness objectives. Preprocessing, postprocessing, inprocessing.NIST Sections 3.1 and 3.2
21CourseFinal project progress discussions 
22ExplainabilitySHAP and its variants. Feature selection vs feature relevance on the case of SHAP with different baselinesRudin
23Explainability and causality vs lawOther explainability measures, causation in the law 
24CausalityBayesian networks and causal discovery methods based on conditional independence and scoring functions, model selection revisited. Temporal sense of causality, Autoregressive models, Granger causality, transfer entropy 
25CausalityInverse probability weighting, propensity score matching, augmented inverse probability weighting 
26CourseFinal project presentations 

Readings (subject to change)

Textbooks (optional)

The course does not have a required textbook, but we will discuss selected papers (listed above) and some chapters or materials from these textbooks. All readings for this course are available online.

Policies

Attendance Policies

While in-person class attendance is not obligatory, a small percentage of the grade reflects our assessment of student participation in the course. There are many ways to participate in classes, including asking and answering questions in class, asking and answering questions online, actively participating in meetings with course staff during office hours and feedback meetings, being active during poster presentations, and writing a course reflection.

Extensions on Deadlines

Course staff will grant additional extensions in the case of serious and documented medical or family emergencies. We will consider other serious reasons, such as religious observances (job interviews and other schoolwork are not excuses for late homework), but students must inform course staff substantially in advance of the deadline. Apart from that, for homework assignments all students have three late days to use. After all late days have been exhausted, future delays will involve 10% score penalty for each day of delay.

Accommodation Statement

The University of Massachusetts Amherst is committed to providing an equal educational opportunity for all students.  If you have a documented physical, psychological, or learning disability on file with Disability Services (DS), you may be eligible for reasonable academic accommodations to help you succeed in this course.  If you have a documented disability that requires an accommodation, please notify me within the first two weeks of the semester so that we may make appropriate arrangements.  For further information, please visit Disability Services (https://www.umass.edu/disability/)

The use of Large Language Models (LLMs)

The use of LLMs is discouraged. However, LLMs can be useful for a limited set of tasks, especially to non-native speakers as a more sophisticated spell-checker to improve writing. Students are asked to disclose their use of LLMs briefly in 1-2 sentences. Students are responsible for all content they produce and submit.

Academic Honesty Statement

Since the integrity of the academic enterprise of any institution of higher education requires honesty in scholarship and research, academic honesty is required of all students at the University of Massachusetts Amherst.  Academic dishonesty is prohibited in all programs of the University.  Academic dishonesty includes but is not limited to: cheating, fabrication, plagiarism, and facilitating dishonesty.  Appropriate sanctions may be imposed on any student who has committed an act of academic dishonesty.  Instructors should take reasonable steps to address academic misconduct.  Any person who has reason to believe that a student has committed academic dishonesty should bring such information to the attention of the appropriate course instructor as soon as possible.  Instances of academic dishonesty not related to a specific course should be brought to the attention of the appropriate department Head or Chair.  Since students are expected to be familiar with this policy and the commonly accepted standards of academic integrity, ignorance of such standards is not normally sufficient evidence of lack of intent (http://www.umass.edu/dean_students/codeofconduct/acadhonesty/).

Inclusion Policy

In this course, each voice has something of value to contribute. Please take care to respect the different experiences, beliefs, and values expressed by students and staff involved in this course. We support UMass Amherst’s commitment to diversity, and welcome individuals of all ages, backgrounds, citizenships, disability, sex, education, ethnicities, family statuses, genders, gender identities, geographical locations, languages, military experience, political views, races, religions, sexual orientations, socioeconomic statuses, and work experiences.

The instructor reserves the right to modify this syllabus to account for current events and to better support student learning.