Machine Behavior
Spring 2025 — COS 598B
Machine learning models are everywhere, and their role in society may increase as they become more popular and influential. At the same time, recent work has shown that LLMs can simulate and predict human behavior remarkably well. Thus, understanding and steering the behavior of such systems can amplify their benefits, mitigate their harms, and increase our understanding of human behavior. This seminar course aims to facilitate publishable student research on these broad topics. Coursework is a mix of readings and a research project.
Table of Contents
- Basic Information
- Schedule
- About the course
- Detailed Schedule
- Acknowledgments
2. Schedule
3. About the course
This course has three central components: 1) reading activities, 2) a research project, 3) guest lectures
3.1 Reading activities
TL;DR: For each class, all students write a short commentary (“reading response”) to assigned readings, and some students take turns as “discussants,” summarizing their peers’ reading responses.
3.1.1 Reading response
- Each day, we will have 1 to 2 assigned readings;
- Everyone should write a short response for each.
- This should be done as a slide to be added to a collective deck.
- This should be done at least 24 hours before the class.
- Your response should reflect on a couple of the following questions:
- What are the contributions of the paper?
- How would you extend this work?
- Do you disagree with any of the authors’ methodological decisions?
- What connections did you find between this work and your own?
- Did you gain any insights (directly or indirectly) by reading this paper?
- Do you agree with all the assumptions made in the paper?
3.1.2 Discussants
- We will have one or two discussants per assigned reading. They should:
- Read the assigned readings like everyone else.
- Synthesize the paper and the reading responses in 3 to 5 slides.
- Present (~20 min) the synthesis you prepare.
- Co-host (~30 min) a discussion in the class. Discussants should jointly prepare prompts to spur a discussion in the class.
3.2 Research Project
TL;DR: Along with a team, you will conduct original research and write a paper summarizing your project.
- Scope. A central component of the seminar is a research project. For example, your project might 1) use LLMs to simulate human behavior, 2) examine the behavior of machine learning models, or 3) analyze the interaction between humans and machine learning models. Above all, the project should be about a topic that interests you, e.g., something you find useful or that may contribute to your dissertation.
- Teams. Team formation will be flexible, and the project scope will be commensurate with the team size. The final paper will include a “credits” section describing how each group member contributed to the project.
- Process. We will have milestones, informal presentations, and feedback throughout the semester. Specifically, we will give peer feedback on projects throughout the semester simulating a single-blind peer-review process. I will also meet with you outside the class to help with your project.
- Outcome. Your team should write a research paper summarizing the projects with the typical sections, e.g., Intro, Related Work, Methods, Results, and Discussion. It should have around 5,000 to 7,000 words. You may tailor the paper to a specific venue you want to target for publication (e.g., ACL venues, ICLR, CoLM), and the instructor can help students think about whether and where to submit their project.
3.2.1 Deliverables
- The project has five different deliverables, all to be done in groups:
- Project proposal.
- Description: Two-pager on: 1) what your intended project is; 2) why is it relevant, and 3) how you are going to do it.
- Deadline: Feb 20.
- Submission link
- Brief synthesis of relevant related work.
- Description: Two-pager. Select ~3 papers relevant to your project and 1) summarize and their contributions; 2) discuss their limitations; 3) describe how your envisioned work differs from/expands prior work.
- Deadline: Mar 06.
- Submission link
- Project clinic presentation.
- Description: Short presentation discussing what your group has accomplished and, most important, the roadbloacks you are currently facing.
- Deadline: Mar 25.
- Submission: Add slide to this drive folder.
- Project final presentation.
- Description: Short presentation of your project.
- Deadline: Apr 22nd.
- Submission: Add slide to this drive folder.
- Due Apr 24th: Final project rep dort.
- Description: Final report in the format of a paper. The report should have around 8 pages and 4,000-8,000 words, and should be structured like a paper. You must include a `contributions’ section outlining what group participant did what. I encourage you to link a github repo with the code you used for the project within the manuscript.
- Deadline: Apr 24nd.
- Submission link.
3.2.2 IRB
- Additionally, you may also need to apply for an IRB. I will help you with this!
3.3 Guest Lectures
After spring break, we will have a series of guest lectures. Students are expected to attend and meaningfully engage with guest speakers.
3.4 Grading
- 10% In-class active participation.
- 10% Discussant presentation.
- 20% Reading responses.
- 50% Research project:
- 35% Final write-up.
- 15% Final presentation.
3.5 Expectations
- I expect you to:
- Attend and actively participate in class.
- Be respectful and collegial to your classmates and guests.
- Complete readings early and submit responses on time to help discussants.
- Present your work when the time comes and serve as a discussant when necessary.
- Deadlines. Deadlines exist to help the class run smoothly. However, if you have any extenuating circumstances, please contact me about whether and how you can receive an extension. You must be proactive in letting me know so that we can plan together and others are not disrupted.
- A note on diversity and respectful conduct. This course welcomes all students of all backgrounds. You should expect and demand to be treated by your classmates and myself respectfully. If any incident challenges this commitment to a supportive, diverse, inclusive, and equitable environment, please let me know so the issue can be addressed.
- Disability, Religious, and family accommodations. If you have any questions about disability or religious accommodations, please refer to university policies. Feel free also to contact me for any reason.
- Academic integrity. We will follow the University’s rules and responsibilities guide. Also, if you need IRB approval, we can work together to apply for it early!
4. Detailed Schedule
Week #1 — Introduction (Pre-read)
Jan 28
- General plan:
- Introductions.
- Go over the plans for the seminar.
- Quick overview of readings.
- Set expectations: things might change based on your feedback.
- Project:
- Brainstorming areas of interest.
Jan 30
- Readings:
- Rahwan et al. “Machine behaviour.” Nature (2019) (link)
- Wagner, Claudia, et al. “Measuring algorithmically infused societies.” Nature (2021) (link)
- Project:
- Brainstorming areas of interest.
Week #2 — Simulations (Pre-read)
Feb 04
- Readings:
- Argyle, Lisa P., et al. “Out of one, many: Using language models to simulate human samples.” Political Analysis (2023). (link)
- Messeri, Lisa, and M. J. Crockett. “Artificial intelligence and illusions of understanding in scientific research.” Nature 627.8002 (2024): 49-58. (link)
- Project:
- Brainstorming areas of interest.
Feb 06
- Readings:
- Hu, Tiancheng, and Nigel Collier. “Quantifying the persona effect in LLM simulations.” ACL 2024. (link)
- Wang, Angelina, Jamie Morgenstern, and John P. Dickerson. “Large language models should not replace human participants because they can misportray and flatten identity groups.” ArXiv preprint 2024. (link)
- For fun:
- Project:
- Brainstorming areas of interest.
Week #3 — LLMology
Feb 11
- Readings:
- Binz, Marcel, and Eric Schulz. “Using cognitive psychology to understand GPT-3.” Proceedings of the National Academy of Sciences 120.6 (2023): e2218523120. (link)
- Project:
- Brainstorming areas of interest.
Feb 13
- Readings:
- Santurkar, Shibani, et al. “Whose opinions do language models reflect?.” International Conference on Machine Learning. PMLR, 2023. (link)
- Project:
- Brainstorming areas of interest.
Week #4 — Interlude
Feb 18
Feb 20
- Project:
- Project proposal: Students present project proposals and receive feedback.
Feb 25
- Readings:
- Guess, Andrew M., et al. “How do social media feed algorithms affect attitudes and behavior in an election campaign?.” Science 2023. (link)
- Wagner, Michael W. “Independence by permission.” Science 2023. (link)
- Project:
Feb 27
- Readings:
- Haroon, Muhammad, et al. “Auditing YouTube’s recommendation system for ideologically congenial, extreme, and problematic recommendations.” PNAS (2024) (link)
- Hosseinmardi, Homa, et al. “Causally estimating the effect of YouTube’s recommender system using counterfactual bots.” PNAS (2024). (link)
- Project:
Week #6 — Humans and Machines
Mar 04
- Readings:
- Costello, Thomas H., Gordon Pennycook, and David G. Rand. “Durably reducing conspiracy beliefs through dialogues with AI.” Science 385.6714 (2024). (link)
Mar 06
- Readings:
- Krügel, Sebastian, Andreas Ostermaier, and Matthias Uhl. “ChatGPT’s inconsistent moral advice influences users’ judgment.” Scientific Reports (2023). (link)
Week #7 — Guest Lectures
Mar 18
Mar 20
Week #8 — Project Clinic
Mar 25
- Project:
- Project clinic: Students present projects and get feedback.
Mar 27
- Project:
- Project clinic: Students present projects and get feedback.
Week #9 — Alignment
Apr 01
- Readings:
- Shen, Hua, et al. “Towards Bidirectional Human-AI Alignment: A Systematic Review for Clarifications, Framework, and Future Directions.” arXiv preprint arXiv:2406.09264 (2024). (link)
Apr 03
- Readings:
- Zhi-Xuan, Tan, et al. “Beyond preferences in ai alignment.” Philosophical Studies (2024): 1-51.
. (link)
Week #10 — Guest Lectures
Apr 08
Apr 10
Week #11 — Guest Lectures
Apr 15
Apr 17
Week #12 - Final Presentations
Apr 22
- Project:
- Final presentations: students present their project and submit their report.
Apr 24
- Project:
- Final presentations: students present their project and submit their report.
5. Acknowledgments
I used the following seminar courses as references to help structure this: