Projects

In my opinion, the best way to do a grad-level project course is to use it as an opportunity to make progress on research on a related topic, especially if it aligns with your existing research agenda. I see the best outcome of the course for you to be getting as close to a publishable research artifact as possible, given the timeframe of the course. With that in mind, you will be expected to submit a 6-9-page research paper (following the NeurIPS style guide) and give a presentation on your work. Note that the NeurIPS submission deadline is 5/22, which is also the due date for the class paper.

That said, we don’t require you to produce a NeurIPS-tier publication in a 10-week course. The goal is for you to get research experience on a topic that you’re excited about which relates to the course. Expectations for the novelty, significance, and scale of the contribution and results of your paper will be scaled back dramatically relative to a full NeurIPS paper, and are more akin to a workshop paper. As examples of smaller-scale projects that would fit with the class, you could start with replicating an existing method, and apply it to a new experimental domain or dataset. You could try applying your favorite single-agent RL method in a multi-agent setting, and see if it needs any modifications to allow it to work.

Note that it is totally acceptable, and in fact ideal, if your normal research aligns with the class, and you can continue making progress on your research agenda and write up the results for the class project. You are also encouraged to form groups of up to four students for the project. Through these avenues, you may be able to tackle more challenging research problems with more novelty, such as developing a new algorithm or technique. We encourage you to pursue this if you think it’s possible!

Possible topics

Just like a Call for Papers (CfP), here is a list of suggested topics appropriate for the class project:

• Multi-agent RL
• Coordination with other agents
• Cooperation with humans
• Population-based training
• Theory of mind in multi-agent systems
• Emergent complexity
• Curriculum generation
• Interpreting multi-agent systems
• Social Learning for AI
• Learning from human feedback
• Inverse RL
• Language-conditioned RL
• RLHF
• RLAIF / Multi-agent LLMs / LLM debate

If you would like to do a project on a topic you think is related that you don’t see listed here, please contact us.

How will the project be evaluated?

Peer review Your paper will be evaluated via peer review. You will submit your paper to a conference reviewing platform like OpenReview (exact platform TBD) which we will set up. Each student in the class will be assigned to 3 papers and will be asked to provide reviews on those papers by 11:59pm on May 31. The review procedure will be single-blind, since students will have seen the class project presentations. 15% of your final grade depends on the quality of the three peer reviews you submit. Reviews will be graded based on whether they are thorough, complete, fair, and whether they give the authors useful feedback for improving their paper. Examples of how to write good reviews of machine learning papers can be found online, for example here. Your final grade will not be based solely on peer reviews. Rather, your instructors will act in a role similar to an Area Chair. They will review the paper and the reviews and make a final decision based on both.

Criteria Your paper will be evaluated similarly to how research papers are evaluated for conferences: on the novelty, significance, soundness of the experimental or theoretical results, writing/presentation, and coverage of the related work. However, the degree of novelty, significance, etc. is expected to be less than a full conference paper (potentially more similar to a workshop paper).

Deliverables

For all project deliverables, only one member should submit on Gradescope on behalf of the entire group. Please list all group members in your submission. The same member should consistently be the one submitting for the group.

Proposal (5% of total grade, due 11:59pm April 17)

• Maximum 1 page. Consider this to be a 1-pager, a useful document to sell the value of your project quickly to potential collaborators. It’s good to get practice writing this kind of document, as it’s often used in research and in industry.
• Describe the proposed project. What is the idea? What is the central contribution relative to related work? What experiments do you plan to run? Why would it be valuable to complete this project? What will the impact be?
• Submit as a typeset PDF on Gradescope.
• Your instructors will provide feedback on the proposal which will be designed to help steer your project towards a feasible, interesting, and ideally impactful direction.

Presentation (10% of total grade, due in class on May 22 or 29)

• Students will give a presentation about their project to the class. Exact times alloted for the presentations are TBD, as they will depend on how many projects we need to cover across 2 classes.
• The goal of the presentation is to convey the important high-level ideas and takeaways of your project.
• All group members should participate in the presentation.
• To minimize time spent switching computers, provide a link to your presentation in the schedule spreadsheet by 11:59pm the day before you must present. We will choose which group presents when after the project proposals have been submitted and reviewed.

Writeup (35% of total grade, due 11:59pm May 22) Please submit a 6-9 page paper structured like a typical research paper, following the NeurIPS style guidelines (page limit applies only to the content, not the references or appendices). Suggested sections include the abstract, introduction, related work, (optional: technical preliminaries, which explain the technical details behind existing methods that your work builds on), methods (the technical details of what you’re proposing), experiments (describing the experiments you will run, environments you used), results, and conclusion/discussion. These are not a hard requirement; you may also have theoretical results you wish to include. Note also that this standard format can sometimes be tweaked effectively.

At the end of your writeup, include a brief Statement of Contributions section that details the contributions made by each group member (for an example see p. 16 of this paper). This section does not count towards the page limit.

Create a github repository for any code related to your project, and include a link to the repo in your paper (if you are submitting the same paper to NeurIPS, remember to anonymize or remove this link!). The repository should include all relevant code needed to reproduce the results. It should also come with a README file documenting what commands you ran.

Note that your submission should not be anonymized, and should include group member’s names as authors. This is because we will scale expectations down for papers with fewer authors, so reviewers need to know how many authors contributed to the paper.

Compute

Students working with a research lab can use their lab resources, if feasible. Otherwise, options include:

• Using Google Colab
• There are dedicated compute resources available for this class on the CSE unagi server. You can SSH-ing to unagi.cs.washington.edu, which is administered by CSE. This has two GPUs (RTX 3080ti). You may store materials in the “/local1” folder.

Please approach the instructor in-case the available compute is a bottleneck for your projects.

Guidelines for authors

Students are encouraged to form groups of up to 4 students. The expectations for the project will scale with the number of students, such that projects with more students are expected to be more impactful. To be concrete, we expect roughly that the length of the report should also scale with the number of students:

• 1 person project -> 6 pages
• 2 people -> 7 pages
• 3 people -> 8 pages
• 4 people -> 9 pages

It’s okay to go over these heuristic limits, but do not submit a report longer than 9 pages for any group size. Please include a statement of contributions in the appendix of the paper describing what each person did. However, don’t be discouraged from teaming up into groups! In the ideal case, you can aim to write an impactful research paper that could potentially be submitted to NeurIPS, and you may need more students to pull this off.

Environments

For your paper, you may want to run experiments using multi-agent environments. For an informal list of such environments, please see this list of possible multi-agent environment. If you have ideas for further environments or datasets, feel free to leave a comment suggesting them.

• Starcraft (SMAC) https://arxiv.org/pdf/1902.04043.pdf
• Highway driving env: https://github.com/eleurent/highway-env
  • Experiments are quick
  • Discrete action space
• Melting pot 2.0
  • https://github.com/deepmind/meltingpot
  • Includes social dilemmas
  • May include other agents to test “social generalization” (ability to play with agents not experienced during training)
• The hanabi challenge: A new frontier for ai research
• Social Simulacra: Creating Populated Prototypes for Social Computing Systems
• Multi-agent gridworlds based on minigrid:
  • https://github.com/google-research/google-research/tree/master/social_rl/gym_multigrid written by Natasha
  • https://github.com/kandouss/marlgrid written by Kamal Ndousse, likely has more performance optimization
• SocialAI School: https://arxiv.org/abs/2307.07871
  • multigrid environments with social tasks (a scripted confederate points to what box you need to open or demonstrates a task)
• Overcooked from Micah & Rohin (Berkeley)
  • https://github.com/HumanCompatibleAI/overcooked_ai
  • https://bair.berkeley.edu/blog/2019/10/21/coordination/
  • For human-AI coordination (ask Yancheng or Daphne for tips)
• minecraft open-ended construction challenge environment: https://twitter.com/risi1979/status/1337000376353087490?s=20. https://arxiv.org/abs/2012.04751.
• VirtualHome-Social, a visually realistic helping environment from paper:
• watch-and-help: a challenge for social perception and human-ai collaboration http://virtual-home.org/watch_and_help/
  • Human-AI collaboration benchmark env
  • https://arxiv.org/abs/2010.09890
  • Multi agent env
• baby intuitions benchmark https://arxiv.org/abs/2102.11938
• Minecraft envs in google: https://arxiv.org/abs/2002.11089
• Minecraft environments in open-source?
  • The code for the ICLR 2020 paper is available at https://github.com/srsohn/msgi.
• Behavior suite for RL https://arxiv.org/pdf/1908.03568.pdf
• Google research football
• neural MMO
  • Project Page: https://jsuarez5341.github.io
  • v1.3 Whitepaper: https://jsuarez5341.github.io/neural-mmo/bui

Includes language

• HandMeThat (ambiguous language instructions): https://arxiv.org/abs/2310.03779
• Collaborative Construction and Communication in Minecraft http://juliahmr.cs.illinois.edu/Minecraft/
• MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks https://arxiv.org/abs/2109.06275
  • Dataset of 100 games where 2 players collaborate on tasks in Minecraft by chatting with each other and periodically answering questions about their beliefs about the state of the world, including what they think their partner thinks
• Simulcra; A sims-like toy environment? https://arxiv.org/pdf/2304.03442.pdf
  • “​​we describe an architecture that extends a large language model to store a complete record of the agent’s experiences using natural language, synthesize those memories over time into higher-level reflections, and retrieve them dynamically to plan behavior. We instantiate generative agents to populate an interactive sandbox environment inspired by The Sims, where end users can interact with a small town of twenty five agents using natural language”
  • Similar vibes, maybe better evaluation” https://drive.google.com/file/d/1Y0f7qovlfNR6g4Z0LG32HTnY7jvEYeR9/view
• Room across room (vision and language navigation): https://arxiv.org/abs/2010.07954
• Multi-agent LLM conversations: https://github.com/microsoft/autogen?tab=readme-ov-file

Datasets

• Human actions in a kitchen: https://proceedings.neurips.cc/paper_files/paper/2022/file/5985e81d65605827ac35401999aea22a-Paper-Datasets_and_Benchmarks.pdf