Build a retrieval-augmented generation system end to end: compile your own knowledge resource about Pittsburgh and CMU, chunk and index it, implement dense, sparse, and hybrid retrieval yourself, and generate answers to factual questions. There is no starter code and you collect the data yourself, so start early. Individual assignment, 100 points.

Everything is also in the hw2-public repo, including the LaTeX sources. Use public_evaluation_data.json — 157 questions with reference answers — as your development set and run your ablations against it.

The public leaderboard is live; its queries are in leaderboard_queries.json. Those queries differ from the public evaluation data, their reference answers will not be released, you have at most 10 submissions, and leaderboard scores do not count toward your grade. Your final system is evaluated on a private test set released Monday, October 26, one day before the deadline.

Two Gradescope submissions, both due October 27 at 11:59 PM ET:

  • Assignment2: Report — report.pdf, built from the ACL template, 7-page limit.
  • Assignment2: Code — a zip named for your lowercase Andrew ID:
ANDREWID/
├── github.txt
├── data/
│   └── train (optional)/
│       ├── questions.txt
│       └── reference_answers.json
├── system_outputs/
│   ├── system_output_1.json
│   ├── system_output_2.json (optional)
│   └── system_output_3.json (optional)
└── README.md

github.txt holds a link to a private GitHub repo with your code, shared with the TAs. Do not put report.pdf inside the zip; it is graded on its own Gradescope assignment. A code submission without system_outputs/ will be marked incomplete — your system outputs are what the results points are scored on.

Late submissions cost 5% per day (see the syllabus). If you plan to submit late, post on Piazza first — we release a different test set for each late day.