Schedule

Main readings:

• Natural Language Understanding with Distributed Representation (Ch. 1) (Cho 2015)
• Machine Learning: a Lecture Note (Ch. 1) (Cho 2025)

Main readings:

• Natural Language Understanding with Distributed Representation (Ch. 2, Ch. 3) (Cho 2015)

Main readings:

• Natural Language Understanding with Distributed Representation (Ch. 5 up to 5.4.2) (Cho 2015)

Main readings:

• Natural Language Understanding with Distributed Representation (Ch. 4, Ch. 5.5-5.6, Ch. 6) (Cho 2015)

Main readings:

• Attention Is All You Need (Vaswani et al 2017)
• The Annotated Transformer (Rush et al 2018)

Main readings:

• Neural Machine Translation of Rare Words with Subword Units (Sennrich et al 2016)
• From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models (Sections 1-3) (Welleck et al 2024)

Main readings:

• Language Models are Unsupervised Multitask Learners (Radford et al 2019)
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (Penedo et al 2024)

Main readings:

• Language Models are Few-Shot Learners (Brown et al 2020)
• Deep Learning Scaling is Predictable, Empirically (Hestness et al 2017)

Main readings:

• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al 2021)
• Sequence-Level Knowledge Distillation (Kim & Rush 2016)

Main readings:

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al 2022)
• From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models (Sections 4-7) (Welleck et al 2024)

Main readings:

• Retrieval-based Language Models and Applications (Asai et al 2023)
• Dense Passage Retrieval for Open-Domain Question Answering (Karpukhin et al 2020)
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al 2020)
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (Asai et al 2024)

Main readings:

• Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens (Liu et al 2024)
• Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index (Xu et al 2025)
• OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens (Liu et al 2025)

Main readings:

• Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (Miller 2024)

Main readings:

• Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (Miller 2024)
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al 2024)

Main readings:

• Denoising Diffusion Probabilistic Models (Ho et al 2020)
• Flow Matching for Generative Modeling (Lipman et al 2022)

Main readings:

• Deep Reinforcement Learning: Pong from Pixels (Karpathy 2016)
• Spinning Up in Deep RL (Part 1, Part 3, Vanilla PG, PPO) (OpenAI)

Main readings:

• Training language models to follow instructions with human feedback (Ouyang et al 2022)
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (DeepSeek-AI 2025)

Main readings:

• World of Bits: An Open-Domain Platform for Web-Based Agents (Shi et al 2017)
• WebGPT: Browser-assisted question-answering with human feedback (Nakano et al 2022)

Main readings:

• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Dettmers et al 2022)
• QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al 2023)
• The Ultra-Scale Playbook: Training LLMs on GPU Clusters (Tazi et al 2025)

Main readings:

• Deduplicating Training Data Makes Language Models Better (Lee et al 2022)
• Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (Swayamdipta et al 2020)
• Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (Dodge et al 2021)

Main readings:

• A Review of Sparse Expert Models in Deep Learning (Fedus et al 2022)
• OLMoE: Open Mixture-of-Experts Language Models (Muennighoff et al 2024)

Main readings:

• Self-attention Does Not Need O(n2) Memory (Rabe & Staats 2021)
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Gu & Dao 2023)

Main readings:

• Unsupervised Cross-lingual Representation Learning at Scale (Conneau et al 2020)
• mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (Xue et al 2021)
• When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages (Chang et al 2024)
• Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models (Ahia et al 2023)