Milestone Papers

DeepSeek-R1 86,319

(2025-1) DeepSeek-R1 by DeepSeek

Qwen2.5 Technical Report

(2024-12) Qwen2.5 by Alibaba

The Llama 3 Herd of Models

(2024-05) Llama3 by Meta

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

(2024-05) Mamba2 by CMU&Princeton

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

(2024-01) DeepSeek-v2 by DeepSeek

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

(2023-12) Mamba by CMU&Princeton

Mistral 7B

(2023-10) Mistral 7B by Mistral

Llama 2: Open Foundation and Fine-Tuned Chat Models

(2023-07) LLaMA2 by Meta

Tree of Thoughts: Deliberate Problem Solving with Large Language Models

(2023-05) ToT by Google&Princeton

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

(2023-05) DPO by Stanford

RWKV: Reinventing RNNs for the Transformer Era

(2023-05) RWKV by Bo Peng

PaLM 2 Technical Report

(2023-05) PaLM 2 by Google

Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

(2023-05) Dromedary by CMU et al.

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

(2023-04) Pythia by EleutherAI et al.

Visual Instruction Tuning

(2023-04) LLaVA by UW–Madison&Microsoft

GPT-4 Technical Report

(2023-03) GPT 4 by OpenAI

PaLM-E: An Embodied Multimodal Language Model

(2023-03) PaLM-E by Google

Resurrecting Recurrent Neural Networks for Long Sequences

(2023-03) LRU by DeepMind

Language Is Not All You Need: Aligning Perception with Language Models

(2023-02) Kosmos-1 by Microsoft

LLaMA: Open and Efficient Foundation Language Models

(2023-02) LLaMA by Meta

The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

(2023-01) Flan 2022 Collection by Google

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization

(2022-12) OPT-IML by Meta

Galactica: A Large Language Model for Science

(2022-11) Galactica by Meta

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

(2022-11) BLOOM by BigScience

Holistic Evaluation of Language Models

(2022-11) HELM by Stanford

GLM-130B: An Open Bilingual Pre-trained Model

(2022-10) GLM-130B by Tsinghua

Scaling Instruction-Finetuned Language Models

(2022-10) Flan-T5/PaLM by Google

Improving alignment of dialogue agents via targeted human judgements

(2022-09) Sparrow by DeepMind

Language Models are General-Purpose Interfaces

(2022-06) METALM by Microsoft

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

(2022-06) BIG-bench by Google

Emergent Abilities of Large Language Models

(2022-06) Emergent Abilities by Google

Unifying Language Learning Paradigms

(2022-05) UL2 by Google

OPT: Open Pre-trained Transformer Language Models

(2022-05) OPT by Meta

An empirical analysis of compute-optimal large language model training

(2022-04) Chinchilla by DeepMind

PaLM: Scaling Language Modeling with Pathways

(2022-04) PaLM by Google

Training language models to follow instructions with human feedback

(2022-03) InstructGPT by OpenAI

Using Deep and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

(2022-01) Megatron-Turing NLG by Microsoft&NVIDIA

Solving Quantitative Reasoning Problems with Language Models

(2022-01) Minerva by Google

LaMDA: Language Models for Dialog Applications

(2022-01) LaMDA by Google

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

(2022-01) COT by Google

Scaling Language Models: Methods, Analysis & Insights from Training Gopher

(2021-12) Gopher by DeepMind

Improving language models by retrieving from trillions of tokens

(2021-12) Retro by DeepMind

WebGPT: Browser-assisted question-answering with human feedback

(2021-12) WebGPT by OpenAI

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

(2021-12) GLaM by Google

Multitask Prompted Training Enables Zero-Shot Task Generalization

(2021-10) T0 by HuggingFace et al.

Finetuned Language Models are Zero-Shot Learners

(2021-09) FLAN by Google

On the Opportunities and Risks of Foundation Models

(2021-08) Foundation Models by Stanford

Evaluating Large Language Models Trained on Code

(2021-08) Codex by OpenAI

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

(2021-01) Switch Transformers by Google

Language models are few-shot learners

(2020-05) GPT 3.0 by OpenAI


1 2