vintagellms.com
~ * ~ language models trained on the past ~ * ~
Reading List
- Vintage Large Language Models
— Owain Evans
- Transcript of a talk on training language models exclusively on text from a
chosen period — 2019, 1900, 200 AD. Argues the payoff is twofold: forecasting
and re-derivation of inventions on the one hand, and historical inquiry into how
knowledge actually accumulated on the other. Also covers the practical obstacles,
mainly data availability and training cost.
- Vintage LLMs (Historical language models)
— Owain Evans, video
- The recorded talk behind the transcript above.
- Introducing talkie: a 13B vintage language model from 1930
- Announcement of a 13-billion-parameter model trained only on English text
published before 1931. Describes what the model can do, what made training it
difficult, and plans to scale vintage models up.
- A Model of the Language Process
— Calcifer Computing
- Introduces TLM-1, a transformer trained to predict a document's content and
its date jointly, then uses it to trace how American English shifted between
1990 and 2019.
- Time Masking for Temporal Language Models
— Rosin, Guy & Radinsky, 2021NEW!
- Introduces TempoBERT, which folds time metadata into the text itself and
trains with a masking scheme built around it. Evaluated on detecting semantic
change and on guessing when a sentence was written. An earlier academic run at
the idea TLM-1 above develops.
- Synthetic Pretraining
— Pierre-Carl Langlais
- Argues pretraining is moving off scraped web text and onto deliberately
designed synthetic corpora, in three stages: rephrasing for memory, hardwiring
formal reasoning, and simulating agent behaviour.
UNDER CONSTRUCTION