tpg-onan: Previous-Token Prediction

Title: tpg-onan
Tokens predict in opposite order
With both models, you can predict text in two directions at once!

Inspiration

Back in 2022, ChatGPT was lots of fun to use. You'd ask it questions like "Are you conscious?", "What's your favorite Hamlet quote?", and "How can I tie my shoes?". Harmless nonsense - nobody had ever seen something like this before!

Generative AI is a great tool for research and work, but it's lost that original sparkle. I wanted to create a fun tool that captures the magic we all remember, and put an interesting spin on our generation's most important technology.

What it does

tpg-onan is a GPT-nano model completely pre-trained from scratch, but with one change: it performs last-token prediction, not next-token prediction. You give it an input and it predicts what came before. A separate forward-facing model predicts what comes next. Now, you can have bi-directional text prediction!

The user simply inputs text, like with language models today, and text is generated on both sides. The user can toggle for forwards-only or backwards-only prediction. A handful of demo-questions are curated that allow the user to inspect differences in behavior related to prose, mathematics, and reasoning.

How we built it

I used two cloud GPUs and Karpathy's nanochat GitHub project (https://github.com/karpathy/nanochat) to pretrain two separate GPT-nano models in a few hours.

I modified the repository's pretraining code to use previous-token prediction, instead of next-token prediction. The entire project costed less than $30 to train, less than a week's worth of my Gregg's addiction. One model goes forward (next-token prediction), the other model goes backwards (last-token prediction). Both were trained with 20 layers of depth. Hyperparameters were kept as defaults. The website was written in plain JavaScript and HTML.

Accomplishments that we're proud of

I've always wanted to use Karpathy's nanochat for something, but never had a suitable project until now. The complete training of the model - from pretraining to inference - was done during the course of the Hackathon. It's immensely satisfying!

I'm also in love with the effect that the two models together produce. It's incredibly fascinating to see how the model works differently when it works backwards, and to compare that answer with what happens when it works forwards.

What comes next

There are interesting scientific questions that my model can poses. For example, does this model perform better on problems where working backwards is the optimal strategy?