One Attention Head is all you Need

• 

  attention heat map matrix

• 

  loss and accurarcy graph

Inspiration

Given we are in the era of LLMs, we both taught it would be a fun idea to challenge ourselves with a lower-level problem to solve.

What it does

How we built it

a google colab notebook

Challenges we ran into

Generated the

Accomplishments that we're proud of

We were able to watch and be guided by a researcher from a company who built a major LLM model.

What we learned

We learned about Transformer lens.

What's next for One Attention Head is all you Need

• More research

Reference

• (https://docs.google.com/spreadsheets/d/1oOdrQ80jDK-aGn-EVdDt3dg65GhmzrvBWzJ6MUZB8n4/edit#gid=0)[200 Concrete Problems In Interpretability Spreadsheet - Google Sheets]
• 200 Concrete Open Problems in Mechanistic Interpretability: Introduction — AI Alignment Forum
• One Attention Head Is All You Need for Sorting Fixed-Length Lists by MatthewBaggins (itch.io)

Built With

• torch