Simone

Inspiration

Fun LLM challenge that I'm thinking about: take my 2h13m tokenizer video and translate the video into the format of a book chapter (or a blog post) on tokenization. Something like:

1. Whisper the video
2. Chop up into segments of aligned images and text
3. Prompt engineer an LLM to translate piece by piece
4. Export as a page, with links citing parts of original video

More generally, a workflow like this could be applied to any input video and auto-generate "companion guides" for various tutorials in a more readable, skimmable, searchable format. Feels tractable but non-trivial.

-- Andrej Karpathy, X/Twitter, Feb 2024

This is what inspired to start this project. I wanted to create a FOSS solution to this challange.

What it does

Simone is a Python-based CLI tool that converts your YouTube videos into blog posts with context-aware screenshots.

How I built it

The blog post is created using Gemma 7B which also helps in generating keywords. The video is then matched against these keywords and each frame is rated based on it. In the end, the top 3 frames are saved as keywords.

Challenges I ran into

Finding a reliable LLM which can be used for free for development.

Accomplishments that I'm proud of

Completing this in such a short time.

What I learned

A bit more about the Tesseract OCR and how to use it in development and production.

What's next for Simone

1. Platform neutrality
2. Availability of multiple LLMs

Built With

• gemma
• ocr
• opencv
• python
• tesseract