Dual Wield

DualWield

Inspiration

Education is a core value for many. Yet we all know how addictive games can be—and we’ve all sat through those dead moments: you’ve just died, you’re spectating, or you’re idling in a lobby waiting to queue. Rather than letting that downtime feel frustrating, we treat it as a prime window to drop concise knowledge and help you prep for exams. Gamers have logged over 2.85 million years into video games and our mission is to convert every micro‑pause of that into hyper‑efficient progress.

What it does

At a high level: you load in the study material you want to review (videos, PDFs, etc.), then jump into your game. The app detects natural gameplay downtime and, in those moments, seamlessly surfaces bite‑sized chunks of that material.

The pipeline is as follows:

You first gather and select the study materials for the session (pdfs and videos).
We parse, segment, and analyze that content.
While you play, we detect natural downtime and inject concise voiceover explanations.
Post‑session, we synthesize everything covered and give you a comprehensive review/test to reinforce retention and surface weak spots.

How we built it

There are many moving components to this app. We use an electron front end, with a python/flask backend.

For data collection, we use a chrome extension to inject code to ur browser to detect whether you have a pdf or youtube video open. If you do, it send this information to a flask backend. This updates internal states of the app and the UI accordingly.

For video inputs, we support processing youtube videos. We download the video using yt-dlp, and merge separate audio/video streams using FFmpeg. Finally, once the video is downloaded, we use the TwelveLabs API to parse video inputs.

Once we have aggregated all the data for a session, we use the Gemini API to format the data into a JSON schema, which returns a pre-dictated data type that we can use to better structure our output. We use the Vellum SDK to assist us in refining our methodology and steps of this process.

During Gameplay, the electron app constantly monitors your screen, capturing 10 frames a second, and feeds your contents into a Tesseract OCR model to try detecting whether you are in downtime.

If you are in downtime, we use ElevenLabs API to read out the contents of the processed input data for you. Once you are playing again, the assistant stops teaching, and will resume during the next down time.

Lastly, once the session ends, we utilise the Ribbon API to consolidate all the material taught, and quiz you regarding what has been taught.

More detailed documentation regarding each component can be found in our github repo, and attached are links for easier navigation. Twelve labs API integration: https://github.com/WhyILived/HT6ix/blob/main/documentation/Twelve_labs.md Ribbon API integration: https://github.com/WhyILived/HT6ix/blob/main/documentation/Ribbon_ai.md Vellum SDK usage: https://github.com/WhyILived/HT6ix/blob/main/documentation/Vellum_ai.md OCR for downtime detection: https://github.com/WhyILived/HT6ix/blob/main/documentation/Game_Management.md Youtube download process: https://github.com/WhyILived/HT6ix/blob/main/documentation/Youtube_download.md

Challenges we ran into

Some of the many challenges we ran into as a team include:

Determining a seamless way to pull relevant study content automatically; we opted to inject lightweight browser code to detect the active resource/context without manual user input, allowing users to add content simply with a press of a button.
Ensuring coordination between the real‑time game/downtime detector UI and the backend services generating summaries and voiceovers.
Stitching together components (ingestion, NLP processing, adaptive quiz engine, voice synthesis, and delivery layer) into a reliable, low‑latency end‑to‑end flow.

Accomplishments that we're proud of

Having a working product, since there are many components and integration did not seem obvious. Getting all the moving parts to interoperate smoothly under hackathon time pressure was a major win.

We're also proud of being able to build a low latency downtime detector that distinguishes active gameplay from idle states.

What we learned

We learned how to build an electron app, use various APIs for the first time, and were exposed to many new technologies.

What's next for Dual Wield

There’s a long list of features we still want to add. For example, we didn’t get time to implement in‑game quiz bursts during idle states. We also want a more robust, generalized idle‑state detector (works across more game genres and edge cases) and additional input methods for the user.