Ducky

Inspiration

As high schoolers getting started with programming, we have all struggled with debugging. The classic rubber-ducky method is very useful because it allows developers to challenge their own ideas, but what if an intelligent agent could also provide helpful feedback? So, we had the idea for Ducky - an intelligent rubber-ducky-style program with a friendly interface!

What it does

Given code and a verbal user explanation/question (because talking always helps), Ducky generates a realistic video explaining the answer to the user’s question and refines their code.

How we built it

Firstly, we used HTML webkit to extract the user’s speech and convert it to text. Then, we sent the collected speech-text and given program code to Vectara for retrieval augmented generation and then to SambaNova, where an agentic pipeline handles the query. Finally, a Heygen video is generated and displayed to the user with the pipeline's response.

Challenges we ran into

There is no official Vectara API - we had to figure out how to use raw Python requests to mimic the API and still query our RAG data. Also, we struggled with deploying Heygen as the quickstart provides use cases for React apps only - and we created a website. We eventually got around this by using frames and specialized request calls.

Accomplishments that we're proud of

We did our best and we explored new technologies. We created an end-to-end speech-to-speech interactive experience with an embedded agent pipeline, and learned a lot about new tools and methods along the way.

What we learned

We learned all about the amazing new sponsor technologies at the hackathon. We also learned the theory behind agentic pipelines and how LLMs can solve complex problems.

What's next for Ducky

We will integrate Exa in order to allow our model to scrape the web and collect code snippets. Additionally, we plan to make Ducky a VS-code extension to make it easier for users to access!