Pokey

GIF
Pokey

Inspiration

We were inspired by the early iterations of desktop assistants, such as Clippy and Bonzai Buddy. We wanted to bring that fun personalization back to the desktop while also leveraging the power of new virtual assistants. In matching with the Cyberpunk theme of this event, the models of Pokey were based off of the character Skippy from Cyberpunk 2077, which itself is derivative of Clippy.

What it does

Pokey is a desktop assistant designed to extend Google Gemini's capabilities with local interactions on the user's machine, in an attempt to create a more fun version of the Windows search bar. Thus, Pokey doubles as a local Gemini interface and automation tool.

How we built it

The primary component of Pokey is a tkinter application window that takes user input and passes the user's request in a custom-engineered prompt to Google Gemini by API. The prompt tells Gemini to match the user's request to a list of preset function prototypes, but also allows for Gemini to respond to any general questions that do not match the prototypes. Once the response has been generated, we manipulate the widgets of our tkinter window, use a speech synthesis module, and create a new window for text display, all in order to emulate early desktop assistant interaction and give Pokey's communication a more personal feel.

We did leverage Copilot's integration with Visual Studio Code in order to speed up prototyping, as well as investigate some errors in our usage of tkinter.

Challenges we ran into

There was a lot of considerable challenges with how we handled the actual visual display of Pokey. Notably, we had to wrangle the built-in tkinter module of Python and configure the window in order to give the effect that Pokey is a free-standing entity on the desktop. We also were lacking in Blender knowledge, so the creation of the models did take a little longer than intended.

Other than that, a time constraint combined with lacking group experience in used modules did create a time-sink of learning the components properly in order to implement them.

Accomplishments that we're proud of

Despite the increased time cost, we are very proud of the models that we animated for Pokey. We are also proud of the practicality and potential extensibility of the framework that we have created. The idea moving forward would be to create more functionality that can be run locally on the machine by the Gemini call. Permitted time, a user could extend Pokey with whatever functionality they would like.

What we learned

We learned some prompt engineering, as well as some useful information about web requests, in order to integrate the Gemini API with Pokey.

What's next for Pokey

There were a few features that we were not able to implement for a lack of time. These would take precedent in future development of Pokey.

Notably:

Include the already-created pokey-sad.gif to animate whenever an error occurs, such as running out of API calls.
More configuration options, such as being able to change the personality and GIFs used for rendering
A precise overview of system statistics (resource usage, temperatures. This feature was in beta but we decided to cut it due to instability.)
File access (This feature would need much more time to make stable, but the privacy implications of sending file data to Google would turn this into an "opt-in" feature.)
Custom-trained voice model (also swappable by config)
Dynamic movement settings (to allow Pokey to wander around!)
Styling the input and output text boxes, as well as add a visual indicator for when speech input is being recorded.