Aurora GeminiLit

Inspiration

We were inspired to build this for professionals who have little to no coding experience but want to create beautiful, insightful, and customizable dashboards quickly. No drag and drop, no learning a new UI, just drop in your data and tell Aurora GeminiLit what you want. This will allow non-technical people such as marketing associates to leverage data more easily, without ever needing to install Python!

What it does

Aurora GeminiLit allows users to drop in a CSV file containing a dataset. It then uses the Gemini API to examine the data, ask the user questions about their intentions and once it understands what needs to be done, generates a Streamlit dashboard. Once generated, the dashboard is validated using Gemini to run the code and examine any errors. The code is iteratively improved and, finally, the dashboard server is launched. The UI then renders the dashboard in the same window as the chat. The user can then ask to iterate in any way they want. For example, rename the dashboard, add graphs or interactive elements or add more parameters to certain graphs. The user can then cycle through all the past revisions as needed.

How we built it

• We used Python with FastAPI to create a webserver. We expose a WebSocket API, which allows the client to connect. We decided on WS instead of REST because it allowed us to easily maintain a session with the user's conversation with Gemini (1.5), the generated code and other metadata, without the constant need for a persistence layer (although we can use one to store dashboard between browser sessions)
• We used Pandas to load the data and store a copy of it on the server (for now, a bucket is more appropriate for production).
• Each time the client sends a message, the server processes it accordingly. For bytes types (file upload), we read the file and extract a preview. The preview is passed to Gemini (1.5) and it is prompted to ask the user questions to understand it better. For text and JSON, we either pass it to Gemini to process the user's query or send the JSON payload to a processing function to perform actions such as going to "Previous" or "Next" revisions of the dashboard.
• Once a dashboard is generated, we perform a self-healing loop. This uses Python's exec method to run the code and capture any errors it throws. The errors are captured and passed to Gemini (1) to fix the code. This is done in a loop until no errors are thrown.
• Once the code is clean, we use a subprocess to run a Shell script. The script navigates to the directory containing the saved code from the previous step. It then executes a command to run the dashboard using steamlit run.... We keep a pointer to the process so we can terminate it when needed.
• We keep track of port numbers for Streamlit. We expose a range of possible ports.
• When a dashboard is no longer in use (iteration or switch to previous rev), we kill the process and put the port number back onto a queue to be used later. The ports are tracked globally so multiple clients can connect and iterate through dashboards without fear of collision.
• The UI is built with React/NextJS and Tailwind-CSS for style. It connects to the WebSocket for communication and renders an iFrame showing the content of the port that has the dashboard

Challenges we ran into

• There was no tool for easily testing websockets with files. We had to improvise a quick solution while the UI was being built.
• Using exec with 3rd party libraries. It took some time to set up a good code execution environment inside our server so that we could iteratively test the generated code.
• Model inconsistency. The code was not always consistent, and sometimes the regular text responses. Through a combination of prompting and validation, we were able to overcome this to an extent.
• Cross-platform development. Certain system-level apis and shell scripts don't work on Windows or need to be written differently for Windows/Unix. Docker came in handy.
• We had a lot of difficulty terminating the Streamlit subprocess.

Accomplishments that we're proud of

• Revision system. We can dynamically keep track of multiple clients having multiple revisions simultaneously.
• Ease of use. This is a key feature of our app, as it allows no-code users to generate the same quality dashboards are users with coding experience.
• Self-healing code. We are very proud of how we are able to dynamically generate and sanitize code all within our server, with no 3rd party deps except for Gemini

What we learned

• Gemini is good but is not mature enough for a production setting.
• We learned a lot of low-level details of how the Python runtime interacts with the OS apis.
• Learned websockets

What's next for Aurora GeminiLit

We joined the hackathon late and didn't have time to implement some of the features we initially planned for. These include:

• Allowing users to connect their database. Provided a connection string and table name(s), we can query the database from our server and use that data to generate the dashboard.
• Persist dashboards between browser sessions. By implementing an account system we can store dashboards in a database and bucket, allowing the user to see all revisions of all dashboards they have created.
• A more robust way to launch dashboards. Using something like Terraform to launch cloud instances of a dashboard server might take a bit more time, but is much better than running a subprocess inside the same container. Key for scalability.

Built With

• docker
• fastapi
• gemini
• google-cloud
• nextjs
• python
• react
• tailwind-css
• typescript
• v0
• websockets