Rosetta

Inspiration

I've been an air mission commander for complex, high intensity operations and experienced the chaos of trying to understand simultaneous, ambiguous, information from multiple sources. In that environment, critical information is missed and many experience task saturation leading to inaccurate decisions or no decisions at all. There's needs to be a tool to help decision makers build a complete and accurate operation picture and free them up to make the right call at the critical moment.

What it does

Rosetta captures the realtime communications from multiple information streams, and synthisizes them into digestible, actionable pieces. It can do this at the channel level, the unit level, and the mission level, in any language enabling seamless coalition operations.

How we built it

We broke the problem down into a few components:

• How do we store radio / audio recordings: We accepted audio recording uploads and real time computer recording via a python script to simulate the stream of audio clips that would come in from a variety of radios. This was done via a FastAPI backend and the OpenAI API.

-How do we surface the various radio streams to users: We enabled users to view transmissions by channel number and to summarize all of the transmissions in a particular channel, along with summaries across all channels. Users can also ask specific questions about transmissions across all channels. Finally, for incoming audio recordings, if the audio contains the use of specific trigger phrases, the UI highlights the relevant channel to indicate to the user that they should investigate the comms in that channel. This UI / UX was built using Next.JS.

Challenges we ran into

It was tough scoping down our problem to something we thought we could achieve in 24 hours. For example, we had to forego more complicated alerting. We also were not able to integrate other forms of communications even thought it is clear how we could have accomplished that with a longer timeline.

Accomplishments that we're proud of

We are proud that we were able to build a working end-to-end application that, with just a bit of additional work, could actually make a difference on the battle-space today.

What we learned

We learned about some prompt engineering nuances to ensure that we got summaries that were informative but that also did not infer anything beyond what was explicitly stated. Furthermore, I think we all developed a better intuition of what can and can't be accomplished in 24 hours when trying to solve problems that integrate multiple data types and modalities.

What's next for Rosetta

We want to make the UI / UX more user friendly so that everyone can get the information they need in 3 clicks or less. We want to improve the underlying LLM architecture by using smaller, locally hosted models so that this could be run in remote environments. We want to make the anomaly detection / alerting system more complex so that we can detect significant events. Finally, we want to integrate other data modalities.

Built With

• fastapi
• nextjs
• openai