Inspiration

We were inspired by Jarvis in Iron Man and thought if we could recreate it for your laptop.

What it does

You can speak/type things you want to do on your computer and Flow will do it for you. It can create workflows for you for a list of actions and can run them for you. The workflows are semantically stored so that Flow can understand and execute them.

How we built it

We used SwiftUI on macOS to create the frontend and FastAPI/Python for the backend. Everything is run and stored locally apart from the APIs that we call.

Challenges we ran into

Connecting the shell to the natural language model and making it useful was tough. Also, we had to learn SwiftUI from scratch for this application which is a steep learning curve, especially with Xcode not being nice to us.

Accomplishments we're proud of

We got detailed workflows with a set of instructions to run smoothly. When commands in the workflow did not work, the assistant tried debugging first before giving up.

What we learned

We learned SwiftUI from the ground up to make this, learned about many different APIs that helped us with text-to-speech, speech-to-text, and more.

What's next for NaturalOS

We want to extend the capabilities so it can control your mouse and keyboard for you, and do more complex tasks that require more thought and planning. We want the assistant to see what the user sees and perform all different kinds of tasks for you just from a few words spoken/typed. We believe that natural language will be the new way to interact with your operating system.