Inspiration

I have worked on accessibility projects before, and it kind of just popped into my head as I was brainstorming how I could use GPT4 in a novel way.

What it does

App developers could provide a mini-grammar interface for their app, that AccessAbility will use to coerce GPT4 into a sort of natural-language to control adaptor.

My demo app is a basic drawing canvas, you can -- with your voice -- draw shapes, navigate around the canvas, etc.

How I built it

There are 3 major components:

  1. Speech recognition
  2. Speech -> grammar translation
  3. Command ingestion
  4. Graphical interface

The first three are the actual service AccessAbility provides.

For speech recognition, I am simply using Apple's Speech framework to easily get transcribed speech.

I then use a community made OpenAI API package for Swift to precondition GPT4 into being used in this manner.

I made a tokenizer and made an MVVM variant of the command pattern to parse, validate, and execute the commands on the canvas model.

The graphical interface is a simple canvas structure I made for another project a long time ago, I adapted it to support some shapes, and facilitate animations.

Challenges I ran into

Luckily for me working on this project was pretty smooth sailing.

The biggest challenge was timing when a transcribed query was done. This wasn't really that hard, I just have a roughly 1-2 second timer that counts down since the transcription was last changed. When the timer hits 0, the query is passed to GPT4 and along the pipeline it goes...

Accomplishments that I'm proud of

I have never implemented the command pattern in a Swift/SwiftUI environment before, so it was cool to do it in MVVM instead of MVC.

I made sure to write the code to be as expressive, modular, and robust as I could so that I could easily make changes, resolve issues, and reuse these structures in the future.

What I learned

GPT4 has much more spacial awareness than GPT3.5.

What's next for AccessAbility

Nothing. I will not pursue this project further.

Built With

Share this project:

Updates

Submission history