AutoScroll Front End App Design

Inspiration

We realized that even for a very big market like short-form media, the technology that is used for it is still not very advanced. They still require you to manually scroll using your thumb and have not always the most accurate recommendation algorithms. This gap in innovation inspired us to build AutoScroll.

What it does

AutoScroll is an iPhone app with an AI-powered Safari extension that determines how satisfied the user is with the current short-form video that they are watching and based on this, either automatically scrolls or watches the video.

How we built it

We used Xcode IDE as well as SwiftUI on the frontend and Javascript on the backend. The design choice to use JavaScript was because Safari extension's runtime only works with JavaScript which means that we could run the AI compute on-device rather than having to pay for servers. In terms of AI models, we used a variety of them to fine tune the prediction. This includes ones that perform image classification to gauge emotion and ones that analyze audio and gauge user sentiment.

Challenges we ran into

One major challenge was finding the best AI model to use. Given our tech stack, we had to tediously download and set up each AI model before running the code to test. It was not as streamlined of a process as if we were to use Python and we could just change the reference link of the transformer library's pipeline function.

Accomplishments that we're proud of

We are proud of the fact that we were able to get a high-fidelity prototype done relatively early so that we could test the extension with different AI models in real time. We are also proud that we were able to do this while having not only one but multiple different sensors to make our extension more robust.

What we learned

We learned the importance of time management and prioritization. Engineers love to overestimate the time that it will take and the amount of capabilities that they can add within a given time frame. But, we learned through gaining experience at HackGT to give yourself and your team a reasonable amount of time for each task and that not every feature needs to be added to the minimum viable product.

What's next for AutoScroll

We would like to do more research on the various different sensors that can be added to the extension along with the ones that we already have. For example, adding a screen reader to the user's screen might provide a larger understanding of their interests and based on that, provide good data points to determine whether to automatically scroll or watch.