SnapShot

Controller
Canvas

Inspiration

As university students, one issue that we have with online learning is the need to seamlessly move physical notes and images to a laptop in a way that allows us to collage. Currently, one would have to take a picture, upload it to your laptop, edit the image and then add it to the document. We wanted to demonstrate an alternative mode of interaction and organisation of work/life.

What it does

Our product allows users to directly scan items, and place them onto anywhere on our website. First the user takes a picture of something they want to use, the image is then automatically formatted to remove the background and then instantly transferred to the online canvas. The image can then be moved around on the canvas by using the phone as a pointer. This has many use cases when it comes to hybrid learning and workplaces, education, notetaking, mood boards or just creating a fun collage of all your favourite things.

How we built it

Our frontend is built in in ReactJS and Tailwind CSS. With most phones having an onboard IMU, we utilised this via the Web Sensor API. Our backend was developed in NodeJS and hosted on Heroku, with SocketIO used for real-time communication. We used a background removal API and experimentation with OpenCV.

Challenges we ran into

For image segmentation, we had the option of either using a pre-made online API or making our own model with OpenCV. One issue we had with OpenCV was that it did not work well with shadows and another issue was that JavaScript's implementation of OpenCV is extremely unintuitive and was a struggle to get the hang of. In the end, the remove.bg API was simply better than our OpenCV implementation so we ended up deciding to use the API. Although we hope to work and implement the OpenCV model in the near future. We also ran into many synchronisation issues, when trying to keep multiple clients and controllers in sync - a key aspect of remote collaboration.

Accomplishments that we're proud of

We are proud of how seamless our project is, the background removal and image transfer happens almost instantly making it very easy for users to collage. We managed to complete the project which does everything we intended it to do, and it already surpasses our own expectations with multiple members of our team pulling off an all-nighter to ensure it happens. Utilising multiple technologies, we are proud of our achievements in creating an end-to-end proof of concept for an alternative mode of digital interaction.

What we learned

Through research with OpenCV, we learned many image processing techniques that we are unfamiliar with and we also learned a lot about the library itself. Many of our group were also completely new to React and JS in general so we all learned a lot more about making web-apps which will definitely serve us in the future. This was especially true for working SocketIO and ironing out synchronisation issues, as well as using the relatively new Sensor API.

What's next for SnapShot

Integrating with other software (e.g. Word, Powerpoint, Webex etc.) with the ability to directly drop images into your notes, presentations etc. to organise you life.
- Additional functionality such as editing, resizing and video communication.
- Using OpenCV to detect more specific to target subgroups, such as book notes and text.