Draw n' Drop

Inspiration

When we decided to build Drag n' Drop, we set our purpose to be to allow people to express their artistic creativity in a more immersive way than just a paper or an image on a computer screen. Our two main inspirations were Geocaching, a service that lets people leave notes around the world for others to see, and Google's Tilt Brush, a VR game that let the player paint in 3D space. We decided to combine these two ideas to turn drawing into an immersive and social experience by allowing users to leave their drawings behind for others to see, engage with, and be inspired by.

What it does

In short, Draw n' Drop is an iOS Native app that gives users opportunities to find special locations where users can make their own 3D drawings and review those left by others. These locations are indicated using QR codes and coordinates. To access a scene, a user can scan the QR code, which fetches scene information from the server, updates the location the QR code was last scanned at, and loads the scene onto the user's phone. Then, the user can use their phone to draw lines one foot in front of them, as well as choose the color and width of their strokes. Once they are done, they can save their drawing as an art piece for the next person to see. The app also has a map feature that highlights the last seen coordinates of QR codes within a set distance of the user, and each location carries a hint with it that clues the user into where the QR code's exact location is. Users can also review each other's art pieces.

How we built it

Draw n' Drop is an iOS Native app, meaning the app's frontend was written entirely in Swift by Lexline and Matas. We made use of RealityKit, ARKit, and Apple Vision to render the drawings in 3D space around the user and have the drawings change orientation as the user's position changes relative to the QR code, which we set as the origin. Andres worked on the backend, which consists of a RESTful API created using FastAPI and Python and connected to a MongoDB database with Beanie. Our project uses a Docker image to make sure the backend performs the same for every system, and to make deployment easy. We deployed the backend on Render. Throughout the project, we consulted with Claude and Gemini to ensure we were moving in the right direction and that our code had as little errors as possible.

Challenges we ran into

We mainly faced two challenges in development. The main challenge we faced was learning all the technologies involved in the project for the first time while also writing the project code on the hackathon's limited timeframe. The other main challenge was getting RealityKit and ARKit to render the drawings in the correct locations and move them around when the user moves around.

Accomplishments that we're proud of

By far the best thing we did (in our opinion) in our project was getting RealityKit and ARKit to work correctly. Doing this was a massive undertaking that took us 20 of the 36 hours to do, and we were very proud of being able to get it to work. A notable second achievement was getting the backend to interface correctly with the frontend, which is always one of the most exciting parts of working on a full-stack project.

What we learned

Lexline and Matas learned a lot about location data and the things you can do with it. They also learned how to leverage RealityKit and ARKit for alternative reality projects in iOS such as this one. Andres learned how to use FastAPI and Beanie to create a RESTful API in Python that interfaces with the MongoDB database, and also how to apply knowledge of Docker to deployment on Render.

What's next for Draw n' Drop

The database and API structure we have right now is somewhat inefficient due to meeting the hackathon deadline being the first priority. With more time, we could change the structure so that large transfers of data happen as little as possible, making the app faster and easier to lose. We could also build out full fleshed accounts that allow users to save their data across different devices, instead of generating one ID per phone. Overall, these are just a couple of examples of what we could do with this, and we definitely believe it brings a new dimension to drawing.