Snugly

Home screen
Scanning items
Find your flight
Trip details
Item suggestions
Suitcase modeling
Final checklist

Inspiration

As college students, we find ourselves at the airport every couple of weeks, whether it be visiting home during break, traveling to a conference, or flying across the country to the largest collegiate hackathon in the world 🤩. However, packing for each trip is extremely stressful: not only do travellers have to create personalized packing lists, we also have to abide by TSA liquid limits, airline weight limits, lithium-ion battery rules, and customs restrictions that vary by destination. Travellers will usually eyeball a pile of clothes, stuff them into the suitcase, and sit on it to make everything fit. Even worse, we find out what is and what isn’t allowed on the flight at the worst possible moment: when TSA confiscates the items in the security line. In fact, on the way to Cal Hacks 13.0, our team experienced first-hand how TSA confiscated a very expensive cleanser because it exceeded their liquid limit. All these rules induce unnecessary travel stress and lead to tons of wasted time and money.

What it does

Our project is an iOS app which helps you with all your packing needs. Ever find yourself scrambling to pack everything into an already overflowing suitcase, or wondering what kind of clothes you need to pack, or even whether some specific lotion you have can actually go through TSA? Download Snugly! All you need to do is snap a large picture of the stuff you want to pack, with a credit card/id card for dimension confirmation in the picture. Our optimization algorithm will show you exactly how to pack everything neatly into the suitcase, even accounting for weight, fragileness, softness/flexibility, and more. By simply inputting your flight airline and number, as well as your travel dates, our app will take into account what you’ve already brought, and give you recommendations on what to bring based on the weather, common items you might be forgetting, and items you should avoid bringing through TSA and customs of countries around the world!

How we built it

Our app is built as a native iOS app, using SwiftUI, with a 3D item scanner and bag viewer for packing visualization, destination planning and forecasting, and TSA checking.

For the scanning portion, we allow the user to take an image of a collection of their items, leveraging Apple’s Vision framework to outline all of the objects. By fitting a known item (credit card) dimensions, we can use pixel measurements and camera angles to gauge the floor’s position and scale. From there, we fit a box or cylinder to outline each object and get the dimensions of the detected objects. Finally, Claude takes each captured object and names them and guesses its weight, fragility, etc.

For the packing portion, we use a custom deterministic package to order and arrange the items. First, it tries 18 ordering strategies (e.g. heavy first, fragile late, items you need at security last, and more). If an item is unable to fit, it lets the user know. It ultimately renders a 3D model of the suitcase container with all of the user items, with a progression for how the user can more effectively layer the items to reach the best capacity.

For the backend, we use Node, TypeScript, and Hono. We have endpoints that interact with Claude, handling object detection and identification, packing suggestions, and rule matching. We use Apify actors to scrape for flight lookups, TSA, airline baggage, and customs pages. Finally, we use Open-Meteo for weather forecasting at the destination location.

Challenges we ran into

The main challenge we ran into was missing LiDAR imaging. Our original plan was to extend our packing optimization to backpacks as well as suitcases. However, this was based on being able to use LiDAR for extremely accurate depth measurements into backpack pockets and curves in luggage. However, none of our phones were a model able to use the built in LiDAR in Apple iPhone cameras. Thus, we had to quickly pivot and figure out a streamlined way to keep our optimization and packing algorithm.

Accomplishments that we're proud of

Completing a product that can accurately identify an item, its bounding boxes, and its dimensions without LiDAR was something we are particularly proud of. Using the Apple Vision Framework and Claude, our final product is equally as capable as one that uses LiDAR. Using the identified items, Snugly can determine how to pack a bag so more fragile items can be packed in safer positions of the suitcase. Furthermore, our use of Aplify enables Snugly to easily find flights and determine the user’s travel destination, allowing for Claude to recommend items that the user has not packed yet.

What we learned

We learned to use the right tool for the right job, or in our case, the right model, for what it's good at. Initially, for the objects in frame, we were delegating all tasks to Claude, however, this increased response times and resulted in inaccurate bounding boxes framed. We instead pivoted to solely using Claude for things it was good at, naming, categorizing, and estimating weight, while delegating the outlines to Apple’s Vision framework resulting in much more reliable bounding boxes and ultimate dimension estimates.

What's next for Snugly

We would like Snugly to work for all types of travel equipment including backpacks, duffel bags, and other sorts of travel bags that aren’t as rigid and deterministic as suitcases.