Inspiration

As people living with food allergies, we know how sometimes, telling the waiter isn't enough to keep you completely safe at a restaurant. Eating out involves a question that a menu cannot answer: what ingredients are in the food, and what touched the tools, hands, and surfaces that touched it.

We built Spork to address this invisible problem. A kitchen is a dynamic environment where a single fork can contaminate fifteen plates; and connect allergen sources to otherwise safe dishes. Our system, by managing these many inputs, ensures that dishes are made correctly and safely, one at a time.

What it does

Spork takes in a customer input, processes the language to determine allergies, and monitors an order in behind the kitchen doors. Spork maintains an entity-based world model: each object, from forks to plates, has a persistent identity, state, and history living in the world, dictating its location, allergens, and more. Allergens propagate along time-ordered interactions detected by our neural network, connecting the original source to the plate, preventing a contaminated dish from reaching the customer.

Spork can see if a contaminated spoon that previously touched peanut butter was used to stir soup, and will point out each dish that spoon touched. Spork can see if a fork was washed before it was moved from the peanut pile into the soybean pile, increasing confidence of allergen safety. Spork can distinguish between physical contamination and visual overlap sans contact. Spork can output whether you should give the dish to the customer, or remake it in the kitchen.

To add more detail, in the unfortunate event of contamination, Spork also shows the exact chain of events that led to the outcome, from the start of when the order was made. It can also point out repeated mistakes the chef made by analyzing past behaviors.

How we built it

Spork uses a holistic world monitoring system to ensure safety. With two fine-tuned language model decision engines for customer allergen reports and ending ultimatums, a custom-trained YOLO26s vision component with 800+ VLAM-annotated training images, and a ConvLSTM for detecting whether a contaminated instrument actually touched or moved over another dish.

We have a custom decision model that takes in customer input, reading allergen warnings and sending it to the kitchen monitor. During food prep, our custom-trained YOLO detector and pose model run on-device at over 20 frames a second, tracking the bins, tools, hands, and plates. Combined with our custom object permanence algorithm, we give identities to each object and establish a persistent store of these values. We keep track of allergen and contamination data, all while using our ConvLSTM to detect interactions between objects (stored as pairs of entities). We can then log these interactions over time to have a full, complete model of how our kitchen and instruments changed over time, and whether the final dish is safe to give. All of the information is stored in SpacetimeDB's relational databases, and indulges inference rates of around 500 read/writes a second.

To be more specific, perception updates the world and downstream systems reason over it. The world and its behaviors are stored as a holistic world provenance graph that represents everything that happens in the kitchen; every touch becomes a time-stamped edge in this structure. The graph tracks the actual chain of interactions, so a clean preparation can pass instead of treating every nearby object as contamination.

In the end, our second decision model gives an ultimatum on whether to serve the dish to the customer ot not. For additional help, we also developed a system to detect repeated mistakes. We traverse this provenance graph and encode entity-interaction scenarios, from peanut-butter-spoon contamination to soup stirring, as vectors. We upload these vectors to a custom vector search database layer we hand-wrote and applied over SpacetimeDB. By searching for similar vectors whenever a new scenario occurs, we can track repeated mistakes and common situations to help the chef find out what they did wrong and what they should keep track of in the future. If the pineapple knife was used to cut fish more than once, especially if a customer is allergic to pineapples, Spork will point it out.

Challenges we ran into

We had to retrain each of our models many times; for example, we had to re-train our YOLO26s object-detection CNN multiple times to ensure it could properly detect our white plastic utensils in white paper bowls. We ended up gathering our own training data for multiple hours. Furthermore, ensuring the process was end-to-end was a bit difficult, as we had to slot together all of our various networks into a working UI and a tabling system. Finalizing a design for the provenance graph was a long but rewarding process, as we eventually settled on having it resemble time-scale entity version control.

Accomplishments that we're proud of

We are proud of the complexity of our model and how we can output a full world understanding through our provenance graph. It comes to show how well our models fit together in a general sense. When it comes to more specific features, we are really proud of how we trained our ConvLSTM; in order to ensure it only marked interactions that actually contaminated dishes as significant, we trained it on bounding box collisions AS WELL AS an image-gradient mask that dictated how much change occurred in certain pixels since the last time step. This not only slots well into the LSTM's intended nature of learning temporal data but ensures that we can properly differentiate between a spoon going over a plate and it actually mixing with its contents!

What we learned

We learned a lot about synthetic dataset generation, especially pertaining to our LSTM and CNN. To be more specific on the latter, we dove deep into VLMs, concerning our YOLO model's training, and managed to create a model with 95% accuracy when used!

What's next for Spork

We have the capacity to run our models on a Raspberry Pi 5, yet it remains relatively untested. We would love to take this further and ensure more hardware independence in the kitchen!