Celebrity GPT Ordering

Gamifying the ordering experience by bringing celebrity order-taking to life through ChatGPT.

Comment

Inspiration

Thanks to ChatGPT 2023 has seen an explosion in innovation. Tech can be built faster than we ever dreamed. Deepfake and other technologies have blown our minds and made us rethink what's possible.

We have also seen lots of recent advancements in restaurant technology. Self-ordering from kiosks, apps, and web apps took off after COVID.

Now we are seeing the industry try to put these two together, and as AI approaches human proficiency it is still not as good! So, it might seem like a good idea to the restaurants because it can save them money but the consumers will not adopt it. Just like hitting "0" to an automated phone teller, people will look for a way out.

But, what if we could add something to the AI that actually benefits the consumers too? That is what led us to the idea of celebrity ordering.

Gamify the ordering experience to make it fun. It might even end up being something people prefer!

What it does

Celebrity GPT Ordering is a chatbot that takes a customer's order in the character of some public figure.

It starts by asking the customer what character they would like it to impersonate. The restaurant's Square catalog is loaded and the bot then asks them questions until they are ready to check out. Because it is AI, it can switch to ANY character on-the-fly, provided it is publicly known enough for the GPT model to have information on it.

They can then check out in normal ways using Square's Web Payments API. Because it uses Catalog API, Orders API, and Payments API, all the information is perfectly synced with the vendor's Square account and things like taxes, printing, reporting and back-of-the-house operations are all seamless.

How we built it

We started by feeding catalogs in JSON format into regular ChatGPT to see if it could understand them. We were convinced that it was hilarious--and therefore worthwhile before we were convinced it was possible. More on this in the next section...

Some of the parts that normally would be the hardest were really easy because of the awesome power of ChatGPT. In order for it to play along with the whole impersonation premise we simply told it:

"You are a restaurant employee taking orders from a customer over the phone."

and...

"You should act like Snoop Dogg and embellish what you say."

We then told it to give us its responses in JSON format with this prompt:

Every response should only be JSON with the followiong format

{
    "message": the next message that should be asked to the user.,
    "finished": a boolean to indicate that the user is done ordering,
    "items": an array of JSON objects with this format:
        {
            "itemID": item ordered,
            "id": a unique integer for this item in this order,
            "quantity": number of items ordered,
            "modificationIDs": an array  of modificationIDs
        }
}

We also told it: Assume that the customer can see the menu so you don't need to list anything unless the customer asks for it. Make sure the customer has specified required modifications before putting anything in the items array. If the customer asks for anything not on the menu suggest something similar if available but never make up a new item.

and...

The following is the menu in JSON. In the JSON structure each item will have a list of modification groups that has a minimum and maximum of modifications that can be added. The modifications are ids of the modifications listed in the "modifications"

We then worked with a local company called Flash Order who is a Square Partner and makes really sleek web apps for restaurants. We were able to use the Flash Order API to add our own custom javascript module to their web app. This way we were able to leverage a ton of integration groundwork and result in a really seamless order flow.

Since we wouldn't bet our lives on GPT-generated cart, Flash Order and Square both provide layers of safety. This ensures that everything added to the cart follows the correctd pricing, rules, discounts, and tax calculations.

Challenges we ran into

  1. 3.5 vs 4.0 Initially we only had access to ChatGPT 3.5. Not only did it prove to not be "smart" enough but after a few commands it would ignore some of the rules we gave it! It would return data in invalid JSON formats, which obviously was problematic.

  2. API Access We also did not have access to the 4.0 API. Even with a "plus" account we could only work with it interactively. Believe it or not we actually implemented a system that automated an intermediate web browser window that would enter text into the ChatGPT website and then collect its responses. This was an ugly hack but it worked! Fortunately though we eventually gained 4.0 beta API access and were plenty happy to abandon this workaround :)

  3. Hallucinations One thing ChatGPT is famous for is making things up. These are called hallucinations. For example, we would ask it a question and it would sometimes make up new menu items. In one example the restaurant had porridges and we would ask what kind of porridges we could get. It came back with four options that sounded really good--and had really good descriptions but they weren’t things that the restaurant even offered. So we had to figure out how to adjust the prompts to tell it not to do this.

  4. Data formats Having ChatGPT give back a response that we could read every time and know exactly what it was supposed to happen was another challenge. To get ChatGPT to give us a response that the computer could read we had to specify exactly how we wanted that response. The final format that worked (listed in the previous section) took quite a bit of work to determine.

Also, the catalog format had to be simplified quite a bit for ChatGPT to be able to use it. Fortunately, Flash Order's API already does a lot of heavy lifting by importing and processing the catalogs from Square. It provides a simplified menu that already accounts for available times, locations, inventory, and other adjustments. However, we found that the catalog still needed even more simplification for ChatGPT. Our final format became: 

fileprivate struct ModifiedCatalog {
    struct Menu: Codable {
        let categories: [Category]
        let modifications: [Modification]
    }

    struct Category: Codable {
        let title: String
        let items: [Item]
    }

    struct Item: Codable {
        let id: ItemID
        let title: String
        let subtitle: String?
        let price: String
        let modificationGroups: [ModificationGroup]?
    }

    struct ModificationGroup: Codable {
        let id: ModGroupID
        let min: Int
        let instructions: String
        let title: String
        let modificationIDs: [ModID]
        let max: Int
    }

    struct Modification: Codable {
        let id: ModID
        let title: String
        let price: String
    }
}
  1. Request sizes Part of the data format limitation was simply that ChatGPT could not process a menu that was too big. We probably could have delved much deeper in training our own private model, but for now we had to send all of the context for each request each time. This made our request sizes a limiting factor. Although we never hit a limit, we are sure there are catalogs out there that would be too large for this approach.

Accomplishments that we're proud of

  1. It is real The entire system actually works and is LIVE!!! As shown in our video we actually tested it on the very last day of the hackathon deadline at a real restaurant. They had no idea that we were even doing it, and it went through their system perfectly.

  2. "Content" on-the-fly We can't take as much credit for this, but we are just amazed at the power of these transformer models. What we did in a fairly short amount of time would have taken vastly more time just a short time ago. The impersonation is not only AMAZING, but it allows our system to operate without any specific data that could be copyrighted or trademarked.

  3. It has potential We started down the road of generative voice, imagery, and video to go along with the system. While we struggled to find the right combination we are confident that those tools either exist already or are just around the corner. With all of them combined you could make the application of this really exciting.

  4. It solves a real problem As we said before, we feel like ordering is not going to get better by trading humans for robots. Adoption of mobile ordering is still really low--even Starbucks, the king--only has 25%. However, there seems to be real pressure to do this and we feel like our project provides motivation for the customers to get on board.

  5. You can try it, and it is FUN! Go ahead and try it with our sample restaurant, or use the easter egg "?gpt=true" at a Flash Order-based restaurant near you! A couple caveats though:

  6. be patient, it isn't currently super fast.

  7. we might hit our rate limit depending on how many people are trying it so be patient and let us know if something goes wrong

What we learned

Waiting for beta API access is excruciating!

Generative Pretrained Transformers (GPT) are amazing

Ordering can be fun! As we showed it to family members they all wanted to go use it.

What's next for Celebrity GPT Ordering

  1. Market testing As we inform people about the opportunity to try it we will be able to see how well it performs, gather feedback, and validate some of our assumptions about consumer behavior. Fortunately, we can start this in hundreds of locations today.

  2. Public rollout As we engage with the restaurants where this is possible, we can encourage them to advertise the use of the AI. We also believe that AI is enough of a buzz word right now that we could do several press releases, news articles, etc to get attention and support.

  3. Video impersonations Going beyond text-based chats to enjoy the unique opportunity of having real conversations with the AI representations of selected characters. We’re also exploring the possibilities of generated icons and text-to-voice, voice-to-text interactions, further boosting customer enjoyment and satisfaction. Imagine a virtual Facetime. Now think about how good Deepfakes is getting. The future is exciting!

  4. Better API access including private model training and higher rate limits.

Built With

Share this project:

Updates