Inspiration

So much time is spent waiting and watching for things to happen, such as waiting for a sports game to start, waiting for water to boil, and watching to make sure that someone doesn’t do something dangerous.

Instead of spending your valuable time waiting and watching, why not have AI models do it for you? Meerkat does exactly this, letting AI models watch and wait with the same patience and alertness 24/7, so you can go do something else. When something comes up, Meerkat notifies you immediately and you can wake up from your nap and respond.

Meerkat can look at your live webcam, microphone, or watch any other website and tab that you assign it to watch, with you specifying a prompt in natural language. It can also track state for you and let you when things change.

How I built it

Meerkat uses GPT-5.6 as a planner to build a funnel of cheaper, realtime gates, which are run in realtime on the live video and audio streams. Then, when these gates fire, the information is sent to GPT-5.6 through the funnel, which serves as a final gate to determine whether the user is notified.

This allows for a more efficient solution compared to taking continuous screenshots and sending to GPT, since a lot of the time there is likely nothing relevant to the prompt and therefore can be safely ignored by the user. Instead, the cheaper gates can continuously run at nearly zero cost, and GPT can be run when there is important relevant information.

The entire project was built with Codex, mostly in Python for the backend.

What I learned

First, I learned that modern LLMs can remove the need to build a custom pipeline for each different task by hand, instead letting GPT-5.6 do the planning necessary for this task. This could involve using OCR, object detection, or keyword matching from realtime transcription, for example. Previously, solving a task like waiting for a dog to do a certain action would have required building a custom object detection and motion tracking pipeline. In this case, GPT-5.6 can figure out what to build.

Second, I learned that for LLM inference time, there is at least 300-500 ms from round trip network cost from browser to data center and 300-500 ms from time to first token (TTFT), which is very hard to optimize away. Furthermore, scaling input tokens does not have a great effect on the inference time, while scaling output tokens has a much greater impact, so this became one of the focuses of my inference optimization.

Challenges

The biggest challenge was to reduce the latency of the system to respond quickly when an event occurred. Ultimately, my approach was twofold: 1) reduce the number of calls to GPT after a cheap gate fires to a maximum of 1, and 2) reduce the total number of tokens that GPT has to generate. For 1), this meant building a flow that consisted of running fast realtime gates, then as soon as a gate fires, immediately send it to GPT, meaning there was only one round trip. For 2), this meant building a response tree beforehand and only asking GPT to verify YES/NO in many cases, and only writing a small verification piece of evidence in others. For example, if the task was to alert the user when a dog picked up a beachball, then the GPT would be asked only to verify whether this had occurred (YES/NO), with YES resulting in a pre-generated response being played.

The second challenge was to find a sufficiently diverse and varied set of tools and gates for GPT to compose. Ultimately I created this through a mixture of telling Codex to create certain basic gates, asking it to brainstorm gates, and seeing what gates were necessary for a series of sample tasks. Ultimately, this resulted in a broad mix of gates and tools.

Built With

• codex
• gpt
• ocr
• python
• stt
• tts
• vqa
• yolo