Tilly - Your Business Orchestrator

Segment your data any way you like, create powerful promotions in your system to ensure the best experiences for both customers and staff.
Check and reorder stock in seconds or go even further and just set it to auto levels all balanced to seasonal and other considerations.
Get to really know your team, guide, train, support and enhance every and any you may so wish, the power of unification!
Tie in real time driver tracking, delays or rerouting against production stages and capacity bottlenecks for optimal performance.
Go wild with reports! Simple ones or crazy custom SQL customised requests and on the fly dashboard generation, there really are no limits.
Architecture

Inspiration 💡 90% of people use only 10% of deep or complex platforms and their tools. 🤦‍♂️ Those who wear many hats often identify features that would help them run their businesses better, but they struggle to find the time to truly make those tools work for them. We wanted to change that.

What it does 🛠️ Tilly breathes light into the darkest of corners, making it easy, fast, and simple to find even the most hidden tools—and more importantly, use or connect them to benefit from their capabilities properly. All in minutes, with little to no prior knowledge required from the end-user.

Under the hood: Tilly is a pure-voice enterprise orchestrator. Users interact entirely hands-free via a real-time audio stream. Tilly translates that natural conversation into complex backend actions—like live stock routing, loyalty point adjustments, and multimodal email generation—executing them silently in the background while keeping the conversation flowing.

How we built it 🏗️ We’ve spent the last decade building a comprehensive (and very cool 🎺) business platform. However, we realized that powerful tools are often underutilized due to UI complexity. This project allowed us to rethink the user experience entirely, positioning voice as the stepping stone into the future of business management.

To achieve this for the hackathon, we implemented a robust dual-layer AI architecture:

The Voice Layer: We used the Gemini 2.5 Flash native audio model over WebSockets to handle low-latency, real-time voice conversations. This acts as Tilly's "mouth and ears."

The Action Layer: Simultaneously, we utilized a secondary background text model that strictly handles complex JSON Function Calling (Tool Use). This ensures backend actions execute perfectly without interrupting the live audio stream.

Challenges we ran into 🧗 Reliable tool calling, context window limits, API drift, and multi-model delivery timing.

The biggest technical hurdle was the classic state management dilemma: forcing a real-time audio stream to pause, think, and accurately output structured JSON for tool calls often caused latency and dropped audio frames. We solved this by separating the conversational stream from the tool-execution logic, allowing Tilly to use conversational "fillers" while the background layer securely executed the APIs.

Accomplishments that we're proud of 🏆 The core logic of this project harps back to our roots: the belief that even the smallest businesses should have the most powerful tools at their disposal at a fraction of the cost. We achieve this every day for our existing customers, and Tilly takes it a step further.

Technically, we are incredibly proud of the UI/UX and State Management. By implementing an event-driven state machine, an "Action Timeline," and optimistic UI placeholders, the user interface updates seamlessly alongside the voice stream. It proves that a "keyboard-less" app can still provide deep visual context without the clutter.

What we learned 🧠 Plan, plan, plan... and then plan again. Update constantly, and try to keep up with how fast these models and capabilities are maturing—it really is a wonderful time to be alive! We learned that while bleeding-edge tech can be buggy at first and require clever workarounds (like our dual-layer approach), those constraints often reveal unexpected, robust architectural patterns. Never be afraid to try!

What's next for Tilly 🚀 Tilly is actually a slice of a much deeper stack—an orchestrator-based system with field-relative experts and specific action sub-agents. We have features like full email designers built out, and Tilly uses them to actually build things from scratch, component by component. This approach works for brand websites, mobile apps, self-service kiosks, kitchen displays (KDS), warehouse systems (WDS), and lots more.

For this hackathon, we wanted to showcase how you can take this conversational orchestrator approach and use it to take meaningful, complex actions within your business. How deep that rabbit hole goes is up to you!