Link to Final Results, Conclusions, and Visualizations of the Benchmarking and Evaluation for Intuit Challenge

https://drive.google.com/file/d/16lG1VE8G2e9JPCOcMfxKjApzN-q78pZG/view?usp=sharing

Inspiration

Customers are the gods of every business. They are its foundation, its lifeblood, and the reason it exists.

For a small business, every customer relationship represents more than a transaction—it represents trust, recurring revenue, financial stability, and the opportunity to grow.

But while businesses work tirelessly to acquire new customers, they often don't realize they're losing existing ones until it's too late.

Customer relationships rarely deteriorate overnight. The warning signs often appear gradually:

A loyal customer begins purchasing less frequently.
A previously reliable payer starts delaying invoices.
Outstanding balances increase.
Once-consistent business activity becomes unpredictable.

Each change may seem insignificant on its own. Together, however, these signals could reveal something far more important: a customer relationship is changing, and the business may need to act.

Research reinforces why this matters. Harvard Business Review has highlighted that acquiring new customers can be significantly more expensive than retaining existing ones.

And the financial pressure on small businesses is real. According to QuickBooks' 2026 Small Business Late Payments Report:

59% of surveyed businesses had invoices overdue by more than 30 days.
Businesses with unpaid invoices were owed an average of $17,700.
QuickBooks' 2025 research also connected higher overdue invoice levels with greater cash-flow difficulties.

These findings reveal a critical challenge: small businesses cannot afford to overlook changes in their customer relationships or the financial signals surrounding them.

The Hidden Intelligence in Accounting Data

The most fascinating part is that the warning signs may already exist inside the tools businesses use every day.

Their accounting systems.

Every invoice, payment, outstanding balance, and purchasing pattern tells part of a customer's story. Accounting platforms like QuickBooks hold a wealth of financial information that can help reveal how business relationships evolve over time.

But much of this information remains trapped in transaction histories, reports, and dashboards.

Accounting systems are excellent at recording what happened financially, yet recognizing what those changes might mean for a customer relationship—and deciding when to intervene—often still depends on someone noticing the patterns.

For a small-business owner managing dozens or hundreds of customers while juggling operations, sales, finances, and daily responsibilities, continuously connecting these signals is nearly impossible.

The data is there. The signals are there. The relationship may be changing right in front of us. But nothing happens.

No timely warning. No deeper investigation. No opportunity to respond while the relationship can still be strengthened.

And by the time the consequences become obvious—declining revenue, unpaid invoices, or a customer who has quietly stopped returning—the opportunity for early intervention may already have passed.

The Question That Inspired SaveMyCustomers

That realization led us to a simple but powerful question:

What if accounting software didn't just record the financial history of a business, but could recognize meaningful changes in customer relationships and help owners act before those relationships are lost?

That question inspired SaveMyCustomers.

We envisioned a system that transforms passive accounting records into proactive customer relationship intelligence—continuously identifying meaningful financial changes, distinguishing routine fluctuations from situations that deserve attention, and helping business owners make informed, financially responsible decisions.

Because small businesses shouldn't have to discover that a valuable customer relationship was weakening only after the damage is done.

The best time to save a customer relationship is before you lose it.

Research References

Harvard Business Review — The Value of Keeping the Right Customers
Intuit QuickBooks — 2026 Small Business Late Payments Report
Intuit QuickBooks — 2025 research on overdue invoices and small-business cash flow

What it does

What if accounting software could continuously detect meaningful changes in customer relationships, identify emerging revenue risks, and generate three personalized, financially evaluated retention strategies—before valuable customers are lost?

SaveMyCustomers transforms QuickBooks-style accounting data into an event-driven customer relationship intelligence and retention system.

It combines financial state reconstruction, Jev-powered System One inference, selective LLM reasoning, and deterministic financial validation to move beyond traditional accounting reports toward proactive customer retention.

1. Event-Driven Financial Context Reconstruction

A Python-based financial context engine processes:

Customer records
Invoices and payments
Purchasing activity
Accounts receivable (AR)
Outstanding and overdue balances

Whenever a new business event occurs—such as a payment delay, declining purchases, customer complaint, or potential switching signal—the engine reconstructs the customer's financial state before and after the event.

Rather than treating transactions independently, the system captures changes in purchasing frequency, payment behavior, outstanding balances, and overdue exposure, creating a structured representation of how the customer relationship is evolving.

2. Jev-Powered System One Change Detection

The reconstructed financial context is passed to Jev, our first-stage System One inference engine.

Jev evaluates each event and produces three structured decisions:

Meaningful change detection: Has the customer's financial behavior changed significantly?
Relationship direction: Is the relationship improving, deteriorating, or remaining neutral?
Escalation decision: Does this change warrant deeper AI investigation?

This enables intelligent event filtering.

Routine fluctuations can be handled without invoking expensive generative AI, while potentially important relationship changes are escalated for deeper analysis.

3. Conditional LLM Risk Analysis

When Jev determines that escalation is necessary, the event is routed to GPT-4.1-mini for contextual risk analysis.

The model examines:

The customer's financial history
Recent business events
Detected behavioral changes
Emerging financial risk signals

It then generates a structured, owner-facing alert explaining:

What changed
Why it matters
Which financial signals require attention

This separates high-frequency change detection from more computationally intensive reasoning.

4. AI-Generated Retention Strategies

The system goes beyond identifying risks.

A downstream AI retention engine generates three personalized, context-aware retention strategies for the business owner.

These strategies are informed by the customer's financial behavior and identified risks, and may include:

Proactive customer outreach
Payment-plan adjustments
Targeted retention offers

Each recommendation provides a potential course of action, helping owners move from risk identification to practical customer retention decisions.

5. Deterministic Financial Feasibility Evaluation

Before recommendations reach the business owner, a deterministic financial evaluator checks them against available financial constraints, including:

Profit margins
Available budgets
Proposed offer costs

Each strategy receives one of three classifications:

Classification	Meaning
Feasible	Satisfies the available financial constraints
Rejected	Violates one or more financial constraints
Unverified	Cannot be reliably validated with the available data

This ensures that generative AI recommendations are subjected to explicit financial checks rather than being presented as automatically safe business decisions.

6. Persistent Intelligence and Human-in-the-Loop Dashboard

The system uses SQLite to persist:

Customer events
Reconstructed financial states
Risk analyses
Saved recommendations

A Streamlit dashboard provides a unified interface where business owners can:

Review customer histories
Inspect escalated relationship changes
Compare three recommended retention strategies
Prepare editable customer communication drafts

Importantly, the owner remains the final decision-maker.

The system supports informed action without automatically sending messages or executing retention offers.

Core Architecture

Financial Event
      |
      v
Financial State Reconstruction
      |
      v
Jev System One Screening
      |
      v
Conditional GPT-4.1-mini Analysis
      |
      v
Three Personalized Retention Strategies
      |
      v
Deterministic Financial Feasibility Evaluation
      |
      v
Business Owner Review

The result: More than an accounting dashboard, SaveMyCustomers is a proactive financial relationship intelligence system that helps businesses recognize changing customer behavior and make informed retention decisions.

How we built it

We approached SaveMyCustomers as both an engineering challenge and an experimental investigation into whether System One models could make continuous financial intelligence more practical than conventional AI approaches.

Our development involved four major stages.

1. Financial Data Exploration and Context Engineering

We began with the synthetic QuickBooks accounting datasets provided for the hackathon, using Python, pandas, and PyYAML to investigate customer, invoice, and payment relationships.

Our initial analysis focused on Company 003.

Dataset Metric	Count
Customers	136
Invoices	5,132
Payment applications	9,813
Reconstructed financial events	14,945

By joining invoice and payment histories, examining payment delays, and analyzing purchasing frequency, we identified behavioral changes preceding ended customer relationships.

We then reconstructed 14,945 chronological financial events, preserving each customer's financial state before and after an event.

This created the foundation for contextual financial intelligence rather than isolated transaction classification.

2. Building and Benchmarking System One Intelligence

To evaluate Jev rigorously, we constructed a 9,024-event supervised benchmark across:

100 customer contexts
42 financial and communication scenario families

These scenarios covered:

Supplier switching
Purchasing declines
Payment difficulties
Revenue fluctuations
Other meaningful business changes

We created customer-disjoint training, development, and test splits, separated model inputs from expected labels, and performed consistency and leakage checks.

We then compared four approaches:

Deterministic rule-based classification
TF-IDF and XGBoost machine learning
GPT-5.4-mini conventional generative AI
Jev System One inference

Our experiments evaluated:

Meaningful-change detection
Relationship direction
Escalation decisions
Inference latency
Estimated inference costs

3. Cross-Company Financial Intelligence Evaluation

We expanded our investigation beyond one business by exploring 100 synthetic companies spanning 71 industries.

To evaluate whether contextual financial intelligence could transfer across different business environments, we constructed an additional cross-company benchmark incorporating:

Revenue changes
Invoice frequency
Purchasing activity
Overdue payment signals

This allowed us to investigate the strengths and weaknesses of System One inference under more varied financial conditions rather than relying exclusively on a single company's transaction patterns.

4. Building the End-to-End Application

We implemented the application using:

Python — Financial data processing and application logic
pandas and PyYAML — Data exploration and configuration
Jev — System One financial-event screening
OpenAI API — Contextual risk analysis and retention strategy generation
SQLite — Persistent event and financial-state storage
Streamlit — Interactive business-owner dashboard

Our architecture separates financial computation from AI interpretation.

SQLite manages persistent event history and financial states, Jev screens events for escalation, and GPT-4.1-mini generates contextual explanations and retention strategies when requested.

A deterministic financial evaluator uses precise monetary calculations to assess proposed offers against available budgets and profitability constraints.

Finally, we integrated these components into a Streamlit dashboard, allowing business owners to explore customer histories, review AI-generated insights, and make the final retention decisions.

The System One Technical Unlock

The central technical unlock of SaveMyCustomers is transforming contextual financial intelligence from an occasional, manually requested AI task into a potentially continuous, event-driven capability.

Traditional accounting software records financial transactions.

Rule-based monitoring can flag predetermined thresholds, but struggles to interpret complex changes expressed through different financial signals and customer communications.

Conventional generative AI can provide contextual reasoning, but processing every invoice, payment, and customer event through a large language model introduces ongoing inference costs.

Jev changes where and how intelligence is applied.

Instead of treating every financial event as requiring full generative reasoning, Jev acts as a specialized first-stage decision engine, identifying meaningful changes and determining which events deserve deeper investigation.

What Our Benchmarks Demonstrated

On our 690-event Company 003 development benchmark:

85.0% escalation F1 achieved by Jev.
+28.2 percentage points in escalation F1 over our rule-based baseline.
+32.1 percentage points in escalation F1 over our XGBoost baseline.
22 fewer false escalation alerts than GPT-5.4-mini, alongside higher escalation precision.
~10.8× lower estimated per-event inference cost compared with GPT-5.4-mini under the supplied pricing assumptions.
Subsecond median hosted inference demonstrated by Jev.

Important Evaluation Trade-offs

While Jev demonstrated promising results, our evaluation also revealed limitations:

GPT-5.4-mini retained higher overall escalation F1 and recall.
The approximately 10.8× cost advantage is a preliminary estimate, not a verified production billing comparison.
Although Jev demonstrated subsecond median hosted inference, our experiments did not establish a direct latency advantage over GPT under equivalent request conditions.

The significance is the combination of contextual detection, selective escalation, and promising inference economics.

Rather than replacing generative AI entirely, our architecture assigns different responsibilities to different components:

Component	Responsibility
Deterministic software	Establishes accurate financial context
Jev System One	Identifies and prioritizes meaningful changes
Generative AI	Performs deeper investigation and develops recommendations
Financial guardrails	Evaluate proposed decisions
Business owner	Retains final decision-making authority

This creates a foundation for always-on financial relationship intelligence without requiring every event to undergo the same expensive reasoning process.

How SaveMyCustomers Satisfies the QuickBooks Track

QuickBooks: Reimagining Accounting With System One Models

SaveMyCustomers directly addresses the track's challenge: using System One capabilities to create meaningful new experiences for QuickBooks users that conventional approaches may be too costly, difficult, or limited to deliver continuously.

1. A Meaningful System One Technical Unlock

Our solution uses Jev as a contextual financial-event screening layer, rather than simply adding a chatbot to accounting software.

We demonstrated substantial improvements in escalation detection over our implemented rule-based and classical machine-learning baselines, alongside promising inference-cost advantages compared with conventional generative AI.

2. A New Accounting Experience

Instead of requiring business owners to manually search reports for problems, SaveMyCustomers introduces proactive, event-triggered financial relationship monitoring.

The system can identify changes in customer behavior, prioritize events that deserve attention, and help owners investigate potential revenue risks.

This changes the interaction from:

"What happened in my business?"

To:

"What is changing, why does it matter, and what can I do about it?"

3. Clear Value for QuickBooks Customers

Customer retention, accounts receivable, cash flow, and profitability are closely connected.

SaveMyCustomers helps small businesses:

Recognize financially meaningful customer changes earlier
Investigate potential revenue and relationship risks
Consider personalized retention strategies
Evaluate proposed actions against financial constraints

4. Evidence Through Realistic Evaluation

We did not rely solely on a working demo.

We developed a versioned, customer-isolated financial-event benchmark, compared Jev against rules, XGBoost, and GPT, and measured:

Classification quality
False escalation alerts
Inference latency
Estimated inference economics

We also extended testing into cross-company financial scenarios to investigate generalization and identify weaknesses requiring further development.

Our vision is to make QuickBooks not only a system of financial record, but a proactive financial intelligence partner that helps businesses protect the customer relationships behind their revenue.

Challenges we ran into

1. Distinguishing Meaningful Changes From Normal Fluctuations

One of our biggest challenges was distinguishing meaningful financial changes from ordinary business fluctuations.

A decline in purchasing does not always indicate customer loss, and a delayed payment does not necessarily mean a relationship is deteriorating.

Financial behavior depends heavily on historical context, requiring us to reconstruct customer states instead of analyzing isolated transactions.

2. Designing a Fair System One Evaluation

Another challenge was designing a fair evaluation for System One inference.

We needed to:

Construct realistic financial scenarios
Define meaningfulness and escalation labels
Prevent customer overlap between evaluation splits
Compare models using consistent metrics
Check for label inconsistencies and data leakage

This required treating benchmarking as a core engineering problem rather than an afterthought.

3. Understanding Model Trade-offs

Our experiments revealed important trade-offs.

Jev significantly outperformed our classical baselines on escalation detection, but GPT achieved stronger overall recall and F1.

Cross-company testing also exposed additional weaknesses in certain payment-warning and conflicting-signal scenarios.

These findings helped us identify where the System One screening layer needs further improvement.

4. Making AI Recommendations Financially Responsible

Generating useful retention recommendations required more than language-model output.

We needed to ensure that proposed offers were checked against financial constraints and that missing information did not result in unjustified approvals.

This led us to implement a deterministic financial feasibility evaluator with explicit Feasible, Rejected, and Unverified classifications.

What These Challenges Taught Us

These challenges reinforced our decision to build a hybrid architecture combining:

Deterministic financial computation + Specialized inference + Generative reasoning + Human oversight

Accomplishments that we're proud of

Our proudest accomplishment is building a working financial intelligence pipeline supported by measurable experimentation, rather than relying on AI-generated demonstrations alone.

Key Achievements

Achievement	Result
Financial events reconstructed	14,945
Supervised benchmark examples	9,024
Customer contexts	100
Scenario families	42
Synthetic companies explored	100
Industries represented	71
Jev escalation F1	85.0%
F1 improvement over rules	+28.2 percentage points
F1 improvement over XGBoost	+32.1 percentage points
Estimated Jev per-event cost advantage	~10.8×

We successfully reconstructed financial events, developed a supervised benchmark, and evaluated System One inference against multiple relevant alternatives.

We demonstrated 85.0% escalation F1 with Jev, significantly outperforming our rule-based and XGBoost implementations, while identifying promising cost characteristics for high-volume financial-event processing.

We are also proud of building the complete application workflow:

Financial state reconstruction
Event processing
Jev-powered System One screening
Selective AI investigation
Personalized retention strategy generation
Deterministic financial validation
Interactive business-owner dashboard

Most importantly, we built a system where AI assists financial decision-making without becoming the authority over financial truth or business actions.

Synthetic Data Modifications & Disclosure

Save My Customers was developed and evaluated using the synthetic QuickBooks-style accounting datasets provided for the hackathon.

I extended the provided data to create realistic customer retention scenarios and evaluate context-aware change detection. No real customer financial data was used.

What I Changed

1. Added customer behavior scenarios

I created 42 synthetic scenario families, including 36 message-based scenarios and 6 financial trend scenarios, to represent situations such as declining purchase activity, delayed payments, increasing financial risk, and improving customer relationships.

2. Generated evaluation examples

I constructed 9,024 synthetic benchmark examples using customer financial histories and scenario-based changes. These examples were used to evaluate meaningful-change detection and escalation decisions.

3. Tested across different business contexts

I used accounting data from multiple synthetic companies across different industries to evaluate whether the approach generalized beyond a single company's financial patterns.

4. Simulated incoming accounting events

For the product demonstration, I used synthetic QuickBooks-style invoices, payments, and customer events to simulate incoming accounting activity and demonstrate how historical context changes as events are processed.

Why I Made These Changes

The provided accounting data offered a foundation for financial analysis, but evaluating customer retention intelligence required scenarios with meaningful changes over time.

These extensions allowed me to:

Test how Jev System One responds to different customer behavior patterns.
Compare System One with alternative detection approaches.
Evaluate how purchasing and payment signals interact.
Demonstrate the product's event-processing and contextual alerting capabilities.

What we learned

1. Financial Intelligence Requires Context, Not Just Transactions

The biggest lesson was that financial intelligence requires context, not just transactions.

A single invoice or payment rarely tells the full story.

Understanding customer relationships requires examining how financial behavior evolves over time.

2. System One and Generative AI Can Complement Each Other

We learned that System One and conventional generative AI do not have to compete as complete replacements.

They can complement each other:

System One handles frequent contextual classification and escalation decisions.
Generative AI provides deeper investigation and recommendations when appropriate.

This hybrid design allows intelligence to be applied selectively.

3. No Single Model Wins Across Every Metric

Our benchmarking showed that no single model was strongest across every metric.

Jev offered a compelling balance of escalation precision and potential inference efficiency, while GPT performed better on some recall-oriented measures.

This taught us to evaluate models according to their responsibilities within the overall system rather than focusing on one headline metric.

4. Reliable Financial AI Requires Clear Boundaries

We learned the importance of:

Deterministic accounting calculations
Transparent uncertainty
Explicit financial feasibility checks
Human approval of business actions

The goal isn't to automate every decision. It's to make better decisions possible at the right time.

What's next for SaveMyCustomers

Our next step is to evolve SaveMyCustomers from a synthetic-data hackathon prototype into a more robust financial intelligence platform.

1. Authorized QuickBooks Integration

We plan to integrate authorized QuickBooks data through accounting APIs, enabling the system to work with real business financial records rather than relying exclusively on synthetic datasets.

2. Expanded Cross-Industry Evaluation

We want to strengthen our evaluation by:

Expanding cross-industry testing
Improving payment-deterioration detection
Addressing conflicting financial signals
Validating performance on additional held-out scenarios

3. Production-Grade Financial Event Processing

We plan to develop more robust infrastructure for:

Continuous financial-event processing
Secure multi-business access
Persistent monitoring
Improved alert prioritization

4. Better Retention Recommendations

We want to gather feedback from business owners to understand which recommendations are most useful, actionable, and financially responsible.

This feedback will help us improve the relevance of retention strategies and the usefulness of the owner-facing experience.

5. The Long-Term Vision

Longer term, we envision SaveMyCustomers becoming an intelligent financial relationship layer that helps businesses:

Monitor customer health
Recognize emerging risks
Understand changing financial behavior
Make financially responsible retention decisions
Intervene before valuable opportunities disappear

Accounting software has always helped businesses understand where their money went.

We want to help businesses understand which customer relationships need attention—and what they can do before revenue is lost.

Because the best time to save a customer is before you lose them.

Built With

jev
json
openai
python
scikit-learn
sqlite
streamlit
xgboost
yaml