OnTripFix

No Trip | No Laptop | Prod Fix

Comment

Inspiration

Every on-call engineer and data platform lead knows the dread of being away from their laptop—traveling, on a weekend trip, or outside regular office hours—when critical batch ETL pipelines crash. Traditionally, an on-call alert like PagerDuty means having to find stable Wi-Fi, open a laptop, decipher stack traces across terminal logs, dig through Jira tickets and Confluence runbooks, and manually patch database schemas while incident timers tick.

We asked ourselves: What if an engineer on the road could resolve a production data pipeline failure in 60 seconds simply by answering a phone call?

With OnTripFix, the data platform doesn't just alert—it actively investigates, consults knowledge bases, dials the on-call engineer via Call-E Voice AI, validates their identity securely, summarizes the proposed fix, captures spoken authorization, and automatically executes deterministic schema healing with zero laptop required.

YouTube Video Link - https://youtu.be/SCRXJr_r0qA

What it does

OnTripFix is an autonomous incident response and auto-remediation voice agent for Apache Airflow data pipelines:

  1. Incident Interception: When an Airflow DAG task fails (e.g., Sunday night Retail Inventory ETL failing due to an unexpected missing column inventory_status), Airflow's on_failure_callback immediately posts the error payload to OnTripFix's FastAPI incident gateway.
  2. Context Enrichment & Knowledge Retrieval:
    • Jira REST API: Searches historical incident tickets and JQL for matching bugs (or falls back to previous incident matches like RETAIL-4021).
    • Confluence Playbooks & Team Calendar: Looks up standard operating procedures and queries the active on-call schedule to identify the responsible engineer.
    • AI Bug Diagnosis: If no existing playbook exists, LangChain AI diagnosis generates the necessary deterministic SQL remediation script.
  3. Interactive Call-E Voice AI Escalation:
    • Places an outbound voice call to the on-call engineer's mobile device.
    • Security First: Prompts the engineer to speak their Employee Security User ID before any operational details or remediation steps can be approved.
    • Summarizes the failing DAG, task, Jira reference, and proposed SQL fix (ALTER TABLE daily_store_inventory_agg ADD COLUMN inventory_status TEXT...).
    • Captures spoken approval and resolution instructions as structured JSON via Call-E's result_schema.
  4. LangGraph StateGraph Auto-Remediation:
    • Parses the spoken resolution instructions.
    • Executes the DDL patch safely on the SQLite data store.
    • Triggers an automated Airflow DAG task retry.
    • Validates post-remediation data load integrity (verifying records processed, stock levels, and revenue aggregations).
    • Transitions the corresponding Jira issue status to Done with full audit logs.
  5. Call-E Confirmation Voice Call: Dials the engineer back with a voice confirmation stating the pipeline is 100% green for Monday morning business operations.
  6. Real-time Telemetry Dashboard: A React frontend providing live queue inspection (Error Queue, Resolution Queue), incident audits, and active on-call roster management.

How we built it

  • Voice AI Agent: Integrated the Call-E Python SDK / REST API (api.heycall-e.com/v1/calls) with strict E.164 phone formatting, result schemas for structured decision extraction, and regional/multilingual localization (supporting US English, India English, Hindi, and Tamil).
  • Remediation Engine: Built with LangGraph StateGraph and Google Gemini (Gemini 2.5 Flash) tool binding to parse spoken instructions, apply deterministic SQLite patches, trigger Airflow retries, and validate database consistency.
  • Workflow & Webhook Server: Developed using FastAPI with background threading queues (Error Queue consumer and Resolution Queue worker) for resilient, non-blocking telemetry enrichment and dispatch.
  • Observability & Ticketing: Implemented bi-directional Atlassian Jira REST API integrations (comment logging, transition updates) and Confluence runbook matching with local fallback safety layers.
  • Frontend Dashboard: Built with React and Vite, featuring glassmorphism aesthetics, live queue inspection, audio call logs, and incident telemetry.
  • Pipeline Simulation: Engineered an Apache Airflow DAG (retail_inventory_etl) and end-to-end Python simulation harness (simulate_e2e_incident.py) reproducing production database crashes, automated triage, and verified recovery.

Challenges we ran into

  1. Deterministic Execution vs. LLM Non-Determinism: Voice instructions from an engineer must be translated into precise, syntactically correct SQL. We designed LangGraph nodes with strict tool-calling validation rather than allowing raw text generation directly against the production database.
  2. Security & Authentication over Voice: Preventing unauthorized voice approvals required embedding a mandatory User ID challenge step into the Call-E call task and enforcing user_id_validated: True in the result_schema before any remediation node could trigger.
  3. Public Privacy & Telephony Standards: Balancing live Call-E testing with open-source reference repository rules required sanitizing personal numbers, implementing phone masking (mask_phone_number()), isolating credentials into gitignored sample templates (*.json.example), and ensuring default dry-run mock paths work offline without live telecom costs.

Accomplishments that we're proud of

  • True Hands-Free Resolution: Complete end-to-end incident handling from Airflow crash to verified database repair in under 60 seconds without typing a single terminal command.
  • Security-Gated Human-in-the-Loop: Autonomous remediation without blind agent actions—every state change requires authenticated human voice authorization.
  • Production-Grade Architecture: Decoupled async queues, automatic fallback mechanisms (graceful degradation if Jira or Confluence APIs are unreachable), and localized multi-language on-call roster configurations.
  • Seamless Developer Experience: Runs with one command (python apps/python/ontripfix/simulate_e2e_incident.py) and passes a 10-test automated verification suite (pytest apps/python/ontripfix/tests/test_flow.py).

What we learned

  • Structured Voice Extraction is Transformative: Call-E's server-side result_schema validation removes the need for secondary prompt chains—spoken confirmations naturally translate into typed, reliable data models.
  • Progressive Fallbacks are Essential for Reliability: In on-call systems, external dependencies (calendars, ticketing systems) can fail simultaneously with pipelines. Building deterministic local fallback layers ensures the system stays operational under severe outages.
  • Voice is the Ultimate Low-Latency Mobile Interface: For urgent operational approvals, a direct telephone call provides vastly higher responsiveness than Slack pings or email alerts.

What's next for OnTripFix

  • Multi-Cloud Data Warehouse Support: Extending LangGraph database patch tools to support Snowflake, Google BigQuery, and Amazon Redshift.
  • Smart Escalation Ladders: Automatically cascading to secondary and tertiary on-call leads if the primary engineer does not answer or declines the fix within a designated timeout window.
  • Interactive Multi-Turn Voice Runbooks: Enabling engineers on the call to ask Call-E questions (e.g., "What was the error log from task 3?" or "Have we seen this error in the last 30 days?") before granting remediation approval.

Built With

Share this project:

Updates