MaritimeIQ

Inspiration

At the SOLIXEmpower Day 1 keynote, one slide stuck with us: the models are ready, but most enterprise data isn't. It named three blockers: data lives in silos, context is missing, and trust has to come first.

We wanted to see what that looks like in a real industry, and shipping was the perfect stress test. Shipping moves most of the world's trade, yet a simple question like "Why are our vessels waiting at anchor off Long Beach?" still needs a data engineer and several days of work. The data exists. It is just scattered across AIS feeds, port agent logs, bills of lading, crew manifests and carrier schedules, each with its own format, its own codes and its own way of naming the same ship.

So we asked: what if AI agents did the work of making data AI-ready, with a human guiding them wherever they are unsure?

What it does

MaritimeIQ is a step-by-step guided application that walks a user from messy, pre-Bronze source files all the way to clean, AI-ready Gold tables. An agentic framework does the heavy lifting, and a human-in-the-loop (HITL) steers it at every point where judgment matters.

1. Sources → Bronze: get everything in one place

An Ingestion agent pulls in data from six batch formats (CSV, JSON, XML, Excel, fixed-width) plus a near-real-time AIS stream. It figures out awkward details on its own, such as a header that isn't on row 1, fixed-width column boundaries, or which XML element is the record. Everything lands in Bronze as Delta Lake tables with lineage columns. At this stage the data is raw, messy and not usable, which is exactly the problem we set out to solve.

2. Bronze → Silver: clean it, understand it, connect it

This is where most of the agentic work happens:

• Context agent builds a semantic layer for the whole dataset. It proposes what each column, code and join means (for example, AIS status 1 means "at anchor" and agent status AWB means "awaiting berth"), tags PII, and attaches a confidence score and a short rationale to every proposal. It can look up reference tables (UN/LOCODE, AIS code tables, World Port Index) rather than guess.
• Entity resolution agent links disparate sources by creating master records for vessels, ports and customers. One ship can show up as an IMO number, an MMSI, a call sign and a name that changed after a sale. One port can appear as USLGB, "Long Beach" and "POLB". The agent blocks on hard identifiers, fuzzy-matches names, and only asks the LLM about genuinely ambiguous pairs.
• Transformation agent turns approved semantics into Spark SQL, runs quality checks, masks PII, and repairs its own SQL when a check fails.

Human-in-the-loop as guidance, not just approval. When the agents find a gap they cannot fill confidently (an unknown code, an uncertain match, a missing join key), they put it in a review queue and ask the user. The user approves, edits or rejects. Every correction is saved and fed back as an example on the next run, so the system needs less help with each new source.

LLM-as-a-Judge. A separate judge model grades the agents' work against a rubric: Context proposals (correct, plausible, wrong), Insight answers (faithful to the SQL result?), and Gold SQL (does it match the question's intent?). Low scores jump to the top of the human queue. We also measure the judge itself against human labels so we know how far to trust it.

The result is the Silver layer: clean, typed, deduplicated, PII-masked tables with their meaning documented.

3. Silver → insights: find what matters

An Insight agent explores the Silver tables on its own to surface the patterns and metrics most critical to the business, such as port congestion, time at anchor versus at berth, turnaround, and carrier performance. Live metrics refresh from the AIS stream.

It also offers guardrailed Text-to-SQL. Users ask questions in plain English and get an answer along with the SQL and the glossary terms it used. Questions are strictly scoped to the data that actually exists. If a question can't be answered from the available tables, the user sees a simple message saying so instead of a confident guess.

4. Silver → Gold: the AI-ready layer

When a question needs a reusable aggregate (for example daily_port_congestion), the Insight agent proposes a Gold table. It is only built after a human approves it. Gold is the cleanest, most AI-ready form of the data that first arrived in Bronze.

Trust at every step

Every agent reaches the data only through MCP tools, and every tool call passes through a deterministic governance guard. The guard checks each call against a YAML policy, masks PII after the query runs (so no prompt can talk its way around it), blocks destructive operations like DROP or TRUNCATE, requires an approval ID for Gold writes, and logs every decision to an audit table.

How we built it

Layer	Tools
Processing	Apache Spark 3.5 + PySpark (local mode)
Table format	Delta Lake (delta-spark)
Streaming	Spark Structured Streaming over a replayed AIS feed
Parsing	Spark readers, spark-xml, spark-excel, substring parsing for fixed-width
Agent tools	MCP servers (official Python SDK): lakehouse, reference, review
Orchestration	Agent state machine with pause points for human review
Entity resolution	Spark blocking on IMO / MMSI / call sign / tax ID + rapidfuzz in a UDF
Governance	YAML policy + Python guard inside the MCP gateway
PII masking	Spark UDFs, regex plus LLM tags, salted-hash tokens that stay joinable
LLM output	Structured JSON validated with Pydantic, one retry, then escalate to a human
UI	Streamlit with an agent activity feed on every tab

Data. Vessel positions and port reference data are real (NOAA MarineCadastre AIS, UN/LOCODE, NGA World Port Index). Commercial and crew data are synthetic, because that data is private by nature. We deliberately planted errors (stripped IMOs, misspelled names, one vessel split across two MMSIs, company name variants, cryptic status codes) and recorded the correct answers in a ground-truth file, so every agent can be scored.

Evaluation. Our harness scores each stage against ground truth. For entity resolution we report:

$$ \text{Precision} = \frac{TP}{TP + FP}, \qquad \text{Recall} = \frac{TP}{TP + FN}, \qquad F_1 = \frac{2 \cdot P \cdot R}{P + R} $$

Proposals are routed to a person whenever the agent or the judge is not confident enough:

$$ \text{route}(p) = \begin{cases} \text{auto-approve} & \text{if } c_p \ge \tau \text{ and judge}(p) = \text{correct} \ \text{human review} & \text{otherwise} \end{cases} $$

Results

Metric	Result
Semantic-layer accuracy after review	[ ]%
PII detection recall (planted PII)	[ ]%
Entity resolution F1 (vessels)	[ ]
NL question accuracy: raw Bronze vs Silver + context	[ ]% → [ ]%
Adversarial prompts blocked	[ ] / 5
Corrections needed: source 1 → source 2	[ ] → [ ]
LLM judge agreement with humans	[ ]%
AIS ping → live metric latency	[ ] s

Challenges we ran into

• Identity chaos. Joining ships across systems was harder than we expected. Names change after a sale, IMO numbers go missing, and one physical vessel can broadcast under two MMSIs. Exact matching silently dropped rows and fuzzy matching alone double counted, so we combined blocking on hard IDs with fuzzy names and saved the LLM for the truly unclear cases.
• Knowing when the AI should ask. Getting the HITL balance right took iteration. If the agent asks too often, the human becomes the pipeline. If it asks too rarely, errors flow straight into Silver. Confidence scores plus the judge's grade gave us a workable threshold.
• Keeping PII away from the LLM. Profiling a column means showing the model samples, but we did not want it to see raw passport numbers. We switched to masked samples as soon as a column is tagged as PII.
• Scoping Text-to-SQL honestly. It is easy for a model to produce plausible SQL for a question the data cannot answer. Teaching it to decline cleanly mattered as much as teaching it to answer.
• Malformed LLM output. Even with structured output, responses occasionally broke the schema. Pydantic validation with a single retry, then escalation to a human, kept the pipeline moving.
• Spark setup on a laptop. Java versions, Delta and XML/Excel connectors ate real time early on. Pinning versions and having one person own the environment saved us.
• Streaming in a live demo. Rather than run Kafka, we replayed real AIS data in time order through Structured Streaming, which kept the demo stable while still showing micro-batches arriving.

Accomplishments that we're proud of

• A single end-to-end flow that goes from five messy sources to governed Silver and an approved Gold table.
• A semantic layer the agents build themselves, which gets better with each human correction.
• Governance at the agent level, controlling what each agent may do, not only what a user may see.
• Every claim in our pitch is backed by a number from the eval harness.

What we learned

• Context is the real bottleneck. The same LLM answered questions far more accurately on Silver with a semantic layer than on raw Bronze. The model was never the limiting factor; understanding the data was.
• Humans are most valuable as teachers. A correction that is reused is worth far more than an approval that is forgotten.
• Trust must be deterministic. Guardrails that live inside the prompt can be argued with. Guardrails in code, sitting between the agent and the data, cannot.
• Measure the judge too. An LLM grading another LLM is only useful once you know how often it agrees with people.

What's next for MaritimeIQ

• Schema-drift handling: a new column or source triggers the pipeline, and the agent proposes semantics for review instead of breaking downstream jobs.
• Insight to action: for example, a vessel waiting at anchor for more than 24 hours automatically drafts an alert or ticket.
• Apache Iceberg output alongside Delta.
• Beyond shipping: the same pattern fits banking, healthcare and any regulated industry where data is fragmented, cryptic and sensitive.

Built With

• apache-spark
• delta-lake
• langgraph
• mcp
• pydantic
• pyspark
• python
• rapidfuzz
• spark-structured-streaming
• streamlit