Faultline

• 

  Faultline hero section

Inspiration

AI is becoming part of how students, researchers, and analysts understand the world around them.

We use it to break down research papers, interpret statistics, compare benchmarks, understand evidence, and answer questions that would normally take much longer to investigate ourselves.

And most of the time, the experience is simple:

Ask a question. Get one confident answer. Move on.

But that made us wonder about something.

What happens if we ask the same underlying question in a slightly different way?

What happens if we give that question to a completely different AI model?

Would the conclusion actually stay the same?

The problem is that normal AI interfaces do not really show us this. An answer can sound incredibly confident while still being sensitive to the way we phrased the question or which model happened to answer it.

That led to the question behind FAULTLINE:

Before relying on an AI-generated research conclusion, can we stress-test how stable it actually is?

That is what we built FAULTLINE to do.

FAULTLINE is a pre-flight check for AI-generated research conclusions. It does not claim to determine absolute truth. Instead, it exposes when an AI conclusion changes under controlled variations and shows the user exactly which claims deserve a second look.

What it does

Imagine a student reading a study and asking an AI:

"Does this study prove that remote work increases productivity?"

Normally, they would receive one response and have to decide how much they trust it.

FAULTLINE takes a completely different approach.

Instead of generating another answer, it runs an experiment on the answer.

FAULTLINE:

1. Takes the user's original research question.
2. Generates controlled variations of that question.
3. Screens those variations to make sure they preserve the original intent.
4. Tests the original question and accepted variations across different AI model families.
5. Breaks the resulting responses down into individual claims.
6. Compares those claims across both wording and model conditions.
7. Measures how stable the overall conclusion remained.
8. Shows where meaningful changes occurred.

The result is not another wall of AI-generated text.

It is an interactive diagnostic of how the AI conclusion behaved.

Experimental Stability

The first thing FAULTLINE shows is an Experimental Stability score.

This measures how consistently the conclusion survived the conditions we tested.

There is one distinction we intentionally make very clear throughout the product:

Stability is not truth.

A conclusion can remain stable across every model and every prompt variation and still be wrong.

FAULTLINE is not trying to replace fact-checking. It is trying to expose uncertainty that a normal AI interface hides.

Semantic Fault Map

This is where FAULTLINE goes deeper than simply asking several models the same question.

The Semantic Fault Map identifies the wording transformations associated with the biggest changes in the resulting conclusions.

A user can inspect a fault and see what wording changed, what alternative wording was tested, which claims were affected, and which model families changed their conclusions.

Instead of only saying:

"The models disagreed."

FAULTLINE tries to answer the much more useful question:

"Where did the answer start becoming unstable?"

Safe Core

After identifying disagreement, we wanted FAULTLINE to still give the user something actionable.

The Safe Core contains the claims that remained comparatively stable throughout the stress test.

These are not guaranteed facts. They are simply the parts of the answer that survived the tested variations more consistently.

What Needs Verification

This became one of the most important parts of FAULTLINE.

Finding disagreement is interesting, but a user still needs to know what to do with it.

So FAULTLINE identifies the claims that changed enough across the experiment to deserve an independent check before the user cites them, submits them, or relies on them.

The goal is not to tell someone:

"This claim is false."

The goal is to tell them:

"This is the part of the AI answer you should double-check."

That turns model instability from a technical measurement into something a student or researcher can actually use.

How we built it

FAULTLINE is a full-stack TypeScript application built with Next.js and React.

From the beginning, we structured the analysis pipeline as a controlled experiment instead of a normal chatbot interaction.

For every analysis, FAULTLINE varies two different things.

The first is the prompt.

We generate candidate semantic perturbations of the original question and screen them before allowing them into the experiment. This is important because if a rewrite accidentally changes the actual question, comparing the resulting answers would not tell us anything useful.

The second is the model family.

For heterogeneous model inference, we integrated Featherless AI.

Our experiment uses model families including:

• Qwen with Qwen3-32B
• Google Gemma
• Mistral with Mistral Small 24B

Featherless is not just an AI API we added to the project for the sake of having another model provider.

It actually makes the core experiment possible.

If FAULTLINE repeatedly asked one model the same question, we would mostly be measuring that model against itself.

Instead, we use different model families as separate experimental perspectives.

That gives FAULTLINE two different signals:

Prompt sensitivity: Does the conclusion change when the same underlying question is phrased differently?

Model sensitivity: Does the conclusion change depending on which model family answers it?

Featherless gives us a common inference layer for running those heterogeneous open models without designing an entirely separate integration for every family.

And importantly, we never treat majority agreement between those models as truth.

After collecting the responses, FAULTLINE analyzes them at the claim level rather than comparing raw paragraphs. It then derives the stability signals, disagreement matrix, Semantic Fault Map, Safe Core, and What Needs Verification output that the user sees.

We also built a full technical trace so the experiment does not have to be a black box. The interface can expose the model family, actual model ID, experimental condition, latency, status, and failures behind the final result.

Challenges we ran into

One of the hardest problems was figuring out what "disagreement" should actually mean.

Two models can write completely different paragraphs while still making essentially the same claim. Comparing raw text would make FAULTLINE report disagreements that were not really disagreements at all.

That pushed us toward comparing answers at the claim level instead.

Another major challenge was generating useful prompt variations.

If we change the wording too little, we learn almost nothing. If we change it too much, we might accidentally ask a different question.

Because of that, candidate perturbations are screened before they are admitted into the experiment.

Model latency and rate limits

One of the biggest practical limitations we encountered was live inference latency and provider rate limiting.

FAULTLINE intentionally performs much more inference than a normal chatbot request.

We are not sending one prompt to one model. We are testing multiple prompt conditions across multiple model families and then analyzing the resulting claims.

That makes the experiment much more informative, but it also makes it more computationally expensive.

During development, the Featherless-hosted models had different latency and availability characteristics. Some Gemma requests, for example, encountered HTTP 429 rate limits, while other model calls could take significantly longer to return.

Because later stages depend on those model outputs, slow or rate-limited inference can increase the runtime of a complete live experiment significantly.

We did not want to hide that limitation.

Instead, FAULTLINE explicitly tracks experimental coverage.

If one model fails, times out, or becomes rate-limited, the successful observations are preserved. Coverage is reduced, the result is marked as partial, and FAULTLINE never silently invents the missing response.

For our recorded walkthrough, we use FAULTLINE's deterministic demo fixture so external provider latency cannot interrupt the demonstration of the complete analysis workflow. The application separately implements the live Featherless inference pipeline.

This limitation also showed us one of the most important engineering problems we would tackle next: making robustness testing fast enough to feel interactive.

Being honest about what FAULTLINE can prove

There was also a more fundamental challenge.

It would be very easy to build this project and say:

"Three models agreed, so this must be true."

But that would be wrong.

Multiple models can share biases, training data, assumptions, and even the same mistakes.

The reverse is also true. A model changing its answer does not automatically mean the original answer was false.

That is why one principle became central to the entire product:

Instability does not imply falsehood, and consensus does not imply truth.

FAULTLINE measures behavioral stability under controlled conditions. It does not claim to be an oracle.

Accomplishments that we're proud of

What we are most proud of is that FAULTLINE became much more than three AI responses placed next to each other.

The final system creates an actual experiment around a user's question and then turns the result into something understandable.

Some of the parts we are especially proud of include:

• Controlled semantic prompt perturbation
• Screening of prompt variations before experimentation
• Multi-model experimentation through Featherless AI
• Separation of prompt sensitivity from model sensitivity
• Claim-level comparison instead of raw text comparison
• Semantic Fault Map visualization
• Deterministic Safe Core extraction
• Actionable What Needs Verification output
• Explicit experimental coverage reporting
• Graceful handling of provider failures and rate limits
• A technical trace exposing the underlying model runs
• A responsive full-screen experimental interface
• Server-side protection of provider credentials

We also built automated validation around FAULTLINE, including 30 unit tests and end-to-end browser tests, alongside TypeScript checking, linting, responsive testing, and production build verification.

But the part we are probably most proud of is much simpler:

FAULTLINE does not respond to uncertainty by generating another confident paragraph.

The uncertainty is the product.

What we learned

Building FAULTLINE changed the way we thought about AI reliability.

At first, it is really tempting to think that if several models agree, we have somehow verified the answer.

We quickly realized that is not a defensible assumption.

The more interesting question became:

How dependent is this conclusion on the conditions under which the AI produced it?

That changed the direction of the project.

We stopped thinking about FAULTLINE as something that should tell users whether an answer is right or wrong.

Instead, we started thinking about it as something that should show users where they need to be more careful.

We also learned that uncertainty by itself is not very useful.

If FAULTLINE only tells someone:

"Your answer is 42% stable."

they still do not know what to do next.

That is why the Safe Core and What Needs Verification became such important parts of the product.

One shows what survived.

The other shows what deserves another look.

What's next for FAULTLINE

The biggest immediate priority is speed.

The current architecture performs substantially more inference than a conventional AI application because FAULTLINE intentionally evaluates multiple semantic variations across multiple model families.

Under provider rate limits or high inference latency, a complete live experiment can therefore take much longer than the interactive experience we ultimately want.

The next engineering iteration would focus heavily on reducing that latency through:

• parallel execution of independent model conditions
• batching compatible inference requests
• caching repeated experimental conditions
• adaptive experiment sizing once enough evidence has been collected
• latency-aware model selection and fallback
• better retry and backoff behavior around HTTP 429 responses

We would also want to test FAULTLINE with real students, researchers, and analysts.

The important question for us is not simply whether people think the interface is interesting.

We want to measure whether exposing semantic instability actually helps someone notice a research conclusion they otherwise would have trusted without checking.

From there, the same methodology could expand into richer evidence grounding, larger controlled evaluations, additional perturbation strategies, and testing how conclusions change across different versions of the same model.

The long-term idea is simple.

Spellcheck checks our writing before we submit it.

Tests check our software before we deploy it.

AI is increasingly influencing what we learn, write, and believe.

FAULTLINE asks whether an AI-generated research conclusion deserves one more check before we trust it.

Built With

• ai-reliability
• ai-safety
• artificial-intelligence
• eslint
• featherless-ai
• gemma
• generative-ai
• llm
• metamorphic-testing
• mistral
• nextjs
• node.js
• playwright
• prompt-engineering
• qwen
• react
• semantic-analysis
• typescript
• vercel