EQUALL — Diagram Lens

Inspiration

A graph can tell a thousand stories. A rising line can signal progress, a sudden drop can indicate a problem, and the difference between two points can change how we interpret an entire dataset.

But what happens when someone cannot see that graph?

For visually impaired students, the challenge extends beyond accessing individual numbers. Understanding trends, comparing values, interpreting changes, and developing statistical reasoning can all depend on information traditionally communicated through visual representations.

Existing accessible chart tools can describe data, provide sonification, or enable nonvisual navigation. We wanted to explore a different question:

What if a graph could not only be explored without sight, but could also teach someone how to understand it?

That question led to EQUALL — Diagram Lens.

Our goal was to make statistical learning more accessible by connecting data exploration with interactive instruction, misconception detection, and opportunities to demonstrate understanding.

Don't just hear the graph. Understand it.

What it does

EQUALL transforms supported line graphs into accessible, interactive learning experiences.

Instead of merely reading numbers aloud, it helps learners explore relationships, reason about changes, receive targeted instructional support, and assess their understanding.

The experience follows three stages:

Explore → Understand → Learn

Explore: Make the data accessible

Learners can:

• Import CSV datasets and turn them into interactive line-graph lessons.
• Navigate individual data points using keyboard controls.
• Listen to graph values and descriptions.
• Explore graph patterns through pitch-based sonification.
• Ask natural-language questions about the data.
• Adjust explanation detail, hint support, and speech speed.

EQUALL also includes experimental chart-image extraction. An AI model proposes structured values from an uploaded chart, which must be reviewed and confirmed before being used for a lesson.

Understand: Go beyond the numbers

Consider a graph showing student performance over six months.

EQUALL asks:

"Which month experienced the largest increase?"

A learner answers "June" because June has the highest overall value.

However, the highest value and the largest increase are not necessarily the same.

EQUALL identifies this supported misconception and provides a targeted explanation grounded in the actual dataset.

Instead of immediately revealing the answer, it can guide the learner through comparisons and smaller reasoning steps.

The learner can ask for hints, request a different explanation, or revisit their reasoning.

Learn: Demonstrate understanding

EQUALL uses follow-up assessments to check whether the learner can apply the concept.

For supported instructional pathways, this includes a transfer question using different data.

The system records observable learning evidence, including:

• The learner's original answer.
• The misconception identified.
• Whether assistance was provided.
• Subsequent assessment responses.
• Whether the learner demonstrated the concept independently or with help.

A teacher-facing summary makes these interactions inspectable.

EQUALL does not claim to establish long-term mastery from a single successful response. It records what the learner actually demonstrated during the session.

How we built it

EQUALL is built around a fundamental engineering principle:

AI should help learners interact with information, but mathematical correctness should not depend on a language model guessing the answer.

Model Context Protocol architecture

We implemented a self-hosted Model Context Protocol (MCP) server using the official TypeScript SDK.

The server exposes 13 functional tools through Streamable HTTP, using MCP protocol version 2025-11-25.

These tools support graph exploration, dataset processing, statistical relationships, learning sessions, assessments, and learner preferences.

The application uses a server-side MCP client to execute real tool calls.

This architecture provides a modular foundation for an Alexa+-oriented conversational experience. The hackathon implementation uses an independent web interface and MCP server; it is not a production deployment inside Alexa+.

Deterministic statistical engine

EQUALL's statistical engine is responsible for:

• Numerical comparisons and calculations.
• Maximum and minimum values.
• Trends and interval changes.
• Assessment generation and grading.
• Supported misconception classification.
• Hint progression.
• Learner-state transitions.

We deliberately separated mathematical reasoning from generative language-model output.

Gemini-assisted interaction

Gemini helps interpret unfamiliar natural-language questions and select appropriate read-only MCP tools.

The statistical facts presented to the learner come from deterministic tool results.

Gemini also supports optional server-side voice transcription and experimental chart-image extraction.

Accessible interaction

We built keyboard navigation, speech controls, a graph sonification player, accessible text alternatives, and configurable learning preferences.

For CSV import, we added column mapping, preview tables, input validation, and row-specific error messages.

The application also includes persistent learning sessions, structured progress reporting, and a repeatable demonstration workflow.

Our implementation uses TypeScript, Node.js, JavaScript, HTML, CSS, the MCP SDK, Gemini APIs, PapaParse, the Web Audio API, browser speech capabilities, and automated testing with Vitest and Playwright.

Challenges we ran into

1. When mathematically plausible AI answers were wrong

One of our most important discoveries came from adversarial testing.

Our initial grounding validator checked whether an AI-generated response used numbers and labels present in the graph.

But that wasn't enough.

A statement could mention real values while making a false comparison, identifying the wrong maximum, or reversing the direction of a trend.

We redesigned the interaction so Gemini could interpret requests and select tools, while the deterministic engine retained responsibility for statistical statements.

2. Floating-point precision and incorrect grading

We encountered cases where a displayed answer such as 0.3 could be graded incorrectly because of floating-point representation.

We introduced consistent numerical handling and regression tests to align the displayed answer with the grading logic.

3. Silent CSV data corruption

Malformed rows containing unquoted thousands separators could be interpreted incorrectly.

For example, a value intended as 1,200 could be split into multiple columns.

We strengthened CSV validation to reject malformed rows and provide actionable error messages instead of silently building lessons from corrupted values.

4. Making audio interaction reliable

Browser speech capabilities were inconsistent, particularly in Safari.

We introduced an alternative recording and server-side transcription pathway, alongside coordinated audio controls for speech, microphone input, and sonification.

The audio system also required careful handling of Pause, Resume, Stop, and navigation between lessons.

5. Teaching without revealing the answer

An instructional system must balance guidance and independent reasoning.

We had to ensure that graph descriptions, explanations, and hints did not accidentally reveal answers to active assessments.

We introduced tests for answer leakage, misconception-specific feedback, and appropriate learning-state transitions.

Accomplishments that we're proud of

1. A real, functioning MCP-based learning application

EQUALL implements 13 discoverable MCP tools with actual operations and persistent learning state.

2. Turning unfamiliar data into lessons

CSV files can be imported and used in the same exploration, reasoning, assessment, and reporting workflow as built-in datasets.

3. Combining accessibility with instruction

EQUALL connects keyboard navigation, graph descriptions, sonification, misconception-aware teaching, and structured follow-up assessment.

4. Keeping numerical reasoning deterministic

Rather than relying on a language model to calculate statistical answers, the system uses verifiable computations and structured instructional logic.

5. Building for reliability through independent auditing

We repeatedly tested EQUALL using unfamiliar datasets, incorrect answers, malformed inputs, accessibility checks, and adversarial statistical claims.

At the latest completed development checkpoint, the project passed 251 unit, protocol, and agent tests, along with 90 browser test executions across repeated runs. These are automated engineering results, not evidence of educational effectiveness.

6. Maintaining a transparent view of limitations

Experimental image extraction requires human confirmation. The supported teaching concepts remain focused, and formal testing with visually impaired learners remains future work.

What we learned

The biggest lesson was that accessibility is not simply the ability to hear information that was previously visual.

Access to data and understanding of data are different challenges.

A learner may be able to hear every point on a graph and still struggle to distinguish the highest value from the greatest increase.

We learned that accessible education requires navigable information, meaningful explanations, opportunities to make mistakes, and structured support for reasoning.

We also learned an important architectural lesson: combining language-model flexibility with deterministic tools can make an application more reliable when correctness matters.

Finally, independent testing proved essential. Several of our most consequential defects were not obvious during normal demonstrations. They emerged only when the system was challenged with unexpected inputs and alternative user behaviors.

What's next for EQUALL — Diagram Lens

EQUALL is currently a hackathon prototype focused on accessible line-graph interpretation and statistical reasoning.

Future development could expand the experience through:

• Additional graph types, including bar charts, scatter plots, and histograms.
• Broader support for mathematical and statistical concepts.
• More reliable multilingual voice interaction.
• Stronger chart-image digitization and verification.
• Further screen-reader and assistive-technology compatibility testing.
• Structured evaluations with visually impaired learners and educators.
• More detailed instructional progress tracking.

Before making claims about educational effectiveness, we would want to evaluate the system with actual learners and measure its usability and instructional outcomes.

Our broader ambition is simple:

A student's ability to understand data should not be determined by their ability to see a graph.

EQUALL is one step toward making that possible.

Explore. Understand. Learn.

Built With

• artificial-intelligence
• css
• gemini
• html
• javascript
• node.js
• typescript