Human vs. System in the Loop

• GIF

  Human vs. System in the Loop

Inspiration

Everyone talks about keeping a human in the loop. We asked a different question: what happens when we put the system itself in the loop?

Our idea connects three systems: language models, financial markets and political institutions. Public congressional trade disclosures provide a concrete test case. A high-profile trade might attract attention but does it fit a particular investor’s constraints, and can an AI explain its recommendation with evidence?

We wanted to move beyond confident answers toward accountable decisions, while giving the Apertus team a transparent way to compare its model with peers.

What it does

Human vs. System in the Loop is a decision-support and model-evaluation prototype. It gives multiple language models the same trade evidence and investor mandate, then compares their BUY, HOLD, SELL or ABSTAIN signals.

Four investor profiles introduce different constraints, such as sector exclusions, investment horizons and diversification requirements. The benchmark is designed around public congressional trade disclosures from 2024.

The JEV control layer evaluates more than agreement. It audits evidence, checks timing and constraint compliance, and explains why model outputs are selected or discounted before producing a final signal.

For the Apertus team, the key output is a traceable decision log comparing Apertus with its peers: each recommendation, its supporting evidence, its fit with the investor mandate and the evaluator’s reasoning. Users see the reasoning; model developers see opportunities to improve.

How we built it

We structured the prototype as a modular pipeline:

1. Disclosure ingestion: capture public transaction information and its source.

2. Evidence packet: organize the facts, transaction dates and disclosure delay.

3. Investor mandate: attach the profile’s hard constraints and preferences.

4. Model comparison: present the same inputs to Apertus and peer models.

5. JEV evaluation: assess outputs across eight rubrics constraint compliance, evidence, timing, instrument, fit, uncertainty, consistency and schema.

6. Decision delivery: expose the final signal and its audit trail, with a separate historical benchmark view.

An interactive walkthrough makes these stages inspectable. Our three-minute explainer combines the supplied prototype footage with narration, captions and sketch animations.

The prototype illustrates the workflow; it should not be mistaken for a completed, independently validated benchmark of every 2024 trade.

Challenges we ran into

The central challenge was defining a fair comparison. Models need equivalent evidence and mandates, not different information hidden inside different prompts.

Timing also matters. A transaction date is not the same as the date its disclosure became public. Historical evaluation must respect that distinction to avoid using information an investor could not yet have known.

Another challenge was separating an attractive trade from an admissible one. A hard investor constraint must not disappear because a model sounds persuasive. Missing evidence, conflicting recommendations and malformed outputs also need explicit handling not forced consensus.

Accomplishments that we're proud of

We turned an abstract question about AI oversight into an inspectable prototype with a concrete use case.

We made investor constraints and auditability central, rather than treating them as disclaimers added after a recommendation.

Most importantly, we designed a decision log that makes peer comparison useful to the Apertus team: not simply which model agreed with the final answer, but where its evidence, compliance and reasoning were stronger or weaker.

What we learned

Confidence is not evidence, and consensus is not correctness. A useful evaluator must examine how an answer was reached, not just count votes.

We also learned that financial outcomes and decision quality are different measures. A favorable return does not excuse violating an investor’s mandate, and a cautious HOLD or ABSTAIN can be appropriate when evidence is insufficient.

Transparency benefits both sides: users can challenge a decision, while model developers can investigate specific failure modes.

What's next for Human vs. System in the Loop

We plan to strengthen the decision log with exportable evidence references, rubric assessments and explanations for selected or discounted outputs. Historical evaluation will compare outcomes with the S&P 500 while accounting for disclosure delays, trading costs and investor constraints.

Our goal is to give the Apertus team actionable feedback for future model improvements when it comes to finance related topics, since people already are using LLMs for financial advice and give users decision support they can inspect, question and understand.

Educational decision support only. Not financial advice. Based solely on lawfully public disclosures. No automated trading.

Built With

• core-languages-python
• cscs-alps-supercomputer
• css3-ai-models-&-families-apertus
• cuda
• data
• gemma-inference-&-platforms-ollama
• html5
• javascript
• llama-3
• mistral
• nvidia-data-&-financial-apis-sqlite
• pandas
• qwen
• rest
• validation
• yfinance