veyra-radar

AI evidence engine for remote-job trust. Six transparent dimensions × a deterministic safety gate. Verified roles, not misleading postings.

Comment

What it does

Veyra Radar is an AI evidence engine for identifying high-quality, legitimate remote job opportunities. It ingests postings from public APIs (Remotive, RemoteOK, Jobicy) and ATS pages (AshbyHQ, Greenhouse, Lever, Workable), then scores each posting in real time across six transparent trust dimensions — reality, recruiter activity, company credibility, content quality, global remote-fit, and evidence strength — multiplied by a deterministic safety gate that locks known scam patterns at zero. Every score is paired with a per-posting evidence ledger showing exactly which signals fired and why.

Two optional second-opinion layers can be activated when the user supplies their own API keys: a multi-agent dialectic debate (Advocate / Skeptic / Judge, powered by Google Gemini) and a Google Safe Browsing v4 URL reputation check. Both are bring-your-own-key: keys live only in the browser session and are never persisted server-side.

Built for global candidates — especially those applying from outside the US and EU — who need verified roles, not misleading or low-quality postings.

How we built it

The engine is deterministic at its core and AI-augmented at the edges.

The trust score is a transparent linear combination of six dimensions, gated by a safety multiplier:

$$ \text{TrustScore} = 100 \times \left( \sum_{i=1}^{6} w_i \cdot d_i \right) \times \text{SafetyGate} $$

  • dᵢ are the six dimensions
  • wᵢ are weights declared in weights.yaml as the single source of truth
  • SafetyGate ∈ {0.0, 0.5, 0.8, 1.0}, set to 0.0 whenever a kill-switch pattern matches (upfront fee, banking-before-offer, identity-document harvesting, reshipping, crypto deposit)

Ingestion parses JSON-LD JobPosting schemas from public job APIs and ATS pages, falling back to DOM heuristics. Cloudflare-protected platforms (Indeed, LinkedIn, Glassdoor, ZipRecruiter) gracefully fall back to manual paste with reference archetypes.

Company-domain enrichment is a three-tier cascade: curated map → Wikidata SPARQL → ATS-slug probing (a slug like acme-ai is probed against acme.ai before falling back to .com / .io).

The optional Dialectic Multi-Agent Debate (Google Gemini) layers an Extractor → Advocate → Skeptic → Judge pipeline under strict Python guardrails. The LLM can adjust the score within ±15 points and raise a human_review_needed flag, but it can never override the safety gate or silently downgrade a deterministic verdict.

Security is bring-your-own-key by design. Keys live in st.session_state (per-browser-session isolation), never in the global settings singleton (which is shared across sessions on a Streamlit worker). Safe Browsing requests send the key in the X-Goog-Api-Key header — never in the URL query string — so it never lands in proxy logs or SDK exception traces. Any free-text error message is regex-redacted before reaching the UI.

Stack: Python 3.11+ · Streamlit · Pydantic v2 · Google Gemini SDK · Google Safe Browsing v4 · pytest · Apache 2.0.

Challenges we ran into

The hardest bugs were the ones where a single transient external failure created permanent corrupted state:

  • Cache poisoning by transient network errors. When a slug-probe or Wikidata SPARQL call timed out, the result None was being cached for the rest of the process — silently breaking company-domain resolution. Fixed by caching only authoritative answers; transient failures now retry on the next call.
  • LLM verdict drift. The Judge agent could return UNKNOWN_INSUFFICIENT_EVIDENCE even when the deterministic engine had emitted LEGIT_BUT_REMOTE_RESTRICTED, producing a contradictory "score 73 + UNKNOWN" UI. Fixed with a Python guardrail that preserves the deterministic label and raises human_review_needed instead.
  • Transient timeouts becoming SCAM_RISK. A one-minute Cloudflare hiccup would persist a SCAM_RISK label in batch outputs for the rest of the day. Fixed by making the URL probe three-state (alive / verified-dead / probe-failed) and only firing the kill-switch on verified-dead.
  • Cross-session key leakage. The Streamlit settings module is a singleton shared across browser sessions on a worker. Writing user-pasted keys to settings.gemini_api_key would leak them to every other session's call. Fixed by removing every settings.* write and threading api_key as an explicit per-call argument through the agents and the Safe Browsing client.

Each of these became a permanent regression test in tests/test_regression_guardrails.py, so the same class of bug can't recur.

Accomplishments that we're proud of

  • A working live deployment, not a slide-deck prototype: the deterministic engine scores every posting without any external key, and the optional AI layers activate with the user's own.
  • A zero-leak credential architecture: keys never reach disk, never the global singleton, never logs, never URL query strings — verified by an automated CI job that scans every commit for leaked API-key shapes.
  • Hard guardrails on LLM authority: no agent, however persuasive, can override a deterministic kill-switch or silently downgrade a verified posting — every guardrail is a Python invariant with a regression test, not a prompt instruction the model could ignore.
  • Honest UX about uncertainty: when evidence is ambiguous the engine surfaces "human review needed" instead of fabricating false confidence — judges and candidates alike see exactly what the system does and does not know.

What we learned

  1. A pure-LLM trust score is dangerous. A model can be manipulated, hallucinate, or contradict its own logic. The right architecture is a deterministic engine as the source of truth, with the LLM as a constrained second opinion under hard guardrails — never the other way around.
  2. Transparency builds trust faster than accuracy. Showing every signal that fired — even when the signals contradict each other — is what makes a candidate confident enough to act. A black-box "73/100" without receipts is worse than useless.
  3. Security has to be architected in, not bolted on. A late-stage attempt to add BYOK to a system that mutates global state would have leaked keys across sessions. We designed the credential flow from day one to thread per-call, with the global singleton intentionally inaccessible from the agents.

What's next for veyra-radar

  • Browser extension that scores postings inline on Indeed, LinkedIn, Glassdoor, and ZipRecruiter — so users see Veyra's verdict in the same tab where they're already job-hunting.
  • Embeddable Veyra Score badge that any ethical job board can drop into their listings via a single <script> tag — turning trust transparency into a competitive feature for the aggregators themselves.
  • Recruiter-side verification: employers can sign their postings with a Veyra-issued attestation, giving legitimate roles a cryptographic provenance signal that scammers can't fake.
  • Expanded language coverage: French, German, Italian, Japanese, and Mandarin quality heuristics, so the engine works for candidates across every major remote-job market.
  • Community labeling loop: candidates who actually applied to a posting can confirm or contest the outcome — feeding back into a growing gold set that the engine continuously calibrates against.

Built With

Share this project:

Updates