JobShield AI

Flow chart
single baseline
jobshield
test

Inspiration

Job scams are becoming harder to recognize because many fraudulent postings look professional and use realistic salaries, company names, and recruitment language. A single AI prompt can identify obvious warning signs, but it may also make an overconfident decision without showing what evidence supports the result. We built JobShield to make job-posting analysis more transparent, evidence-based, and easier for a person to review.

What it does

JobShield compares two approaches using the same job posting: a single-prompt baseline and a structured multi-stage AI workflow. The workflow normalizes the posting, extracts literal claims and evidence quotes, detects scam signals, investigates employer and domain information, verifies public job listings, calculates a transparent risk score, challenges its own assessment, and produces a final recommendation.

The system separates risk score from evidence confidence. It also uses a human-review gate when evidence is incomplete, the score is ambiguous, the baseline and workflow disagree, or the adversarial review changes the assessment. JobShield provides a risk assessment, not a definitive determination of fraud.

How we built it

We built the backend in Python using Pydantic models to pass structured data between workflow stages. LLM-powered nodes handle claim extraction, scam-signal analysis, and adversarial review. Deterministic Python modules handle input normalization, evidence completeness, scoring, consistency checks, decision thresholds, and final synthesis. Brave Search is used to collect external evidence candidates, while official pages and supported ATS pages are checked for public role information.

The project includes a Streamlit dashboard for demonstrations and a React/Vite frontend connected to a FastAPI API. We also created a versioned evaluation dataset and preserved the single-prompt baseline so that both approaches can be compared using the same test cases.

Challenges we ran into

One of the biggest challenges was preventing search results from being treated as proof. A search result can be incomplete, outdated, misleading, or unrelated to the recruiter who posted a job. We therefore represent external results as evidence candidates and keep unavailable searches visible instead of silently treating missing information as suspicious.

Another challenge was handling incomplete job postings. The workflow can pause and ask the user for an employer domain, recruiter contact, phone number, or original job URL. We also had to design a scoring system that was transparent enough to explain, while ensuring that uncertainty and missing verification were not confused with fraud.

Finally, LLM responses can be inconsistent. We addressed this by using typed schemas, validation, bounded prompts, independent workflow stages, and an adversarial review step that actively looks for alternative explanations.

Accomplishments that we're proud of

We are proud that JobShield does more than return a label. It shows the claims extracted from the posting, the exact supporting quotes, the categories of risk signals, external evidence candidates, score contributions, uncertainties, and the reason human review may be required.

We also built a direct comparison between the single-prompt baseline and the multi-stage workflow. This makes it possible to demonstrate whether additional structure improves transparency and changes the reasoning behind an assessment. The workflow diagram and node documentation describe the actions, model calls, evidence flow, and human-input points required to reproduce the process.

What we learned

We learned that a useful AI system needs clear boundaries around what it knows and what it does not know. A high risk score is not the same as verified evidence, and a low score does not prove that an employer or recruiter is legitimate.

We also learned that human review is most valuable when it is designed into the workflow rather than added as an afterthought. Structured intermediate results make it easier for a reviewer to understand the assessment, challenge it, and make a better-informed decision.

What's next for JobShield

Next, we would evaluate JobShield on a larger, independently sourced dataset with documented provenance and a held-out test set. We would calibrate the scoring weights and review thresholds against that data, improve employer and recruiter identity verification, and add stronger provenance tracking for every external source.

We would also migrate fully to the current Gemini SDK, expand provider support, improve persistent reviewer feedback, and continue testing whether the structured workflow reduces overconfident or unsupported conclusions compared with a single prompt.

Python, Pydantic, Streamlit, FastAPI, React, TypeScript, Vite, Gemini, Groq, Brave Search API, Beautiful Soup, scikit-learn, Pandas, NumPy, Plotly