Career Pilot

Inspiration

International graduates in Germany face a brutal job hunt that German citizens never see: every posting is a hidden gauntlet of visa sponsorship, Blue Card salary floors, German-language expectations, and relocation constraints. Generic job boards rank by keyword match and quietly surface roles that will reject you at the visa stage. We wanted an agent that thinks the way a good immigration-aware career coach does — one that reasons about whether a job is actually attainable for this candidate, explains why, and gets better the more it's used.

What it does

CareerPilot is an autonomous job-matching agent. You upload a CV (or paste text), it extracts a structured candidate profile, then it doesn't just rank jobs in one shot — it runs an agent loop:

• Plans a ranking strategy specific to the candidate (e.g. "junior Python eng, Berlin, needs visa → weight sponsorship and seniority fit heavily").
• Ranks jobs against a rubric, with autonomous retry if its own output fails schema validation.
• Self-evaluates each ranking with an LLM judge scoring fit, risk detection, and actionability.
• Self-improves: if the evaluation falls below threshold, it rewrites its own ranking rubric and re-ranks — looping until it's good enough or it hits its iteration budget, then stops on its own.
• Learns across runs: it persists the winning rubric and run history to Arize Phoenix, so the next run starts smarter than the last.

Every step streams live into an "Agent Console" so you watch it plan, call tools, score itself, and improve — not a black box.

How we built it

The agent spine lives in orchestrator.py and runs on Gemini 2.5 Flash via Google Vertex AI (Google Gen AI SDK, vertexai=True) — planning, ranking, evaluation, and rubric self-improvement are all Gemini calls. Structured outputs are Pydantic-validated with a self-correcting retry wrapper that re-prompts the model with its own error on a parse failure.

Our partner superpower is the Arize Phoenix MCP server. At runtime the app spawns @arizeai/phoenix-mcp over stdio using the Python MCP SDK and calls its tools directly: get-latest-prompt and get-spans to recall the last learned rubric and recent run history, and upsert-prompt to persist the improved rubric as a versioned Phoenix prompt. That prompt store is the agent's long-term memory. We also added a Gemini function-calling turn so the model itself decides whether to query Phoenix memory or inspect the job pool before planning. On top of that, Phoenix OpenTelemetry tracing emits a span for every agent step, so the whole reasoning trace is observable in the Phoenix dashboard.

The frontend is Streamlit with custom CSS and a live-streaming agent console. We deployed to Google Cloud Run with a single container that bundles both Python and Node, so the Phoenix MCP server runs as a real subprocess on the hosted URL — secrets via Secret Manager, build via Cloud Build.

Challenges we ran into

• Making it genuinely agentic, not a pipeline. The MVP was a straight line: extract → rank → done. We had to add real planning, model-selected tools, an evaluator-triggered improvement loop, and autonomous stopping criteria.
• Phoenix MCP for real, at runtime. Wiring an async MCP stdio session into a synchronous Streamlit app meant bridging event loops on a background thread. Phoenix also silently strips hyphens from prompt names, which cost us an hour until we switched our memory key to underscores.
• Cross-run memory without a "write annotation" tool. Phoenix MCP has no write-span-annotation tool, so we built persistent memory on versioned prompts instead — encoding the learned rubric as the prompt the next run reads back.
• Cloud Run with a Node subprocess. Getting npx-spawned MCP to run inside a Python container, plus untangling a chain of IAM permissions for Cloud Build and Vertex AI, was the biggest time sink under deadline pressure.

Accomplishments that we're proud of

• A real self-improvement loop we can demonstrate: a deliberately weak first ranking scores 3/5, the agent rewrites its rubric, re-ranks, and reaches 5/5 — on camera, autonomously.
• Memory that actually persists across runs through the Phoenix prompt store, not a mocked stub. The old get_previous_feedback_for_session returned []; now it returns real Phoenix history.
• A fully observable agent — every plan, tool call, and self-evaluation is a Phoenix span — deployed live on Cloud Run with the MCP server running in-container.

What we learned

The line between "LLM pipeline" and "agent" is planning, tool choice, and the ability to judge and correct its own work — and that's mostly engineering, not prompting. We learned how to integrate an MCP server as a first-class runtime dependency rather than a config-file afterthought, and that good observability (Phoenix tracing) is what makes a self-improving loop trustworthy instead of mysterious — you can see exactly why the agent changed its mind.

What's next for Career Pilot

• Richer memory: cluster learnings by candidate archetype so the agent recalls the most relevant past runs, not just the latest rubric.
• Live sourcing: pull fresh postings via search APIs and let the agent decide what to fetch.
• Feedback as a first-class signal: feed user thumbs-up/down back through Phoenix so the rubric improves from real outcomes, not just self-evaluation.
• Beyond Germany: generalize the visa/work-authorization reasoning to other countries' immigration rules.

Built With

• arize-phoenix
• cloud-build
• cloud-run
• docker
• gemini-2.5-flash
• google-cloud
• google-gemini
• mcp
• model-context-protocol
• node.js
• opentelemetry
• pydantic
• python
• secret-manager
• streamlit
• vertex-ai