Inspiration
Legacy analytics migration sounds simple until the translated code runs successfully and still produces the wrong result.
Many research groups, healthcare organizations, government teams, and enterprises still rely on analytical workflows written in SAS. Modern language models can translate syntax surprisingly well, but real migration failures are often semantic rather than syntactic: a different reference level, reversed outcome coding, missing-value handling, or model default can silently change the scientific result.
That led to the core idea behind CodeShift:
Generation is probabilistic. Acceptance is deterministic.
Instead of asking whether an AI can generate valid R code, we wanted to answer the harder question:
Did the migrated analysis preserve the result?
CodeShift was built to turn migration from a code-generation task into an autonomous, verifiable workflow.
What it does
CodeShift is an autonomous legacy analytics migration agent that converts SAS workflows into verified R projects.
A user provides a legacy SAS analysis, data, and a reference contract. From that point forward, CodeShift can handle the workflow autonomously:
Inspect the legacy project and identify its analytical structure. Generate or evaluate an R migration. Execute the R code in a controlled environment. Deterministically compare the produced results against trusted reference outputs. Detect semantic discrepancies even when the code runs successfully. Use Gemini to diagnose the likely cause of the mismatch. Repair the R implementation automatically. Re-execute and re-validate the repaired analysis. Produce a verified R project and an auditable evidence package.
The most important distinction is that Gemini never decides whether a migration is correct.
The language model can translate, reason, diagnose, and repair, but only the deterministic validation layer can mark a migration as verified.
This allows CodeShift to catch one of the most dangerous migration failures: code that executes without errors but produces scientifically different results.
The demo includes a realistic semantic mismatch where the initial R translation runs successfully but uses incorrect modeling semantics. CodeShift detects the discrepancy, invokes Gemini through Google ADK, repairs the code, runs it again, and validates the corrected output without human intervention.
How we built it
CodeShift is designed as an event-driven Taskmaster workflow rather than a traditional chat interface.
The system is built primarily with Python and FastAPI, with Google ADK coordinating Gemini-powered translation and repair agents.
The architecture separates probabilistic AI decisions from deterministic execution and validation.
The main workflow is:
Project received → inspect → translate or evaluate → execute → validate → diagnose → repair → re-execute → verify → package evidence
We use Gemini 3.5 Flash through Google's AI tooling for two core reasoning tasks:
translating SAS analytical logic into R; diagnosing failed validation attempts and producing repaired R code.
Google ADK provides the agent layer around these tasks.
Execution and acceptance are intentionally kept outside the LLM. Generated R code is passed through a safety gate, executed with Rscript --vanilla in an isolated job workspace, and evaluated by a deterministic validator.
The validator compares metrics such as coefficients, odds ratios, row counts, and other expected outputs using explicit absolute and relative tolerances.
For production architecture, CodeShift uses:
Google Cloud Run for the web service and asynchronous worker; Pub/Sub to decouple user requests from long-running migration jobs; Firestore for persistent job state, attempts, and audit events; Cloud Storage for uploaded projects, outputs, and evidence artifacts; Secret Manager for sensitive configuration such as the Gemini API key; Google ADK + Gemini 3.5 Flash for translation and autonomous repair.
Each migration is represented as an explicit state machine, including states such as inspection, execution, validation, diagnosis, repair, verification, and completion.
The repair loop is bounded rather than open-ended. Every attempt records execution results, validation evidence, timestamps, and code changes.
We also built infrastructure abstractions so the same workflow can run locally using local storage and an inline queue, while the cloud deployment uses Firestore, Cloud Storage, and Pub/Sub without duplicating the core migration logic.
Challenges we ran into
The biggest challenge was realizing that code translation accuracy is not the same thing as migration accuracy.
A naïve implementation could simply ask Gemini to convert SAS into R and report success when the script executes. That would create an impressive-looking demo, but it would not solve the real problem.
We therefore had to design a system where the AI is not allowed to grade its own work.
Building deterministic validation introduced several challenges, including defining numerical tolerances, handling missing or incomplete references, distinguishing VERIFIED from merely EXECUTED, and providing useful failure evidence back to the repair agent.
Another challenge was autonomous repair. When validation fails, Gemini needs enough evidence to understand the semantic problem without simply being shown an answer and reproducing it. We separated the initial translation process from validation evidence and only provide discrepancy information to the repair stage.
Safe execution was another important engineering challenge. CodeShift generates executable R, so we needed explicit safety checks, restricted workspaces, subprocess timeouts, file-access boundaries, and protections against unsafe generated commands.
Finally, moving from a local synchronous prototype to an asynchronous cloud architecture introduced the usual distributed-system concerns: persistent state, Pub/Sub redelivery, idempotency, service-account permissions, secrets, failure recovery, and keeping the user interface synchronized with work happening in the background.
Accomplishments that we're proud of
We are most proud that CodeShift does more than generate code.
It creates a complete autonomous feedback loop between AI reasoning and deterministic evidence.
A migration can execute successfully, fail validation, be diagnosed by Gemini, be repaired automatically, execute again, and ultimately pass deterministic validation without requiring a person to intervene.
We are also proud of the architectural separation between:
probabilistic generation and deterministic acceptance.
This design makes CodeShift much more than an LLM wrapper. The language model is one component inside a larger workflow that includes persistent state, controlled execution, validation, recovery, auditability, and artifact delivery.
Another accomplishment is the evidence trail generated for every migration. CodeShift records each attempt, validation result, repair summary, code hash, and workflow transition, then packages those artifacts into a downloadable evidence bundle.
We also designed the same workflow to operate both locally and on Google Cloud, allowing the project to remain reproducible for developers while still demonstrating a production-minded asynchronous architecture.
Most importantly, the project demonstrates a concrete form of agent autonomy:
after the migration job starts, the system can detect a problem, decide what action is needed, modify the artifact, run it again, and prove whether the action succeeded.
What we learned
The biggest lesson was that reliable agents need a source of truth outside the language model.
LLMs are extremely useful for ambiguous tasks such as understanding legacy code, identifying likely semantic mismatches, and proposing repairs. But whenever the outcome can be tested deterministically, the agent should not be responsible for judging its own success.
We also learned that autonomous systems become much more useful when failures are treated as part of the workflow rather than exceptions.
A validation failure is not simply an error in CodeShift. It becomes structured evidence for the next action.
That changes the architecture from:
generate → hope it works
to:
generate → observe → measure → repair → prove
We also learned that a smaller number of well-defined agents can be more effective than a large multi-agent system. Translation and repair require model reasoning, while inspection, execution, validation, state management, and acceptance are better handled deterministically.
Finally, building around explicit state, bounded retries, audit logs, idempotency, and isolated execution reinforced how different a production-oriented agent is from a conversational AI demo.
What's next for CodeShift
The current version focuses on SAS-to-R analytical migration, but the larger opportunity is broader.
The next step is to expand CodeShift from individual analytical scripts into complete legacy analytics projects containing multiple programs, datasets, formats, dependencies, and expected outputs.
We also plan to support richer validation contracts, including table-level comparisons, prediction equivalence, survival-analysis outputs, model diagnostics, and dataset-level reconciliation.
Another direction is organizational deployment. Enterprises could define reusable migration policies describing approved R packages, validation tolerances, security restrictions, coding standards, and required evidence. CodeShift could then apply those policies consistently across large migration portfolios.
Longer term, the same architecture could extend beyond SAS-to-R.
The core idea is language-independent:
AI proposes a transformation, the target system executes it, deterministic evidence evaluates it, and the agent repairs the transformation until the acceptance criteria are satisfied.
That pattern could support migrations between statistical languages, data pipelines, legacy SQL systems, scientific computing environments, and other forms of technical modernization.
Our goal is for CodeShift to make AI-generated migrations not just faster, but trustworthy enough to use in real analytical workflows.
Log in or sign up for Devpost to join the conversation.