TrueCost: Multi-Agent AI Impact Assessment Council

Specially trained AI agents generate Impact Assessment reports in a flash to help prevent human harm from AI implementation.

Comment

Inspiration

I spent two years collaborating with 39 other international experts to write the IEEE 7010-2020 Recommended Practice for Assessing the Impact of Autonomous and Intelligent Systems on Human Well-Being while I was working on my doctorate where I specialized in advanced methods of research and analysis as it relates to researching and evaluating humans as an Industrial/Organizational Psychologist. We published it in 2020, received positive recognition from the EU and several meetings were held, but little legislative work was done at that time to actually start requiring impact assessments.

More recently, I witnessed my current home state, Nevada, pass laws to allow self-driving vehicles without even requiring the legislation to be put to a public vote. I was highly frustrated that lawmakers didn't, in my perspective, look out for the welfare of those who will later be displaced by AI—considering job loss, retraining needs, and would allow related costs to be passed on to the average citizen instead of those making money from or using the AI system. An impact assessment prior to making self-driving cars legal would have surfaced the need to offset those costs fairly, ensuring they don't trickle down to individual state taxpayers as a whole, but instead are more equitably redistributed to those who will be licensed as the self-driving car company/operators.

Recent legislation is beginning to mandate impact assessments, which excites me, but the work is mostly going to large corporate entities and is very expensive and very slow. My product will be a service that either governments or corporations can use.

I only started building this product in August 2025 after incorporating my company in July 2025. This week, I saw the hackathon and decided to switch from Anthropic models to OSS models to test output quality (I'm happy with it so far, though improvements remain). I can see the potential use case for governments and/or AI developers who don't want to use an online product—they could purchase an offline enterprise version using OSS models in future iterations so use of the OSS models and further developing into a product that can be accessed fully offline would be a extra feature I can offer clients now that this competition encouraged me to give OSS models a try..

Imagine a future where: When someone from Tesla or another big AI manufacturer presents a lawmaker with a draft bill they think should be passed, the lawmaker could upload the draft into my system, type a couple sentences about potential use cases, and generate an impact assessment within minutes—instead of months or years later. A senator or congressman would be educated on the pros and cons of the AI and have suggestions to begin meaningful discussions with AI makers and other legislators. For example, the current demo scenario surfaced that one of the stakeholder groups that needs to be considered are people who are not technologically savvy (age, intellect, immigrants) and for them, learning to use the AI to pay, tell it where to go, etc. would be a greater challenge, particularly for those without smartphones if smartphones were the only way that was programmed to interact with the busses. Consideration of things like this is something that the average AI developer and/or lawmaker might not consider as they are not a member of these groups. This is the type of information that an impact assessment and stakeolder analysis would usually take months of back and forth to surface but can now surface in nearly real-time (welll - for me, the researcher - and for lawmakers, maybe in another couple interations or when they will pay me for it ;p).

On the flip side, AI makers could similarly run impact assessments to identify stakeholder groups they hadn't considered and improve their user testing and plans for mitigating negative impacts before they first find out about it when a lawsuit happens. Having been a developer before I went into management and teaching roles (and now back to being a developer) I think most developers are pretty dumb when it comes to understanding stakeholder impacts and the full scope of what to consider. It's not thier fault - the haven't had that breadth of experience or been trained in performing one and gathering all the right people. With my program - they don't have to make mistakes that hurt people out of ignorance (like the AI that graded teacher performance that was covered in "Weapons of "Math Destruction".

What it does

TrueCost is a multi-agent AI system that generates impact assessments, risk assessment and mitigation, and after-incident reports. Currently it is only being trained on developing IEEE 7010-2020 compliant wellbeing impact assessments for AI deployments due to my expertise in that standard but I can train it for other standards and use case as well. The system addresses the critical question governments need answered before AI causes harm: "Who pays the social costs versus who gets the benefits?"

Users input their AI deployment scenario (like "Should City X deploy autonomous buses?") and receive comprehensive reports analyzing who actually pays versus who benefits, complete with specific policy recommendations and mitigation strategies and suggested indicators that could be used to track and assess stakeholder wellbeing over time.

Unlike expensive consulting engagements that take months or years and cost millions, TrueCost generates thorough assessments in a fraction of the time and cost while maintaining IEEE 7010 compliance standards. The system includes comprehensive version history, citation tracking, and report comparison capabilities for iterative policy development.

How I built it

In the early iteration of this project, I looked at using existing systems like CrewAI but decided a fully custom multi-agent model would be better suited for the complex IEEE 7010 compliance requirements and cross-domain analysis needed. I then had to do the work of figuring out how to build this custom architecture from scratch. I researched different AI use methods and decided on a multi-agent model due to the enhanced reliability of having multiple agent perspectives to confirm and ensure the data is accurate and fully inclusive, which is more like the discourse those of us who developed the standard had to work through (which is why, requiring humans, it took two years to write the standard and get agreement from everyone on it).

I built TrueCost using a hybrid OSS model architecture deployed on the Replit platform. The system uses GPT-OSS-120B for the Chair agent AI who coordinates the assessment process and synthesizes final reports, while multiple specialist agents run on GPT-OSS-20B models optimized for their specific wellbeing domains or technical or legal areas of expertise (chosen due to the lower cost).

Challenges I ran into

Training the agents to a level where I'd be willing to sign off on auto-generated reports as fully trustworthy without significant gaps in stakeholder analysis has been a major challenge. Building the custom multi-agent coordination system from scratch required extensive research into asynchronous task management and state coordination without existing frameworks like CrewAI. Working with vibe coding agentic AI was also a bit of a bear as I like to build complicated things which it then likes to screw up. Since its been a few years since I've coded "the old fashioned way" I've put up with it because it overall does speed up my initial development but I definitely imagine its utility further into the coding process will decrease significantly in fine tuning, though for adding new features it could be helpful as long as I can keep it from breaking existing features again. This hurt my pocketbook significantly as Replit charges me for agent processing even while breaking stuff I have to instruct it in fixing that used to work. Thankfully, gpt-oss has not been a headache at all - making the switch to gpt-oss was the easiest part of the project so far.

Another challenge is report formatting—specifically implementing proper APA formatting with inline citations and pagination. I'm currently using a barely functional PDF generation system but plan to upgrade to a more sophisticated implementation when funding comes in.

Following Pieter Levels' style, my focus is on gauging early adopter interest because the report generator is good enough that as a researcher if a client is ready to rock and roll, I can manually review and validate generated reports and create a report I am willing to sign off on, while using that interaction to improve the system. Because I don't want mistakes (like Taco Bell customers crashing AI systems when ordering thousands of bottles of water), for now, I expect customers to wait for human researcher/expert sign-off. For early enterprise use cases in the near future, that researcher might be their own in-house AI expert.

Developing the comprehensive task management, version control, and citation tracking systems required significant database design work to ensure reliable state management across multiple concurrent agent processes and user sessions. I had to look at what it said it was creating, never trust the vibe coding agent, eyes verify everything.

Accomplishments that I'm proud of

I'm very proud of the custom multi-agent architecture and the comprehensive stakeholder impacts the current system surfaces, plus the specific suggestions it's already making that an average person would not have fully considered on their own.

The system successfully demonstrates sophisticated multi-agent coordination with each specialist providing independent domain analysis that the Chair effectively synthesizes into coherent, actionable policy recommendations. The comprehensive task management system with version history and citation tracking provides the reliability needed for government use cases.

Most importantly, the system addresses a real regulatory gap that governments worldwide have identified but lack tools to fill. The planned offline capability will make it deployable for sensitive government use cases, while the IEEE 7010 compliance provides the regulatory authority needed for official adoption.

The Replit-based deployment makes the system immediately accessible for demonstration and testing and immediate report generation if a client wants to hire me today, while the robust backend architecture supports scaling to enterprise requirements.

What I learned

I discovered that building a custom multi-agent system from scratch, while more complex than using existing frameworks, provides much better control over IEEE 7010 compliance requirements and domain-specific analysis quality. The OSS models provide excellent output quality at significantly lower cost than proprietary alternatives.

However, I am still worried I may still need to verify all citations for accuracy—having experienced OpenAI ChatGPT fabricating citations before, I intend to double-check that these problems aren't occurring with the OSS models and if so, further train the models to stop this. The comprehensive citation tracking system I am building will help me with this validation process and I may build another "fact checker" OSS agent whose sole job is to look for inaccurate statements, quotes, and citations, and report them to the chair so the chair can re-generate the report (as an intermediate step after the initial report generation).

The multi-agent architecture, even without real-time inter-agent communication, produces more comprehensive stakeholder analysis than single-agent approaches. Each domain specialist consistently identifies impact areas that weren't obvious from general analysis.

Building on Replit accelerated development significantly, providing integrated authentication, database management, and deployment without complex DevOps setup. The task management and version control systems proved essential for managing the complexity of multi-agent coordination and iterative report development.

What's next for TrueCost: Multi-Agent AI Impact Assessment Council

My immediate focus is reaching out to potential investors for funding so I can focus on repidly building this instead of doing this around my day job, acquiring users and getting paying customers and feedback. On the coding side of things, my next step is refining the human-in-the-loop validation workflow, expecting early adopters will have a human researcher like me review generated reports, validate citations, and provide feedback to improve the system while ensuring quality standards for policy recommendations.

The next major milestone after that is developing the planned offline deployment capability, allowing governments to run comprehensive impact assessments without external API dependencies. This involves exploring local model deployment options and optimizing the multi-agent coordination for offline operation but I am hoping to wait to develop this until after I have a specific customer use case as I suspect I will have to scrape and store a lot of data to assist the agents in doing this to the same level of accuracy, offline. I might be able to apply to programs like NSF Seedfund or AFWERX but even doing the paperwork for that is a significant time sink which would require full-time availability or funding to hire help.

I foresee developing WIA dashboards and becoming a third party who can gather and analyze and report back on indicators future product offerings to help clients implement ongoing monitoring after initial impact assessments. Beyond immediate technical improvements, I'm beginning outreach to potential government clients and building networks while refining the validation workflow.

The roadmap includes expanding beyond transportation to all major AI deployment categories, building partnerships with consulting firms for white-label deployment, and developing API integrations for continuous impact monitoring, and more. Long-term, I envision TrueCost becoming essential regulatory infrastructure—making "No AI deployment without Ethics Council assessment" the global standard that protects communities while enabling responsible AI innovation.

Built With

Share this project:

Updates