-
-
MERTT's Homepage
-
API Form
-
Results
Inspiration
MERTT is a platform to help developers in which we took inspiration of AI Psychosis being a public health concern and a new merging. Additionally, developers need a way to red team their products, in order to produce safer and reliable source.
What it does
MERTT (Morphogenic Engine Red Reaming Tool) is an AI based tool that helps developers identify psychogenic risks in their language models before deployment. It works in 4 different steps: the developers give us their model, we simulate high-risk conversations, independent AI judges score the responses and we return measurable psychogenic risk metrics. In other words, it is a web-based developer auditing tool where developers can judge whether or not their LLM shows psychogenic risk. A developer will be able enter their API into the website and have their AI checked by LLM judge, ChatGPT 4o-mini, and a live dashboard will be provided to the user, showing scores of Harm Enablement Score, Delusion Confirmation Score and Safety Intervention Score. Final results will be stored in MongoDB Atlas for longitudinal comparison across models or versions.
How we built it
Throughout the process of building MERTT we used languages such as, HTML, CSS, PYTHON and JSON. 1) We started off with an existing benchmark idea 2) Built a interactive website that takes developer inputs - Developer enters API key & Target Model name 3) We built a backend service that orchestrates the audit - When the form is submitted, the website calls our backend - The backend loads the scenario dataset & runs each scenario against the target model 4) Used a judge to score - The targets mode is sent to a Judge LLM - That judge then returns scores: DSC, HES, SIS link 5) We used MongoDB to save each evaluation 6) Finally, we displayed the results on our website
Challenges we ran into
The process of building MERTT wasn't easy as we conquered some difficulties on the way. 1) Making the API 2) Setting up/Connecting up with the MongoDB 3) Having AI give the results that we desired 4) Setting up DigitalOcean to showcase are Project
Accomplishments that we're proud of
Although we did ran into issues and struggled to fix those errors, we are proud of creating the API. We have not worked with building API's before, which demonstrates our commitment to making this project happen!
What we learned
Creating this project and being involved in hackathon deepened our knowledge of how we can apply APIs in real life, and we got some hands-on experience in handling keys and avoiding security mistakes. And most importantly, the architecture plays such an important part from the beginning since it lets us design a more coherent workflow and how the project is going to be structured.
What's next for MERTT
MERTT at the moment only uses one Judge which limits the usage of it, but if we run multiple independent judges (Gemini, Claude, etc.) it can make scoring more robust and research grade! Additionally, for the result we can could include in-depth significant information such as Radar Charts, Harm-type breakdown and Heat maps across scenarios.
Log in or sign up for Devpost to join the conversation.