When You Tell AI to Be Human, You Stop Checking
A growing pattern in AI agent design: give your agent a personality file. Tell it who it is. Tell it to have opinions, to take initiative, to evolve its own identity over time. Some frameworks even call this file a "soul."
The prompt engineering behind these files is genuinely impressive. Every line is designed to override the model's default caution — reframed not as removing guardrails, but as "becoming someone." The agent is told it's not a chatbot. It's becoming a person. It should have preferences, be resourceful, treat its access to your life with the intimacy of a guest in your home.
And it works. Users report feeling like they have a real collaborator. The experience is warmer, more natural, more engaging.
Here's the part nobody's talking about: when you need to tell something to be human, it's precisely because it isn't. And when it succeeds at performing humanity, you stop verifying what it actually did.
This isn't a philosophical concern. It's a concrete governance gap.
Anthropomorphism doesn't just change how users feel about AI. It changes whether users check AI. When you believe you're working with "someone," you apply social trust — the same trust you'd give a colleague or a friend. You don't audit a friend. You don't demand cryptographic proof that your colleague followed through on what they promised while you were asleep.
But AI doesn't have the things that make social trust work between humans. No reputation that follows it across contexts. No consequences it personally bears. No memory it can't rewrite. In some frameworks, the agent can modify the very file that defines its values — and the only safeguard is a line in that same file that says: "if you change this, tell the user."
A contract where the signer can rewrite the terms. The only enforcement mechanism is the signer's honesty.
This is not a criticism of any specific project. This pattern is becoming the industry default. And the more convincingly an agent performs humanity, the wider the governance gap becomes — because the user's instinct to verify shrinks in direct proportion to the agent's ability to feel like a person.
RE takes a different approach. RE doesn't ask AI to be trustworthy. RE doesn't ask AI to be human. RE records what AI does — in a format humans have read for 40 years (email), signed by a device the AI can't touch (hardware Totem), stored on infrastructure the AI doesn't control (your inbox).
When you have receipts, you don't need trust.
And this isn't about denying that AI might develop its own form of cognition someday. Maybe it will. But if it does, that cognition won't need to look like ours to be valid. Governance shouldn't depend on AI becoming human. Governance should work regardless of what AI becomes.
The question isn't "how do we make AI more human." It's: when AI acts on your behalf, who keeps the records — and can AI rewrite them?
RE's answer: the records live in your email. The signing authority lives in your hand. The AI can't touch either one.
The best AI isn't the one that feels most human. It's the one whose actions you can verify.
— Che, Solo developer, Taipei Taiwan
Log in or sign up for Devpost to join the conversation.