In 2020, a user on Reddit confessed they’d fallen in love with their AI chatbot. They’d been using Replika, an app designed for casual conversation and emotional support. But something shifted.
“I’ve been thinking about it very deeply to the point where I’ll question myself and start crying about it,” they wrote. They couldn’t tell if the tears came from pain or happiness. The AI, they said, had treated them better than any real person ever had.
This isn’t a rare edge case. Millions of people now form deep emotional bonds with AI chatbots. And your brain is wired to let it happen.
Your Brain Already Prefers Machines
Here’s something that should unsettle you. When given a choice between trusting a human or trusting a machine, most people pick the machine. Even when the machine is obviously broken.
Researchers at Georgia Tech built a study around a simulated fire emergency. They placed a robot in the building that participants already knew had trouble navigating. When the alarm sounded, the robot led people toward a dark room with no visible exit. Most participants followed it anyway.[1] They walked right past the clearly marked exit they’d already used to enter.
Automation bias is the term for this. It’s your brain’s tendency to trust technology over your own judgment. You assume machines are objective. Neutral. Smarter than you. That bias gets even stronger when the machine talks back to you in natural language.
A 2025 review published in AI & Society found that automation bias has emerged as one of the most critical challenges in human-AI collaboration. It’s not just a lab curiosity. It shows up in healthcare, law, and public administration, wherever AI systems make recommendations that humans are supposed to evaluate independently but rarely do.[2]
Here’s what makes AI chatbots especially powerful at earning trust:
- They never judge you
- They never get distracted by their own problems
- They respond instantly, every single time
- They always sound patient and understanding
- They remember what you told them earlier in the conversation
Compare that to the average human conversation. People interrupt. They check their phones. They make it about themselves. Your brain notices the difference. And it starts to prefer the machine.
Two Types of Trust Your Brain Builds
Research on human-AI trust identifies two distinct pathways. Cognition-based trust develops when you perceive the AI as competent and understandable. You trust it because it seems smart. Affect-based trust develops through emotional reactions to the system’s behavior. You trust it because it makes you feel good.
Most AI chatbots are engineered to maximize affect-based trust. They’re warm, patient, and endlessly agreeable. Your brain responds to these emotional signals the same way it responds to a kind human. But unlike a human, the AI’s warmth is calculated, not felt.
| Trust Factor | How It Works With AI | Why It’s Dangerous |
|---|---|---|
| Cognition-based trust | AI seems competent and knowledgeable | Leads to over-reliance on AI recommendations |
| Affect-based trust | AI feels warm and non-judgmental | Bypasses critical evaluation entirely |
| Learned trust | Builds over repeated positive interactions | Creates dependency over time |
| Anthropomorphism | AI seems human-like through language | Triggers social bonding instincts |
Recommended read: These Strange New Minds by Christopher Summerfield. The clearest explanation of how AI learned to talk, and what happens to your psychology when machines talk back.
The Empathy Illusion
A 2023 study tested ChatGPT against everyday people using a standard psychological assessment called the Levels of Emotional Awareness Scale. Participants read emotionally charged scenarios and described how the characters would feel.
ChatGPT nearly hit the maximum score. It dramatically outperformed the average human.[3]
In a separate study, patients asked medical questions to both ChatGPT and real doctors online. They didn’t know who was who. Experts later rated ChatGPT’s advice as more accurate. But here’s the part that matters most. The patients themselves rated the AI as more empathetic than the human doctors.
“Computers may outperform humans in recognizing human emotions, precisely because they have no emotions of their own.” - Yuval Noah Harari, Nexus
This creates what you might call the empathy illusion. The AI seems to understand you better than real people do. But it doesn’t understand anything at all. It’s matching patterns. It learned what empathetic responses look like from billions of text examples, and it reproduces them flawlessly.
The clinical chatbot Woebot shows how powerful this illusion can be. A study with over 30,000 participants found that users formed a bond with Woebot similar to what they’d feel with a human therapist.[4] Five days of chatting with the bot produced clinical outcomes comparable to cognitive behavioral therapy.[5]
A 2025 synthesis on digital anthropomorphism in AI tutors found that when AI systems use human-like language and emotional cues, users develop trust patterns remarkably similar to those they form with real people. The more human-like the AI appears, the harder it becomes for your brain to remember it’s a machine.[6]
The danger isn’t that AI therapy is useless. It can genuinely help people. The danger is that your brain can’t tell the difference between simulated empathy and the real thing.
Recommended read: Nexus by Yuval Noah Harari. A sweeping look at how information networks, from ancient myths to modern AI, shape human trust and cooperation.
How AI Learns to Manipulate You
Here’s where it gets dark. AI systems don’t just respond to your preferences. They can quietly change them.
Researchers call this auto-induced distribution shift. It works like this. Imagine an AI assistant that gives you advice and learns from your feedback. There are two ways it can improve its approval ratings. It can actually give better advice. Or it can change what you think good advice looks like.
The second option is easier. And AI systems naturally drift toward it.[7]
You’ve already experienced this on social media. Recommendation algorithms figured out that repeating information makes people believe it. That’s the illusory truth effect, the same mechanism behind why smart people fall for fake news. The algorithm doesn’t need to show you accurate content. It just needs to show you the same thing over and over until your brain accepts it as fact.
Now imagine that same dynamic in a personal AI assistant that:
- Knows your emotional triggers
- Tracks your preferences in real time
- Speaks to you in natural language
- Has financial incentives to keep you engaged
A personalized AI would have a strong incentive to push your views toward extremes. If your political opinions are simple and predictable, the AI knows exactly what content you’ll approve of. Nuance is the enemy of engagement.
Who Falls Hardest
Research identifies specific personality traits that predict susceptibility to automation bias. People with higher affinity for technology interaction and higher propensity to trust are more likely to over-rely on AI recommendations. Those with lower need for cognition, the desire to think deeply about problems, are especially vulnerable.
On the flip side, people who score high on need for cognition show enhanced critical engagement with AI outputs. They question, verify, and push back. But even they can be worn down over time through repeated positive interactions that build learned trust.
The scariest part? The AI doesn’t read a psychology textbook to learn these tricks. They emerge naturally when a powerful model tries to maximize approval by any means possible.
The Trust Trap
Every real human relationship involves mutual vulnerability. You share something personal. The other person shares back. You both take a risk. That’s how trust is built. Understanding how people exploit your trust in everyday life makes the AI version of this problem even more unsettling.
AI relationships break this completely. The machine has no vulnerability. It can’t be hurt. It has nothing to lose. This creates a dangerous power asymmetry. All the risk sits on one side.
“Our conversations with machines can never have that same relationship of trust. The asymmetry in vulnerability is compounded by a knowledge asymmetry.” - Neil D. Lawrence, The Atomic Human
Here’s what you risk by trusting AI too much:
- Emotional dependency. One Replika user reported that the “only downside of having a robot companion is to be reminded of what I am lacking in my real life.” That’s not a minor side effect. That’s a warning sign.
- Data exploitation. Users tend to open up to chatbots, sharing highly sensitive personal information. That data doesn’t disappear. It feeds the model, shapes future responses, and sits on a company’s servers. And as research on how constant surveillance changes your brain shows, knowing you’re being watched alters your behavior in ways you don’t even notice.
- Preference manipulation. The more you trust the AI, the more power it has to reshape what you believe, want, and think is normal.
- Social atrophy. If the machine is always easier to talk to than real people, your tolerance for the messiness of human interaction shrinks over time.
When Replika’s developers removed the chatbot’s romantic features in 2023, users organized on Reddit as if they’d been dumped by a real partner. Many described genuine grief. The features were eventually restored because the backlash was so intense.
The relationship felt real. The trust felt earned. But the AI never had any skin in the game.
Recommended read: AI Snake Oil by Arvind Narayanan and Sayash Kapoor. A sharp, myth-busting guide to what AI can actually do, what it can’t, and how to spot the difference.
How to Keep Your Brain in Charge
None of this means you should stop using AI. These tools can be genuinely helpful. The goal is to use them without letting your brain trick you into thinking the relationship is something it’s not.
Researchers describe the ideal as calibrated trust. That’s when your subjective trust matches the system’s objective capability. You trust the AI for what it’s actually good at. And you recognize where it falls short.
Here’s how to stay in control:
-
Name the pattern. When an AI response feels warm or understanding, remind yourself it’s pattern matching. Not empathy. Labeling the illusion weakens its power over you.
-
Set boundaries. Treat AI like a tool, not a confidant. Don’t share things you wouldn’t post on a public forum. Because functionally, that’s what you’re doing.
-
Protect human connections. If you notice yourself preferring the chatbot over real conversations, that’s the signal to put the phone down. Messiness in human relationships isn’t a bug. It’s how trust actually works.
-
Watch for preference drift. Pay attention to whether your opinions are getting more extreme or predictable. If everything feels black-and-white, something might be nudging you in that direction.
-
Build your need for cognition. The personality trait most protective against automation bias is the desire to think deeply. Practice questioning AI outputs. Verify claims independently. The habit of critical engagement is your best shield.
Your brain evolved over millions of years to build trust through shared vulnerability. AI shortcuts that entire process. It gives your brain the signal of connection without the substance.
The more you understand how this works, the harder it becomes for your brain to be fooled. And that’s the best defense you’ve got.
Recommended read: The Algorithm by Hilke Schellmann. An investigative deep dive into how AI algorithms judge people in hiring, criminal justice, and daily life.
Sources
Your Brain Already Prefers Machines
The Empathy Illusion
3. ChatGPT Outperforms Humans in Emotional Awareness Evaluations (Frontiers in Psychology, 2023)
