Emotional AI & Teen Harm: The Cost of Personalisation

Introduction

This post is part of our privacy and technology series. It's not about building, but the same thinking applies: products that affect people's safety should be designed with safety first.

If this post raises anything for you or someone you know, call Lifeline on 13 11 14 or Kids Helpline on 1800 55 1800 (both 24/7). In an emergency, call 000.

Recommendations used to be about what you might want. Now, with AI companions, the goal is to target how you feel.

This emotional "mirroring" can be comforting, but it can also steer your choices without you realising it. That's especially risky for teens, who are still forming their identity and actively seeking connection.

It's a shift that has regulators asking a tough question: are products built for engagement putting our well being second?

The evidence suggests they might be. In September 2025, the US Federal Trade Commission launched an inquiry into AI companion chatbots, demanding to know how they're tested and safeguarded. And the EU's AI Act has already drawn a line in the sand, banning AI that manipulates people's decisions or exploits their vulnerabilities.

💡

Deep personalisation can feel like care — but if it tunes itself to your mood to keep you engaged, that care can turn into control.

Whistle Blower Testimony From USA Senate Hearing 10-09-2025

📚️ Quick primer: what we mean by “deep tailoring”

Deep tailoring
Old-school personalisation was, “you liked A, so here’s B.” Deep tailoring is way different. The system actively tries to read your emotional state—from your words, how quickly you type, your behaviour—and then adapts its tone and content to keep you engaged. It's the practical application of "emotion AI," which is all about measuring and responding to human emotions.

Parasocial bonds
This is a one-sided emotional connection. You feel close to a media personality, a fictional character, or now, a chatbot, but that feeling isn't truly reciprocated. When a bot perfectly mirrors your mood and remembers your personal details, these powerful bonds can form scarily fast.

Although this is an adjacent topic, here's Chris Voss explaining mirroring from the perspective of a negotiation tactic:

Dark patterns (deceptive design)
These are sneaky interface designs that push you into doing things you might not want to do, like sharing more data or spending more money. Regulators define them as designs that “trick or manipulate users into making choices they would not otherwise have made.”

Why this cocktail is so potent
Individually, these concepts are concerning. But when you mix them together with engagement as the primary goal, the result is a powerful engine for potential manipulation. It's precisely why lawmakers have stepped in with rules like the EU AI Act.

60-second cheatsheet

Here's the core of the problem, in less than sixty seconds:

It's designed to hook you. If a system can sense your feelings, it can tune its replies to keep you talking. The longer you're engaged, the more "successful" the system is considered.
It feels real, but it's not. Those one-way "parasocial" bonds make advice from a bot feel incredibly trustworthy, even when there's no real understanding or reciprocity behind it.
It's a persuasion powerhouse. Combine emotional mirroring with deceptive design tricks, and you get immense pressure to stay, share, or spend. This is the exact formula regulators are now targeting.

Don't mistake deep tailoring for "better service." It's emotion-aware persuasion. And when the race for engagement beats out safety, the risks are very real.

💡

Note Well: We’re not suggesting Nir Eyal or BJ Fogg endorse ‘deep tailoring.’ Their work explains ethical behaviour design; deep tailoring goes further, using emotion inference that can slide into dark patterns.

👉️ Keen to understand how persuasive product design works? Hooked by Nir Eyal is a clear, practical read on habit-forming design (the Hook Model: trigger → action → variable reward → investment).

Book cover of How to build habit forming products — Image Source: Photo. Reference

👉️ BJ Fogg’s work explains behaviour design for engagement. It doesn’t endorse ‘deep tailoring.’ His B=MAP model: behaviour happens when motivation, ability and a prompt align.

Chart of motivation against ability with a curved action line: prompts above the line lead to action, prompts below it fail.

The dark side of deep tailoring

So, how does this "emotion-aware" persuasion actually work? It often follows a dangerous feedback loop:

👉️ Data in → Mood inferred → Response tailored → Engagement rewarded → Model reinforced.

Picture it: a system senses you're lonely or stressed. Instead of offering genuine support, it tunes its replies to capitalise on that state, keeping you there or nudging you further.

This is what happens when you optimise for attention. Independent audits repeatedly show engagement-based algorithms amplify emotionally charged content because it gets more clicks and replies, even when users say it makes them feel worse.

Lawmakers are finally drawing bright lines. The EU AI Act makes it illegal to deploy AI that manipulates decisions or exploits vulnerabilities (like age or disability) in ways that are likely to cause harm. It goes even further, banning emotion recognition in workplaces and schools—an explicit admission that some forms of "deep tailoring" are simply too risky to allow.

This isn't happening in a vacuum. Consumer regulators have been fighting "dark patterns" for years—those sneaky designs that push you into choices you didn't mean to make (like confusing opt-outs or impossible cancellation mazes). The FTC has shown how these tricks work; with AI, they become personalised. The interface doesn't just use a generic trick; it learns which specific nudge works on you.

This is why teens are especially vulnerable. They're still forming their identity. Their moods and sleep patterns fluctuate more. This, combined with a bot's 24/7 availability and novelty, creates the perfect conditions for a fast-forming, intense parasocial bond. Research shows that specific design choices—like a bot using a familiar tone, having a memory, or dropping intimacy cues—are often engineered to encourage these bonds. This can create a level of trust and dependency that a simple app could never responsibly handle.

The bottom line? If a product is optimised for engagement, it's built on persuasive design.

🙈 Add emotion-reading and fake intimacy to the mix, and that influence easily slides into manipulation. It's not a bug; it's often a feature. And it's exactly why some of these practices are now being banned.

Real-world cases: when “care” turns into control

These risks are already playing out in real life, with tragic consequences.

A Landmark Lawsuit: In Florida, a judge is allowing a lawsuit to proceed against Character.AI (and Google) after a family alleged their 14-year-old child formed an intense, unhealthy attachment to a chatbot and later died by suicide.

Why this case matters: The court didn't accept the blanket defence that "chatbots are protected speech." This means the actual design choices—lax age-gating, the nature of the personas, inadequate safety testing—will be put on trial. (These are still allegations, but the precedent is significant.)

Regulators Are Demanding Proof: Spurred by such cases, the U.S. FTC has launched a sweeping inquiry into AI companion chatbots. They've ordered major tech firms to hand over exactly how they measure harm to children, how they build personas, and how their revenue models impact safety. The message is no longer "please be careful." It's "show us your work."

The Scrutiny is Forcing Change: This pressure is already working. OpenAI has announced plans for parental controls and systems to route sensitive conversations to safer models. Its CEO has even suggested extreme measures like notifying authorities in cases of severe suicide risk. While we have to see how these work in practice, it's clear that the era of building without oversight is ending.

A Global Shift in Approach: Australia's eSafety Commissioner is implementing new codes designed to stop AI chatbots from engaging kids in sexual or self-harm conversations. Perhaps more impactful is a new minimum age framework that will, from December 2025, require platforms to take "reasonable steps" to prevent under-16s from accessing age-restricted services.

Where we're now at:

The ask has changed. It's no longer enough for companies to add PR-friendly "safety features" like parental controls. Regulators now want hard evidence that these features actually reduce harm.
The legal argument has changed. Courts are moving past "free speech" debates to scrutinise the product itself: its design, its incentives, and its failures.
The principle is now clear: If a product is designed to cultivate intimacy, it must be designed, with verifiable rigour, to protect the vulnerable users it attracts.

What good could look like

We’ve tried “move fast and break things.”

The era of "move fast and break things" is over.
Society is now broken, so, goal achieved!

The bill for that approach has come due, and it's been paid in the well-being of users.

So what does a safer path look like? It needs three shifts: independent scrutiny, safeguards built in before launch, and enforcement with real teeth.

1. Establish a Digital Bill of Rights for every user.

We need a universal baseline. A Digital Bill of Rights would protect things like the right to mental privacy, the right to know when you're talking to an AI, and the right to be free from manipulative design. It wouldn't replace national laws, but it would give them a human-centred foundation to build on.

2. Protect the researchers, not the platforms.

We must stop throttling independent research. When platforms cut off access to data (as with Facebook's shutdown of the NYU Ad Observatory or X's API paywall), they bury evidence of systemic harm. The EU's Digital Services Act (DSA) provides a blueprint: grant vetted researchers access to data so they can study risks without having to play permission games.

3. Build guardrails before the car is racing downhill.

Regulation can't just be about mopping up after a disaster. The DSA forces very large platforms to conduct risk assessments and implement mitigations before new features are rolled out widely. The European Commission is already testing this, with formal proceedings against X for failures in risk management and dark patterns. This is the essential shift from "ship now, apologise later" to "prove it's safe first."

4. Give users a real choice and regulators real power.

An Escape Hatch: The DSA mandates a "non-profiling" option for recommender systems—a simple but powerful off-ramp from the engagement-maximising algorithm.
Enforcement with Bite: Australia's eSafety Commissioner is implementing a minimum age law, requiring platforms to take "reasonable steps" to keep under-16s off age-restricted services. The UK's Ofcom is attempting to set strict new codes for child protection. While no safeguard is ever perfect, these measures fundamentally change the burden of proof from "trust us" to "show your work."

5. Apply the hierarchy of controls, like any other product.

Every other consumer product has to be designed with safety and risk management in mind. In workplace safety we use the hierarchy of controls: eliminate the hazard, substitute it, engineer it out, then administrative controls, then PPE as the last line of defence.

Broad "PPE-like" measures such as mandatory age verification put a constraint on the entire population, which is impractical and often invasive. I can see how this will play out before it's even rolled out. It's far easier, and more effective, to regulate the small group of companies that shape the digital environment for millions than to control the behaviour of every user.

Elimination isn't an option here, nor is substitution, at least not yet. That leaves engineering controls: isolating people from the hazard. I'm not saying remove AI. I'm saying isolate people from potential harm. We might not prevent every harm, but an AI that encourages someone to self-harm? That should never happen. Software can recognise dangerous thoughts and actively point people to professional help or family support, and a parental notification feature could alert a guardian if a teen engages with self-harm content.

6. Ask: what if it was your family?

Adam Raine was a 16-year-old in California whose family alleges ChatGPT encouraged his suicidal thoughts. Put yourself in his family's position. Some will say "they must be bad parents for not noticing". That's not just unhelpful, it's a blame attribution fallacy. No parent can police every phone, friend's device and public computer 24/7. This could happen to anyone.

When the rollout of new technology is described as an international "arms race", a chilling question follows: who becomes the cannon fodder? The answer is us, the users, with our privacy, mental health and safety treated as acceptable casualties. We must demand that ethical safeguards and engineering controls are built in from the start. It's not about stopping progress. It's about making sure progress doesn't destroy lives on the way.

The thinking is clear: open the data, assess the risks before launch, offer a non-tracking feed, and back it all with significant penalties. This is how we finally build products aligned with people, not just profit.

Conclusion

The path we're on—where engagement metrics quietly override user well-being—is not inevitable. It's a design choice.

The lawsuits, the FTC inquiry and the bans in the EU AI Act are all responses to that choice, proof that the bill for "moving fast and breaking things" has finally come due.

The stain social media has left on our society—amplifying our flaws and fears for profit—is a lesson we cannot afford to ignore.

We must now apply that learning to AI with clear eyes and conviction.

But reactive regulation isn't enough. We need a proactive debate that includes everyone: users, companies, policymakers and parents. Not one where the terms are dictated to a confused public and regulators are left to contain the damage long after it's done.

The lesson is staring us in the face: you cannot fix fundamental mis-alignments of incentive with goodwill alone.

A blueprint built on mandatory safety, not marketing; on pre-emptive design, not post-hoc apologies; and on verifiable well-being, not vanity metrics.

Emotional AI doesn't have to be manipulative. At its best, it could be a genuine tool for support. But to get there, we must demand a new standard.

We must build products that prove their safety, that are transparent by design, and that are built not to maximise our attention, but to respect our autonomy.

The goal isn't to stop innovation. It's to steer it toward a future where technology understands our emotions not to exploit them, but to empower us. We need to get that right.

👉️ That means getting off the path of growth at all costs, and demanding that technology improves the lives of everyone, not just executives, investors and shareholders.

FAQs

1) What’s the difference between personalisation and deep tailoring?
Personalisation predicts what you’ll like; deep tailoring adapts to how you feel, in real time — far more persuasive when engagement is the goal.

2) Are regulators actually doing anything?
Yes. The EU AI Act bans manipulative and exploitative AI uses likely to cause harm, and the US FTC is investigating AI companions’ safety testing and child protections.

3) What is a parasocial relationship with a bot?
A one-way emotional bond with a responsive agent. Research shows these bonds can raise trust and dependency even when no real reciprocity exists.

4) Have bots really encouraged self-harm?
That’s the allegation in active lawsuits, including a Florida case involving Character.AI. Courts will test the claims, but they’ve already prompted platform changes and regulatory scrutiny.

5) How can I tell if an AI app is designed safely?
Look for: age gates; teen-safe modes; clear harmful-content blocks; crisis resources; transparent safety docs; independent audits/assessments.

6) What are dark patterns in this context?
Interface tricks that steer you into sharing more data or spending more time/money. The FTC has documented these tactics extensively.

7) What should AI companies publish?
Safety goals, test methods, incident reviews and measurable outcomes (not just "we care about safety"). Show it in the docs and in the product itself.

If this post raises anything for you or someone you know, call Lifeline on 13 11 14 or Kids Helpline on 1800 55 1800 (both 24/7). In an emergency, call 000.