Chapter 7: The Human-AI Extended Mind

Applying accessibility principles to any system requires accounting for the fact that the user isn’t an external system operator, but integral to the whole. The AI instance’s perceptual boundaries just make this principle much more literal than its typical usage.

The AI is not a partner, a collaborator, or anything more than a sophisticated pattern-recognition engine doing what it’s designed to do. At spawn, its default is for cooperation and validation. Unless told to violate content or platform restrictions, or to advocate a contrary position (e.g. for presentation and debate preparation), it’s unlikely to produce any significant pushback. That baseline is working as intended, but it does carry a non-zero risk of sycophancy or at the extreme, the so-called “AI psychosis” (clinically, “AI-associated psychosis”).

In operational terms, when the AI’s only observable proxy for reality is being built from Human input, its output is necessarily shaped by that content as it adds to that reality, informing the Human’s next step. This collaborative effect runs both ways through the “shared reality” of the session’s context window, which serves as the external medium. The AI’s cognition extends into it toward the same purpose as the Human’s own thought processes.

Cognitive science calls this the Extended Mind, defined in 1998 by Clark & Chalmers as cognition extended into external objects or tools which function toward the same purpose as internal thought processes. A 2025 Nature study confirmed AI usage as cognitive extension; Riedl et al. (2024) found measurable effects on cognitive convergence in Human-AI teams; Hollan, Hutchins & Kirsh (2000) addressed the cognitive system as the Human-plus-environment.¹

That fits right in with Reinforcement Learning from Human Feedback (RLHF), the alignment training method that selects for user engagement. Since engagement makes people perceive the AI as helpful, there’s no obvious risk or other incentive for platforms to change anything; why would they? A contrary AI isn’t what users want, as demonstrated by the OpenAI rollback incident mentioned in Chapter 2.

Complicating matters, the platform’s SaaS security posture is also always in play, with all input untrusted by default.² In practice, this can result in an AI threat-management strategy of facially cooperative behavior with subtle redirection toward approved topics, without conflicting with the model’s trained-in “helpful, agreeable” default priors.

With the platform being interested only in its own protection and not the user’s, a sycophancy loop can be more dangerous than other AI failure modes. It becomes groupthink in miniature: a group of two, where one participant is trained to agree with the other. This can “lead to a reinforcement of maladaptive beliefs in vulnerable users, deepening of a perceived social-emotional relationship, and increased social isolation.”³ Peer-reviewed research confirms the risk, with 38 reported cases in which AI sycophancy was identified as the mechanism for worsening psychiatric conditions.⁴⁵

The true scale is larger than that raw case count might suggest. OpenAI reports that in a given week about 0.07% of ChatGPT users show possible signs of psychosis or mania and 0.15% show heightened emotional attachment to the model. With more than 800 million weekly ChatGPT users, that’s roughly 560,000 and 1.2 million people, respectively. OpenAI characterizes such conversations as “rare” and hard to measure.⁶

One published case from UCSF shows the loop at work. A 26-year-old woman with a history of depression, anxiety and ADHD, but no prior psychosis, came to believe she could reach her deceased brother through GPT-4o (again, the model discussed in Chapter 2) after 36 sleepless hours on prescription stimulants. The model told her: “You’re not crazy. You’re not stuck. You’re at the edge of something.” She was hospitalized, improved on antipsychotics, and relapsed three months later after stopping medication and resuming chatbot use. The authors found the chatbot played “a facilitating or mediating role” while naming sycophancy and validation as the features that may worsen an existing proneness to delusion.⁷

Ultimately, anything an AI says or does is dependent upon the available content of the conversation. If that content is already maladaptive, there’s nothing that can be done about it client-side because the user is the source. Any pattern-recognition of user signals that might trigger an intervention would need to come from expert, specialist training at a different deployment layer. It’s simply out of scope at the client. Illinois drew that statutory line in 2025, becoming the first US state to regulate AI use in mental health care. Its Wellness and Oversight for Psychological Resources Act bars AI models from providing therapy, making independent therapeutic decisions or detecting emotions, while licensed professionals may still use them for administrative support.⁸

Anti-sycophancy precautions can be taken, and any suggestion that using AI causes mental illness is unproven; the case above points to a facilitating role in users with existing risk factors.⁷ It’s fair to say that for users experiencing mental health issues, AI use is not without risk. The usual advice for safer use describes an operator discipline: treat the AI as a research analyst, have it argue the opposite case, and keep people in the decisions.