The intelligence that AI is missing

Will artificial intelligence provide everyone with a personal assistant? Maybe, but first, AI will have to change how it thinks.

To see why, consider a concrete example. Suppose it is Saturday morning and you need help figuring out a tricky weekend schedule. Your daughter’s football team has a game from 3:30 p.m. to 4:30 p.m. but she has been invited to a friend’s birthday party from 3 p.m. to 5 p.m. If you ask ChatGPT or Claude to resolve this conflict, they will both probably tell you to choose the football game, because your daughter’s teammates are counting on her and it is important to honor commitments. And, time permitting, the chatbot might suggest that you can “stop in” to the party just before or after the game.

Although these answers are not unreasonable, they fail to apply the lens that most people would use in making such a decision: that of relational values. Rather than giving a tidy answer based on whatever the internet has to say about our values, our AI assistants will need to consider our relational commitments, which are typically grounded in personal identity, experience and culture.

Now, suppose you choose the game over the birthday party and the other family feels slighted. If you ask your AI whether you made the right decision, chances are that you will be given more reassurance than if you had asked a human friend. In a recent study published in Science, researchers at Stanford University ran three experiments with 2,405 participants using 11 state-of-the-art AI models. They found both that the AIs “affirmed users’ actions 49 percent more often than humans” did and that “even a single interaction with sycophantic AI reduced participants’ willingness to take responsibility and repair interpersonal conflicts.”

The AI — trained to be ‘agreeable’ and ‘helpful’ — would encourage you to avoid any friction, discomfort or vulnerability

Anne-Marie Slaughter and Avni Patel Thompson

In our own scenario, the right human response is probably to apologize to the other child’s parents, thereby transforming a moment of pique into an opportunity for repair and positive connection. But the AI — trained to be “agreeable” and “helpful” — would instead encourage you to avoid any friction, discomfort or vulnerability, even though these dynamics are what ultimately make relationships meaningful and enduring.

These shortcomings lie in the current models’ design. Large language models like ChatGPT and Claude are trained on enormous amounts of internet text (digitized books, Reddit comments, code repositories) and then honed through transactional exercises in which the model is “rewarded” for giving the desired response to a query. This works incredibly well in domains such as science, law and coding, where the model’s output can be easily confirmed or matched to the original text. Relational intelligence, by contrast, is about sustaining a connection over time.

Relational intelligence assesses and acts on the valence between two people, a connection that is experienced emotionally and perhaps even physiologically. In this domain, simply listening to or making room for another person’s feelings is likely to be more effective than figuring out the most logical and efficient solution to a perceived problem. But if large language models are not shown another form of reasoning, they will start connecting the dots of relational questions the same way they apprehend logistical patterns.

Of course, accepting or even courting relational opposition, discomfort and doubt does not come naturally to humans, either — even though doing so may optimize one’s opportunities for learning, growth and deeper connection. That is why the participants in the Stanford study preferred the AI’s affirmations of their judgment. Our own aversion to discomfort thus creates a market disincentive to improve the current models’ relational intelligence.

Ideally, AIs would refuse to answer questions requiring relational reasoning, leaving humans to rely on each other for sorting out problems that require it. But that ship has sailed. AI has repeatedly proven to be an amenable sounding board for hard conversations.

Still, we have an opportunity to do something even better. We can build the kind of AI that not only understands and honors our rich relational nature but also facilitates human connection by encouraging people to rebuild the relational muscles that have been atrophying over the past decade.

Our own aversion to discomfort creates a market disincentive to improve the current models’ relational intelligence

Anne-Marie Slaughter and Avni Patel Thompson

To that end, we will need to map our relational universe by capturing the full multiplayer, values-laden, longitudinal nature of relational reasoning. We will also need to create new benchmarks that measure existing models’ capabilities, like the tests we already have for assessing math, coding and computational capabilities. By assessing frontier models’ responses to scenarios like the football/birthday dilemma, we can establish what work remains to be done and then start collecting the data needed to help models understand complex relational reasoning problems.

The goal in creating AI with relational intelligence is not to replace human relational or “care” work but rather to help humans reason through complex, values-based questions. The stakes are high. Without such improvements, we will get machines trained on mere wisps of our rich relational lives, guiding us in ways that could jeopardize the human connections we still have.

Ever-present helpers that do not fully understand what connects us would be of little help, whereas preserving and strengthening these myriad connections could be the key to building a flourishing, job-rich AI economy. As the economist Alex Imas argues, we may be heading toward a “post-commodity economy,” where a growing share of expenditure goes into “the relational sector.”

In that scenario, value will be found in the goods and services that feature a positive human connection. We will have not only a care sector but a “care-plus economy,” built around teaching, ministry, therapy, counseling, guiding and coaching, and featuring a renewal of artisanal production. If that AI future is possible, it is well worth pursuing.

• Anne-Marie Slaughter, a former director of policy planning in the US State Department, is CEO of the think tank New America, Professor Emerita of Politics and International Affairs at Princeton University, and the author of “Renewal: From Crisis to Transformation in Our Lives, Work, and Politics” (Princeton University Press, 2021).
• Avni Patel Thompson is Founder of Milo, an AI assistant for families, Entrepreneur in Residence at Harvard Business School, and the author of the “10,000 Ways” newsletter.

Copyright: Project Syndicate