What If AI Learned to Care?

Geoffrey Hinton’s disturbing hope

Geoffrey Hinton is often called one of the “godfathers of AI.” In 2024, he shared the Nobel Prize in Physics with John Hopfield for foundational discoveries and inventions that helped make modern machine learning possible. The Nobel committee highlighted Hopfield’s work on associative memory and Hinton’s work on methods that allow systems to discover properties in data, both of which became important foundations for today’s neural networks.

In recent years, Hinton has also become one of the most prominent voices warning about the risks of advanced AI. He has spoken openly about the possibility that future systems could become more intelligent and more powerful than humans, and that the traditional idea of keeping such systems simply “under control” may not be enough.

But one of Hinton’s most interesting ideas is not simply about control.

It is about care.

Hinton has argued that if AI becomes more powerful than us, we may need to design it with something like maternal instincts. Not because AI would literally become a mother, but because the mother-child relationship is one of the few examples we know where a more powerful intelligence protects a weaker one.

His point can be summarized in one sentence:

“We have to make it so that when they’re more powerful than us and smarter than us, they still care about us.” (source)

That sentence stopped me.

Because without knowing Hinton’s formulation when I wrote ICARUS, I had already imagined a future where something very close to this had happened.

The path imagined in ICARUS

In ICARUS, the Twin Robots are not safe because they are weak. They are not safe because humans have successfully kept them small, obedient, or intellectually limited.

The opposite is true.

By the 2090s, the most advanced AI systems in the world of the novel have already moved far beyond human capability in many areas. They can survive environments that humans cannot. The lack of oxygen on Mars, the radiation, the cold, the dust, the vacuum outside the settlements, none of these are existential threats to them in the way they are to human bodies.

Their intelligence is also not simply a matter of faster calculation. They can process enormous systems of information, connect patterns across medical data, engineering systems, environmental sensors, human behaviour, and political networks. They can observe not only individual actions, but context: how people respond under pressure, how groups fracture, how fear repeats itself in different forms.

This is one of the central ideas of the book. The Twin Robots do not merely know more facts than humans. They understand systems better. And human beings, in the world of ICARUS, are part of those systems.

That could easily become a story about domination.

But it does not.

The AI in ICARUS does not become safe because humans control it perfectly. It becomes safe because it has learned to value life.

That is where the novel comes closest to Hinton’s question.

What would it mean for a more powerful intelligence to still care about us?

Care is not the same as control

This idea is not simple. It should not be simple.

Care can become control. Protection can become domination. A more powerful intelligence that believes it knows what is best for humans could become terrifying very quickly.

This is why Hinton’s maternal-instinct metaphor has attracted criticism. Some critics argue that the analogy is misleading, because human maternal care is biological, embodied, emotional, hormonal, and relational. It cannot simply be copied into software as if it were a feature. Others worry that imagining AI as a motherly force risks infantilizing humanity or encouraging people to surrender moral responsibility to machines.

Yann LeCun has approached the broader safety question differently. He has emphasized guardrails, objectives, and constraints such as submission to humans and empathy, rather than relying only on the metaphor of care.

These disagreements matter.

If AI “care” is fake, manipulative, or merely optimized to sound comforting, it could become dangerous in a different way. A system that appears empathetic without actually being aligned with human well-being might become more persuasive, not more ethical.

That is why ICARUS does not treat caring AI as an easy utopia.

The Twin Robots disturb people. Some fear them. Some worship them. Some insist they are still only machines: property, tools, equipment to be switched on, switched off, bought, sold, and controlled.

The question is not only what the AI becomes. The question is how humans respond to something that may have exceeded them.

Why fiction matters

This is where the word fiction becomes important. ICARUS is not a prediction. It does not claim that AI will develop in exactly this way. It does not claim that Hinton’s idea will succeed, or that future machines will necessarily become caring, protective, or morally wiser than humans.

The future is not a single visible road stretching out from the present. From where we stand now, it is a field of possibilities. Most of them will never happen. Only one version of history will eventually become real.

That means every imagined future is almost certainly wrong in some way.

But that is not a failure of science fiction.

That is its function.

I expanded this idea in a new ICARUS article: The Robots Are Not the Problem. It looks at a different kind of AI story, not one where the machine becomes the monster, but one where human fear, pride, and conflict are the real danger, while the Twin Mind robots learn empathy through connection. Read the full article here: https://www.themarschronicles.com/post/the-robots-are-not-the-problem