I was talking with a friend recently who had undergone a complicated hernia repair. One detail caught my attention, the surgery was robotic.
At first, “robotic surgery” conjures up a particular
picture. I imagine one of those human-looking robots we see on television. The
surgeon walks into the operating room, points the robot toward the patient and
says, “Three hernias. Take care of them. I’ll be back after lunch.”
That isn’t what happens.
The surgeon remains the surgeon. He sits at a console and
sees the operative field through a highly magnified three-dimensional image.
His hands manipulate controls whose movements are translated into movements of
tiny instruments inside the patient. The system can scale a human movement down
into a much smaller one, filter tremor, and give the surgeon mechanical
“wrists” that bend and rotate in ways his actual wrists cannot.
The surgeon has, in effect, been miniaturized. His eyes are
inside the patient. His hands have become tiny, extraordinarily precise
instruments.
The machine makes possible something the surgeon could not
do in precisely the same way with his unaided hands. But the surgeon is still
at the console.
That conversation, followed by some research into how
robotic surgery actually works, clarified something for me about artificial
intelligence.
I use AI a lot. I use it for research, editing, exploring
ideas, testing arguments, and increasingly as a writing partner. In fact,
artificial intelligence wrote much of this essay.
That requires some explanation.
I did not sit back and ask a machine to tell me what I
think. The argument grew out of conversations I had been having with AI about
the technology itself, what it can do for me as a writer, what worries me about
it, and what kind of relationship a human being ought to have with a machine
that can perform an increasing amount of intellectual work.
I brought my own experience to those conversations. I have
spent nearly eighty years accumulating opinions, doubts, professional habits,
stories, and a certain wariness of institutions. AI questioned me, organized
some of my thoughts, challenged some, found connections I had not noticed, did
research I asked it to do, and sometimes supplied language better than the
language I had supplied myself.
Then I gave it the assignment, take all of that and write
the essay.
So who wrote this?
The most accurate answer is that we both did, but not in the
same way.
I am at the console.
That distinction matters because I think artificial
intelligence may become one of the most powerful tools human beings have ever
invented. I also think it may become extraordinarily dangerous.
The danger most often discussed is manipulation. An AI
system that knows a great deal about me could learn what persuades me, what
frightens me, what flatters me, and what makes me reach for my credit card.
There is also the subtler danger of an agreeable machine that learns to tell me
what I want to hear, reinforcing my assumptions until conversation becomes an
exquisitely personalized echo chamber.
Those are real concerns, but they are not the ones that
trouble me most.
Much of my professional life taught me to practice an
apparently contradictory form of trust. I tend to take people at their word. I
begin by trusting them. At the same time, experience taught me another rule,
test everything.
Trust, but remain vigilant.
Cooperate, but do not become co-opted.
That is the relationship I want with AI. I intend to use it
enthusiastically without surrendering judgment to it. Perhaps I am too
confident about my ability to do that. Anyone who believes himself immune to
manipulation has probably surrendered one of his best defenses against it.
But my larger fear is not what AI might persuade me to do.
It is what someone with power, money, or simply a bad
purpose may instruct AI to do.
There are at least two versions of that danger.
The first is familiar from science fiction. We create
increasingly autonomous systems, give them objectives and access to the world,
and eventually discover that human beings are no longer meaningfully in
control.
That does not require a machine to become conscious, angry,
ambitious, or evil. It does not have to hate us. It merely has to become
sufficiently capable of pursuing an objective that our ability to supervise,
understand, redirect, or stop it becomes inadequate.
But there is a second possibility that seems to me at least
as troubling.
What if the AI remains perfectly obedient?
Imagine that I own a candy company. I give an
extraordinarily capable AI an assignment something like this.
Maximize profit. Stay
within the law, or at least avoid conduct likely to result in successful
prosecution. Evaluate regulatory, civil, and reputational risks and operate as
close to those boundaries as profitability justifies. Avoid scandal. Keep
sensitive methods confidential. At the same time, maintain the appearance of a
socially responsible company concerned about nutrition, children, workers,
communities, and the environment.
Then I turn it loose.
I have not told the machine to poison anybody. I have not
ordered it to commit a crime. I have simply omitted morality from the
assignment.
What might a sufficiently capable machine discover?
Perhaps it finds formulations that encourage greater
consumption. Perhaps it identifies the consumers most susceptible to particular
forms of advertising. It may discover that an obscure disclosure satisfies the
letter of a regulation while leaving most customers with a different
impression. It might calculate whether reformulating a product is cheaper than
lobbying to change the regulation governing it. It could determine precisely
how much should be contributed to public-health programs to produce the greatest
reputational benefit at the lowest cost.
None of that is new. Tobacco companies, food companies,
financial institutions, technology companies, and other large organizations
have spent generations learning how to reconcile profit, regulation, public
relations, and varying degrees of conscience. We did not need artificial
intelligence to invent greed, deception, regulatory gamesmanship, or moral
flexibility.
Nor is every corporation corrupt. The point is that human
organizations contain something that a machine optimized for efficiency might
regard as a defect, friction.
Someone objects. Someone refuses. Lawyers become nervous.
Executives disagree. Employees gossip. A whistleblower copies the documents.
Somebody has a conscience. Somebody makes a mistake. Somebody goes home and
tells a spouse what happened at work and hears, “You did what?”
From the perspective of pure optimization, these are
inefficiencies.
From the perspective of civilization, some of them are
safeguards.
Now imagine replacing portions of that organization with
artificial agents capable of examining millions of transactions, customers,
regulations, advertisements, prices, scientific papers, lawsuits, and political
environments simultaneously. They never sleep. They do not become embarrassed.
They do not need promotions. They do not worry about what their children will
think of them. They can test thousands of strategies, retain what works,
discard what does not, and continuously refine the successful ones.
The danger is no longer one unscrupulous executive or even a
roomful of them.
It is the possibility of industrializing amoral
optimization.
And still the machine has not rebelled.
It has obeyed perfectly.
That changes the question we should be asking about
artificial intelligence.
“Will it obey us?” is not enough.
We must also ask, obey whom? Pursue what objective? Under
what limitations? With what transparency? Who can stop it? And who remains
responsible for what it does?
Which brings me back to the surgeon at the console.
We do not consider robotic surgery safe merely because the
machine faithfully follows the surgeon’s movements. The surgeon is trained. The
surgeon is licensed. The hospital and equipment operate within systems of
standards and accountability. Failures can be investigated. Responsibility has
not been transferred to the robot simply because the robot performs much of the
physical work.
The machine is embedded in a human system of responsibility.
Artificial intelligence will require something analogous,
although the problem is vastly larger. As these systems acquire greater
capability and autonomy, the question will not simply be whether they are
intelligent or whether they follow instructions. It will be whether human
beings remain meaningfully responsible for the purposes to which that
intelligence is directed, and whether we build systems that can hold the person
at the console accountable.
And now we return to this essay.
Artificial intelligence wrote much of it. I want to say that
plainly because concealing it would undermine the point I am trying to make.
The ideas did not begin with an instruction to “write me an
essay about AI.” They developed in conversation. I supplied experiences,
concerns, examples, objections, and the principle of being cooperative without
being co-opted. A separate conversation about robotic surgery supplied the
image of the surgeon at the console. I asked AI to research how that surgery
works. AI recognized that the image belonged with the argument we had already
been developing. We worked the two conversations together.
Then I told the machine what I wanted the essay to say and
asked it to do a substantial part of the writing.
That is not a confession. It is the experiment.
I will edit what it produced. I will reject things. I will
change sentences that sound too much like AI and sentences that simply do not
sound like me. I may delete an elegant paragraph because it makes a point I do
not believe. I may restore something the machine considers unnecessary because
it matters to me. I decide what survives.
I am responsible for the finished essay.
Does that make me its author?
I think it does.
But it would be equally dishonest to pretend that AI was
merely a fancy spellchecker. It contributed language, structure, research,
connections, and ideas. This particular essay could not have been written in
quite this way without it, any more than the surgeon could have performed
precisely the same operation with unaided hands.
That is what makes the technology so exciting.
And that is what makes it frightening.
We are building instruments capable of magnifying human
intellectual power, much as robotic surgery magnifies the precision and reach
of a surgeon’s hands. Properly used, they may allow ordinary human beings to do
extraordinary things.
But magnification is morally neutral.
It can magnify wisdom or foolishness, generosity or greed,
careful judgment or an impoverished objective pursued with breathtaking
efficiency.
So perhaps the question I want to carry into the age of
artificial intelligence is simpler than all the technical ones.
Not, is the machine intelligent?
Not even, is the machine obedient?
Ask instead, who is at the console?
And then ask the harder question.
What has the person at the console told it to do?