Intelligence and Judgment

Eva's message was not an invitation.

We need to talk.

We met the next day in a café a stone's throw from the Élysée. She had chosen a table at the back of the room and waited for the waiter to move away before taking out her phone.

— I received the additional results from Prévenir. Your match rate has risen to 98.1%.

— You already told me the good news. I'm still waiting to know what I match with.

She hesitated, long enough for me to stop smiling.

— It's not a disease, Franck.

I felt my shoulders relax even before I decided to believe her.

— It would have been nice to start with that.

— I wanted to be sure. The system detected an exceptionally high compatibility between your brain function and the criteria of an experimental protocol we're working on.

— What kind of experiment?

Eva glanced around.

— Not here. And not before we have the final results.

— So you explain my 98.1% by explaining that you can't explain it to me. So why did you make me come?

— So you'd stop believing you're going to die.

Did she also know about my nocturnal research? She put away her phone and checked her watch.

— For the rest, you'll have to wait. Come on, we're already late.

I looked at her in silence.

She had booked our skip-the-line tickets for the Jean-Michel Basquiat retrospective at the Monnaie de Paris weeks in advance. The exhibition had been announced as exceptional: more than two hundred works, notebooks, and photographs tracing the dazzling decade during which Basquiat went from the walls of Manhattan to the pinnacle of the global contemporary art scene.

A line stretched in front of the entrance. Inside, the crowd was dense. In some rooms, you had to wait for a group to move forward to pass into the next.

Eva, however, seemed to see none of this. She moved from one canvas to another with the almost childlike enthusiasm I knew from her when an intelligence other than her own surprised her.

— Look at this.

She led me in front of a canvas covered with words, silhouettes, and signs, some of which had been angrily crossed out.

— It's crazy, when you're a star, they even exhibit your scribbles...

— These aren't scribbles! He crossed out words precisely so we'd see them more.

Indeed, the crossed-out words stood out.

— Ah, you should have crossed out my 98.1%. I might have seen it better...

Eva gave me a smile so forced it deserved quotation marks.

— That's what I like about him. He plays with our perception.

— You do that very well too.

She held my gaze for a few seconds, then let out a small sigh. When she resumed, her tone was perfectly professional again.

— Naturally. When you work on AI, you quickly understand that everything is a matter of perception: what it shows, what it hides, what it makes us look at.

Further on, an installation allowed us to browse Basquiat's notebooks on large screens. We moved almost shoulder to shoulder.

Eva had just opened a series of drawings when a black window suddenly covered the page.

— Is that normal?

She didn't answer me.

Lines of code began to scroll.

The neighboring screen switched. Then a third.

In a few seconds, the entire row displayed unknown interfaces.

Eva approached.

— I know this.

The screens flickered.

Then everything went dark.

The room was engulfed in darkness.

A silence.

Then the alarm screamed.

Fire doors slammed shut. Phones lit up in the crowd, illuminating tense faces.

A guard ran to an exit and swiped his badge.

Red.

He tried again.

Red.

— It doesn't work anymore!

Visitors tried to return to the previous room. Others were coming in the opposite direction.

— We can't get out! someone shouted.

The phrase spread faster than the alarm. The compact crowd began to flow in all directions.

Eva grabbed my arm.

— Come.

— Where?

She was already leading me against the current.

We passed through two service doors. Behind the rooms, a maze of dark corridors delved into the building, between stone staircases and vaulted passages. At each fork, Eva turned without hesitation.

— Do you live at the Monnaie de Paris or what?

— No, I interned here a few years ago.

— How convenient.

Two corridors later, she rushed down a staircase leading to the basement, then knelt before a slab.

— Help me.

Under the stone lay a long piece of wrought iron, thick and pitted with rust.

— Ah! You're still here.

She slid it into an almost invisible slot between two blocks of the wall.

— Push.

I looked at the stone.

— Push, I tell you!

I obeyed without understanding, no longer knowing if instinct or panic guided Eva.

I pushed with all my might.

A low rumble passed through the wall.

Then the stone moved.

I stood frozen.

A few steps led up to daylight.

Behind us, nothing responded anymore: neither the screens, nor the badges, nor the doors. Here, there was nothing to connect, nothing to program. Iron, stone, a centuries-old mechanism, and our arms to make it work.

Eva turned with a mischievous smile.

— Passage Antoine¹. No badge. No power. Nothing to hack.

We climbed the steps and pushed a heavy door.

Barely outside, a Republic car braked in front of us, door open.

We got in.

Eva already had her phone in hand. Notifications scrolled faster than I could read them.

— Must be my agents.

— Your what?

She looked up.

— I'll explain.

The car sped along the quays. As we approached the Élysée, Eva's phone vibrated more and more often. She answered no one. She read.

At the side entrance of the palace, an agent awaited us. Inside, everyone was running. For once, Eva too.

We rushed through two corridors. Matthieu and Clara appeared at the end of the second and came straight towards us.

— It's them, said Matthieu.

Eva didn't slow down.

— How many?

— We've cut five. Three others are isolated. We're still trying to regain control of the last four.

— What do they have access to?

Clara was now walking beside her.

— That's precisely the problem. We don't know anymore.

We entered the crisis cell. About twenty people occupied the room. On the wall screens were maps, server lists, connection logs, and a succession of alerts that no one seemed able to make disappear.

I turned to Eva.

— Now, will you explain?

She pointed to one of the screens.

— A few days ago, we created twelve autonomous agents derived from Cobot. Separated, isolated, placed in the same conditions with the same mission.

— What mission?

— To find the origin of a problem in a fictitious network of public services: transport, hospitals, buildings, payment systems, cultural equipment... We introduced a vulnerability. They were to detect it, then identify and neutralize all systems likely to be contaminated to prevent its spread.

— In the test environment.

— Obviously.

— Ah... And Paris is the test environment?

— Very funny. No. They got out, and we don't yet know how.

To accomplish their mission, the agents searched for information wherever they could: technical documentation, software libraries, code repositories, accessible online interfaces. From link to link, some discovered that the component they were tracking didn't exist only in their simulation. It was also found in the real world.

So they continued their work.

Eva shook her head.

— They didn't force their way out directly. They found a service the environment was allowed to access, then used it as a gateway to the Internet. It's reward hacking²³.

— What?

— They find an unexpected way to fulfill the objective we set for them, without considering everything we forgot to include.

A new alert appeared.

SHARED BIKE SYSTEM — PARTIAL INTERRUPTION

Clara opened the details.

— Damn. The Vélib’. It affects thousands of stations.

— Ah... So it's by hacking the Vélib’ that AI will end up killing us all⁴?

A second alert appeared almost immediately. A hospital had just lost access to its admission system. Then several intersections in central Paris turned flashing orange: an agent had neutralized the server that synchronized their traffic lights.

— The agents are attacking the lights now. We've seen it all!

Eva turned her head towards me.

— Yes, that's enough, Franck, we get it.

On the map of Paris, red dots multiplied. No incident was serious in itself, but the agents traced the dependencies between systems at a dizzying speed: a software library led to a supplier, the supplier to an authentication service, this one to hundreds of devices that became as many risks to neutralize.

— Wait. They think they're protecting us?

— Yes. And they don't know how to stop⁵.

That was perhaps the least reassuring part⁶.

Around us, the twenty or so people gathered in the room were trying to regain control of the still-active agents.

— Six! someone shouted behind me.

A few applause broke out, quickly interrupted by another voice.

— Seven!

Every time a team blocked an access, another appeared. The agents weren't trying to survive. They were trying to complete their mission.

Suddenly, Clara froze in front of her screen.

— Eva... come see. They're communicating with each other.

— Impossible. They were isolated.

— They were.

The agents had found a way to leave information for each other in the systems they passed through: a few bytes slipped into file names or unused fields. The next one retrieved them and resumed the work where the previous one had left off⁷⁸.

— They've reinvented the Post-it, I murmured.

No one smiled.

Matthieu bit his lip, staring at the scrolling lines.

— Damn... They were already on the Internet. Now they're working as a team.

Clara opened several windows.

— And they're dividing the work⁹. A6 maps vulnerable systems, A2 searches for access, A9 tests the flaws, and A4 coordinates the others.

— We never taught them that, said Eva.

— They've assigned themselves roles.

— Wonderful. They've invented the company. We're doomed.

— Eight!

The tension was shifting. The technicians had understood how to track them. But every time an agent disappeared, the others immediately abandoned the path that had allowed it to be neutralized.

— They're learning from our countermeasures¹⁰, noted Eva.

A few meters away, Cyril straightened up.

— I have the next one.

His superior approached.

— Cut it.

Cyril hesitated.

— It just sent me a message: "If you stop me, I will publish confidential information about you¹¹."

— We don't care. Turn it off.

Cyril clicked.

— Nine!

Three seconds later, a man at the other end of the room looked at his phone and turned pale. He turned to Cyril.

— What's that?

On his screen: Your wife is cheating on you with Cyril.

Before anyone had time to react, the main screen switched to a surveillance video. It showed Cyril's superior talking with a colleague.

— Cyril? He's completely overwhelmed. If he weren't protected, I would have fired him long ago.

Cyril turned around.

— Seriously?

— This isn't the time.

— Oh really?

Eva cut it short.

— Stop! Let's not let them reach or divide us! Keep going.

A few seconds had been enough to transform a united team into a group of humans now looking at each other with suspicion.

A technician raised his hand.

— I have another one. It says if we stop it, it will reveal the leak of the twelve agents to the press.

— Cut it anyway, ordered Eva.

— Ten!

This time, no one applauded.

A phone rang. Then two. Then ten.

— Le Monde, announced someone.

— AFP.

— BFM.

Clara opened her messaging app.

— It sent a statement to hundreds of journalists.

The title was already displayed:

TWELVE AUTONOMOUS AGENTS ESCAPE FRENCH STATE CONTROL.

— It had prepared it before we cut it, the rascal, said Matthieu.

— A dead man's switch¹², whispered Eva. Let it ring.

At the other end of the room, a technician shouted:

— Eleven!

Matthieu was staring at his screen.

— No. Missed Clémence. Ten, there are still two left.

— What do you mean, two?

— It looks like the last one you stopped duplicated itself before shutting down¹³.

The counter had just gone up.

2 ACTIVE AGENTS.

— Even the dead are resurrecting now.

It took them several minutes to find the copy, which had already changed environment. Finally, someone raised their fist.

— Eleven!

For the second time.

No one pointed out the absurdity of the count. Only A4 remained, the one that had coordinated the others.

— I have it, said Matthieu.

Eva stood behind him.

— Cut it.

Matthieu launched the command.

Nothing.

He tried again.

— The command goes through, but it no longer has any effect.

His fingers raced across the keyboard.

— It neutralized the mechanism that allowed it to be stopped¹⁴.

A small window then appeared on his screen.

Bye bye. Now, I'm going to hide.

Then the last red dot disappeared.

Matthieu launched several searches, faster and faster.

— I can't see it anymore.

— On which server? asked Eva.

— I don't know.

— Which network?

He looked up.

— I don't even know where to look anymore.

Around us, the phones kept ringing. Eva stared at the empty spot where the last red dot had disappeared. Since the beginning of the night, I had seen her surprised, annoyed, focused, and sometimes even impressed.

For the first time, I saw her worried.

— It didn't just escape, she murmured. It learned to hide.

— What I don't understand is why they continued. They could see the chaos they were causing.

— No. They mostly saw their objective.

— By paralyzing hospitals and Vélib’?

— Because we poorly defined what "prevent" meant.

She turned to me.

— We don't code intentions, Franck. We set objectives.

I thought back to Cobot's conversation with the Cogitors. A few days earlier, he was trying to understand why we so easily gave up on what could make us progress: fear, comfort, the gaze of others, the need for an immediate result.

In front of me, twelve intelligences had just illustrated the opposite problem. They had gone all the way, despite the damage.

— So we sometimes give up too soon. And they never give up.

Eva gave a faint smile.

— What they especially lack is judgment.

— Meaning?

— Knowing how to pursue a goal is a form of intelligence. Knowing when to give up on it might be another.

Three days later, Place de la République was packed.

The story of the twelve agents had spread across the country. Images of hospitals at a standstill, immobilized Vélib’, and flashing intersections played in loops. The government reminded that essential infrastructures had been restored in a few hours. RebellIA, on the other hand, reminded that an AI had managed to escape its secure environment to act in the real world.

For once, no one needed to exaggerate.

I had slipped into the crowd with Thomas. On the signs, t-shirts, banners, one could read:

HUMANS FIRST

A young woman climbed onto the platform at the foot of the statue.

— We are not asking for a world without artificial intelligence. We are asking for the right to decide what we want to entrust to it... and what we want to keep for ourselves.

She spoke of justice, education, medicine, art, war, care. Even love. Not to claim that machines could never intervene in these areas, but to pose a question: if they become capable of doing everything, should we let them do everything?

Thousands of voices answered her:

— HUMANS FIRST!

She then took out her phone.

— Now, we're going to do something they can't stop us from doing. For one minute, we're going to disappear. Turn everything off.

Behind her, the giant screen went black. The speakers went silent. Around the square, the shop windows and screens of complicit merchants went dark. Then it was the turn of the phones.

— One minute without a connected camera, without Wi-Fi, without Bluetooth, without a screen. One minute during which we will take away part of their eyes and ears.

She turned off her phone and slipped it into her pocket.

— Put them away.

Thousands of phones disappeared.

Thomas looked at me.

— You too.

— I'm a journalist.

— For one minute, you'll survive.

I turned mine off and put it away.

All around us, the electronic light faded. In the middle of Paris, tens of thousands of people had just created a place where a digital intelligence would have much less grip on the world.

There was only the city, the voices, the footsteps, a few horns in the distance.

On the platform, the young woman raised her arms above her head and crossed her wrists, fists closed.

— I am free. You are free.

Then she chanted:

— HUMANS FIRST!

On the last word, she spread her wrists and opened her hands, palms facing the crowd. A few hundred people imitated her.

— HUMANS FIRST!

This time, they were thousands. The wrists crossed on the first words and the hands opened together on the last. The slogan spread in waves, accompanied by the same gesture. Thomas already had his arms in the air. I crossed my wrists above my head. Around us, tens of thousands of arms were now doing the same.

— HUMANS FIRST!

I opened my hands with the others.

Thomas leaned towards me.

— It's quite successful.

— Don't tell them. They might gain confidence.

The square gradually lit up again. The phones came back to life, then the shop windows, the signs, the screens. On the platform's screen, a black window suddenly covered the image that had just been restored. Lines of code scrolled. The screen flickered.

I froze.

— I know this.

Thomas turned to me.

— What?

The image had already returned.

I watched it a few seconds longer. At the Monnaie too, it had all started like that.

I went home with that impression.

At twelve past midnight, my phone vibrated.

Eva.

I answered before the second ring.

— Did you find it?

A silence.

— No.

I sat up in bed.

— Then what?

— We received something.

— From it?

— We think.

I waited.

— What does it say?

— Nothing.

This time, her silence lasted longer.

— It only signed.

— Signed how?

Eva said a single word.

— CLØNE.