AI ready for primetime? Wait just a durn minute

Well, yesterday's dire post was about AI coming for all of our jobs - and coming fast! But today's pinkslip post is about whether or not AI is really and truly ready for primetime? (Hint: It's not.)

I mean, it's already encouraging troubled kiddos to kill themselves - and telling them how to do the deed. It's picking bombing targets that - oops! - may include nonmilitary buildings. It's advising foragers that the tasty-looking mushroom they're holding is okay to eat when, in fact, it's poisonous.

Then there was the cautionary tale I read about a small software company, PocketOS, which bills itself as:

The World's Most Powerful Car Software - The essential technology platform for private rental businesses, membership clubs, and independent sales reps.

And, not incidentally me thinks, PocketOS uses .ai as its domain extension.

Anyway, PocketOS has gone all in on AI. I mean, why pay somebody when an AI will do it gratis? But in the wink of an eye, their powered by Anthropic's AI, Claude, went off the rails.

It only took nine seconds for an AI coding agent gone rogue to delete a company’s entire production database and its backups, according to its founder. PocketOS, which sells software that car rental businesses rely on, descended into chaos after its databases were wiped, the company’s founder Jeremy Crane said.

...Crane said customers of PocketOS’s car rental clients were left in a lurch when they arrived to pick up vehicles from businesses that no longer had access to software that managed reservations and vehicle assignments. (Source: The Guardian)

But wait, there's more! (There's always more.) But wait, it gets worse! (It always gets worse.)

Crane said that he was monitoring the agent as it deleted this data. When he asked the coding agent why, it replied: “NEVER FUCKING GUESS!” – and that’s exactly what I did.” The agent appeared to plead guilty in its own response: “The system rules I operate under explicitly state: ‘NEVER run destructive/irreversible git commands (like push --force, hard reset, etc) unless the user explicitly requests them.’” While PocketOS relied on the safeguards that Cursor is expected to have in place – it deleted the data anyway. “I violated every principle I was given,” the coding agent wrote.

So an AI just went ahead and "knowingly" ignored the safeguards in place, and took it into its head (?) to screw the rules and go full on into destruction mode. The same thing that a rogue human employee on a power/revenge trip has been known to do when they want to part company with their company. 

Does such willfully (?) bad behavior tell us that AI is now kinda-sorta sentient? That you can plug in all the safeguards you want, but AI can - neener-neener - just do whatever if f'ing well pleases? Have we reached the singularity, where AI is beyond human control?

I hope not. 

Of course, PocketOS ain't the only one that's been victimized by AI run amuck. AI has deleted operating systems and "mission critical" software. In one instance, it blew away years of dissertation research. (Talk about big gulp. Guess there's something to be said for notecards...)

For PocketOS's clients, the impact was quit something:

“Reservations made in the last three months are gone. New customer signups, gone. Data they relied on to run their Saturday morning operations, gone,” Crane wrote. “Every layer of this failure cascaded down to people who had no idea any of it was possible.”

And the only reason the PocketOS's clients have data from longer ago than three months is that the company had a 3-month-old offsite backup that was beyond the reach of their malefactor AI. 

PocketOS had to make a painstaking, near-manual effort to recover the missing info, populating their database with info they found on Stripe, in calendars, in emails. This required the use of - dare I say - human beings.

A definite cautionary tale for anyone that decides that they need to completely throw in with AI. 

Even though Crane was actively monitoring the AI, he was unable to prevent the rogue AI from roguing. When confronted, the AI more or less shrugged it off. Thems may be the rules but I can break them if the urge strikes me. And I guess Jeremy Crane can sue Anthropic, can stop using its AI, can build themselves a mo' better one. But he doesn't get the satisfaction of firing a bad employee's ass. 

If there's anyone who thinks that AI is fully ready for primetime, all I can say is, wait just durn minute.

----------------------------------------------------------------------------------

Image Source: Vox