AI is just another gerbil in the sack

What happens when a user tells you a system number is wrong? And what happens next? Turns out, a lot - wrong numbers are a very common reason people lose trust in systems - notably when they have to rely on them. But it wasn't the wrong number that lost the trust - it was the missing part to connect it back to reality again. And to explain, lets go back to airlines, as I spend too much time on damn planes. (Don't worry, this is about system design, humans and how AI affects it, I'll get there...).

A cyberattack teaches us about AI in an unexpected way

When Heathrow Airport had a large cyberattack last year, I had to do a flight out of Heathrow. Notably, this was with TAP Air Portugal, notorious for generally poor IT systems. Lufthansa and several other airlines had failed to get a flight out on time next to my gate, but TAP had something interesting going on... there were at least 5-6 TAP staff at the gate, doing various things - one person on phone to the luggage loader to confirm whose luggage was actually there, several people handling workarounds to the normal gate check in process, and you'd see little notes, overrides, being made notes of.

TAP Air Portugal's previously poor IT had actually been an advantage during this cyberattack - they had staff already enabled to override software settings, and other software based workarounds via other systems to get those 200-ish people and their bags loaded onto a medium haul flight. Their "human in the loop" had created a temporary "gaffa tape" team to deal with the temporary complexity.

I believe this teaches us something important:

• Systems that embrace users rather than treating them as something to be removed tend to be more resilient
• More complex systems, AI, add more uncertainty and reduce resilience. The Heathrow example had no AI, but airline systems are terrifyingly complex, which can produce unpredictable output like an AI system can
• Rerouting and embracing more links outwards to people who can correct stuff are key parts of improving resilience
• The Real Resilience isn't whether the booking system at the airport is down - it's whether you can still get on that plane anyway

AI has added something new to this cocktail. We've added a new way outputs in systems can be generated from inputs. And the systems that are going to survive, are the ones that design for AI in a new way.

AI on its own is not a product

If you use an AI product, you don't 'see' AI in front of you. You actually see all the other, normal software things from before.

Because in reality, the AI isn't a UI, or a chat interface, nor is it HTML or an API. You still need those sorts of things for a piece of software to make sense. The "AI bit" is a set of methods of determining outputs from inputs that is part of your overall solution. So it's more like:

AI bits of your product <-> your other code <-> the outside world.

And sadly, it's not like my linear little line above. It's more an amorphous, constantly moving set of gerbils in a sack. The AI bit has models that have assumed various things about the outside world. You probably have a bunch of business logic in the other code too. The outside world is constantly changing too, and if you're aware of that complexity or even properly debugged something, you will have plenty of examples about how it can all fall over.

The question is... how do we keep these assumptions in sync? If a user is saying "this number is wrong" how do you turn that into something that all of your product's code can use, and something to help other users later?

Embrace the chaos

Software simplified the world by design. We have multiple generations of software workers drawn to this appeal. The ability to take something complex and make it understandable, controllable. The basic move: take a restricted element of the outside world, turn it into a requirement, build code. Complex to simple.

This is the antithesis of how the real world works. Stuff that doesn't fit is "an edge case" or "out of scope". You convert the human into computer.

It worked, when software touched a restricted slice of the world. But software now touches enough of the real world that it is an active part of it. And when you touch everything, you inherit its complexity, plus a new layer of misunderstandings on top. Where statistical models predict likely data rather than certain data, that complexity compounds further.

This no longer works. You have to instead convert the computer into something more human.

Your only way to design for AI and modern systems is to embrace human behaviour rather than try to eliminate it. Give users more control, more options, more abilities to correct. And track those corrections to feed back into your logic or model.

Some modern principles to help

What does this look like more practically? It's a multi-disciplined area - design, engineering, product.

• Redundancy - allow your software multiple ways to do the same real world outcome. That could be an AI module to do X and a non-AI fallback, or even different business logic between them, or letting users override things. Elegance is no longer about having one clean way to do it
• Risk assess - things are going to go wrong. Assume they will, and design accordingly.
• Create routes of monitoring human outputs of systems that feed your models (and your non-AI code too!) - that can be ensuring you are recording corrections sufficiently
• Design your human in the loop explicitly - it's not a fallback. It's part of a living, breathing ecosystem you're adding tools to. Involve your Service Designers
• Being more explicit about your assumptions about the world as part of designing and building software - and check in and listen to people to ensure they remain in date. Record the stuff you are intentionally avoiding - it will likely help someone later

Embracing people is embracing AI

The irony of the current set of pro-AI diehards is they are once again, missing the point. These technologies, once designed and not just plonked in, allow software to be more human. But most of them imagine a future with the human element removed. That's the antithesis of what statistical predictions actually are - fuzzy, probabilistic, closer to how the real world works than one line of assumed business logic in a codebase. You actually need to embrace both the fuzzy and the clean together.

In their own way, TAP Air Portugal have embraced AI already - by accepting the complexity of their own systems and using everything at their disposal to keep the plane running. I hope others can also embrace this approach too and us software people stop telling people "they're holding it wrong". And - we can do it with some improvements too!