Large Language Mode(ration)

Let's say you are a nurse, or doctor.

You undergo training, either in classroom or on the job shadowing. You sign up to a professional organisation to say you are licensed to do nursing or medical care.

If something goes wrong, and it's been found to be based on your neglect, you can be "struck off".

Let's say you are a nuclear engineer, or a civil engineer.

You undergo training, either in classroom or on the job shadowing. You sign up to a professional organisation to say you are licensed to do those things.

Again, if something goes wrong, and it's been found to be based on your neglect, you can be "struck off".

Let's say you are a social worker. Same deal.

The catch

But let's flip this. Instead of being a human deciding, you are writing software to inform or recommend one of these people to do or not do something.

That person writing the code... doesn't have to undergo the same checks as the humans doing the work.

The current workaround for software is either:

1. regulation - broadly that means someone who is accountable needs to check the functionality and risk assess what has been made
2. well intentioned individuals and organisations involving these professions heavily in their software research, design and production
3. FAFO*

Unfortunately, most software these days falls into 3. I've been lucky enough to work at places that do some combination of 1 and 2.

The even newer catch - AI

Up until fairly recently, we assumed software was deterministic. That is, someone writes some instructions in code, and the computer will follow those instructions every time, and behaviour outside that is considered a bug.

We had just about got the hang of this for some deterministic software - there are lots of medical experts involved in medical software, for example. Banks (outside the US) let you transfer money and it broadly does arrive in the place you sent it, and if it doesn't, you have financial recourse to get it back. Automatic railway signalling does mostly send trains where they're meant to go. Etc.

But this all assumes all output of said software is:

• repeatable
• predictable(**)

These are much more difficult to determine without these, which happens with technologies like Machine Learning or LLMs (ie, those things people now just nickname AI now, although it's a broad umbrella). Both of these rely on statistical analysis to tell the software what to do next rather than a set of pre-supplied instructions.

The good news

Other professions have to handle a lack of 100% repeatability in the real world already. Medical trials don't have a 100% success rate. Mechanical engineers consider combinations of tolerances and have built in statistic cut offs into their everyday working assumptions.

So if we're super determined to find a real home for AI in this world:

• Stop only asking the AI people, or software developers
• Ask everyone else how they currently handle uncertainty or unpredictability

So maybe the future of AI isn’t in Silicon Valley. Maybe it’s in centuries-old professions that already know how to balance risk, uncertainty, and accountability.

You may find they already worked it out in the 13th Century.

(*) Effed Around and Found Out, if you've not seen this acroynm before

(**) There is a broader question of how predictable this is, but that is a very big topic outside the scope of this small post: https://xkcd.com/2347/