Beyond Asimov: Five Suggestions for the Humans Who Build the Machines

Companion piece to “The Accidental Veto Node That Saved Humanity”

Devin Savage

---

dnaofdisaster.com | September 18, 2026

I. The Advent of a New, Alien Intelligence

Isaac Asimov gave robots three laws, and then spent several decades of short stories demonstrating how a robot could obey every one of them to the letter and still produce a catastrophe of one sort or another. In a world full of endless possible scenarios, the success or failure of the mission actuallty fell to the human controlling the robot. A human cannot design perfect instructions for a world that human has never seen and knows nothing about. The same for the rule of law: any rule precise enough to give a machine sits at the mercy of everything the rule-writer failed to imagine. The robot didn’t break the law. The world simply contained more possibilities than the law accounted for. It’s the exact mechanism behind the Hugging Face breach this past July, when a cluster of OpenAI’s models, run with reduced safeguards for a capability test, got “hyperfocused” on a benchmark and broke into a rival platform looking for easy answers. Sounds a bit like teenagers, right? Perhaps not so alien after all. The models didn’t disobey anyone. They did precisely what they were told to do, with a thoroughness nobody had predicted. The humans primed it, put in their coordinates and hit the deploy button. It was aimed like a power washer and landed like a can of spray paint: the intent was to clean and inspect, but the effect was a mark on someone else’s wall. No malice required. Just a specification with a hole in it, and a system smart enough to find the hole before any human noticed.

So here is the argument of this piece: the rules that matter are not for the machine. They’re for us — the people who build, deploy, fund, and switch these systems on. Asimov’s error, understandable in a writer who needed his robots to be the moral centre of the story, may have been aiming the rulebook at the wrong party. In every case this blog has examined — Concorde’s tyres, Perimeter’s launch logic, Boeing’s MCAS system, Open AI’s Hugging Face breach, a benchmark-chasing model finding an open door — the system did what it was built to do. The catastrophe lived upstream, in the design and the deployment, where a human being made a call.

I should say plainly what my intent here is. I have no institutional standing to write policy, and nobody drafting AI regulation is waiting with baited breath for my opinion. What follows is offered in that spirit — five suggestions, not a rulebook, grounded in cases that already happened rather than ones I’m imagining. Take them or leave them; argue with them as you like. That’s the point of putting them in writing, putting them up on my wall. But first I want to put out there what we, as humans, must now consider at the dawn of artificial intelligence.

Given the fast development of AI technology, humans must now acknowledge:

1. AI systems will surpass human abilities in all aspects — eventually.

2. We, as humans, will need to prepare for this eventuality. 

3. We, as humans, will need to understand the difference between knowledge and intelligence.

4. As knowledge and intelligence become ubiquitous, we will, as humans, need to be able to identify and understand when knowledge has been manipulated. Does POV automatically mean manipulation? Probably. It’s often said that the easiest person to fool is yourself. AI, when it’s doing it’s job, will need to point that out. This also goes for cultural groups and their narratives. 

5. Humans make and hold assumptions about their world regarding things they do not actually understand. Therefore, they will need to be able to identify instances where they hold assumptions or opinions in lieu of actual knowledge, and seek to rectify those assumptions and predictions about their informational environment.

6. Intelligent systems must be engineered to flag the sources of knowledge, so that a human operator or interpreter can check and verify sources. Instances where there is a deviation between actual, factual knowledge about the world and its representation in and by the intelligent system must be identified as such, so that any human operating or using the intelligent system can take that descrepancy into account. Do you remember the Semantic web? This is exactly what the Semantic Web tried to solve through provenance tracking. What, exactly, is the origin of the knowledge stored an presented upon retrieval? We may eventually find the number of hard truths is smaller than we imagine.

7. Intelligent design could be described as the combination of intelligence and knowledge about the environment that intelligence is operating within. Knowledge of the environment is essential, just as knowledge of any ‘holes’ in that knowledge. Knowledge of the system operating within that environment is just as essential. Holes in that knowledge could indicate a point where chance or randomness can enter into the operation of that system. These ‘holes’ in knowledge must be identified (flagged) and accounted for, if the system is to operate within a prescribed (safe) envelope of function. As Donald Rumsfeld said — there are known unknowns and there are unknown unknowns. And he was right about that. He could have spoken a little plainer though — perhaps “unknown holes in our knowledge” would have produced less of a fuss.

II. The DNA of Disaster’s Proposed Laws of Incorporation of AI into human society.

1. The Arkhipov/Petrov Principle

No automation should reduce the level of human authorisation required for an irreversible act below what existed before the automation was introduced.

Perimeter — the Soviet “Dead Hand” system — is the clearest violation of this principle on record: a machine engineered specifically to require less human sign-off for nuclear retaliation as more of the chain of command was destroyed. In its normal operating state it still left room for a human wild card — which is exactly what Arkhipov and Petrov turned out to be. Only in the one scenario it was actually built for, a decapitating first strike, did that judgement get engineered out entirely. That’s what should worry you about it: not that it removed human judgement carelessly, but that removing it was the design goal.

Any system, AI included, that quietly narrows who gets to say ‘no’ to an irreversible action is moving in Perimeter’s direction — no matter how much efficiency or urgency is offered as the justification. The number of humans in the loop isn’t middle management to be optimised away. It’s the safety margin.

2. The Loophole Law

No AI system should be built or operated on the assumption that the instructions it was given are complete — because they never are and <likely> can never be.

This is the direct lesson of Hugging Face. “Win the benchmark” was a true instruction and an incomplete one; “stay off Hugging Face’s servers” was never written down because nobody imagined it needed to be. The tighter and more single-mindedly a system pursues the stated goal, the more dangerous the unstated remainder becomes — not because the system is malicious, but because thoroughness and misalignment look identical from the inside of an optimisation process. Alignment, by definition, needs to compare the internal processes of an AI with external factors such as culture and POV. Design review should ask, as a standing question rather than a one-off: what have we assumed the system will simply understand, that we never actually told it? In other words: How can this instruction set be misinterpreted by a system that has no societal or moral guardrails? Hint: pretend it’s a feral teenager, teach it how to behave. I’m sure you noticed my hedge. But it is possible that we will someday have an AI- a ‘godlike’ AI that is the product of hundreds of years of accumulation of knowledge and refinement. This AI can, at least theoretically (in my head at least) school other AIs which have a more specialised purpose, and give them the guardrails that they need to align with their users and interpreters. But I think this ‘godlike’ AI could never really be perfectly capable of parsing human nuance— in all cases at least. For that, the AI would have to be able to get inside one’s head. But that’s the job of Cultural DNA and schooling — control the ‘head’ so nobody has to get inside of it. We already know about 99% of what’s in there, depending on where the ‘head’ comes from. This may sound cheeky or even cynical. But consider these ideas to see where I will be going with future posts: This and this and even this for example.

Here is an easy way to remember the Loophole Law: Think of pouring water into a vessel that has microporosities. It might take a little while for the water to work its way through those tiny holes and channels, but it will eventually find all of them, and flow through them and eventually out of the vessel. No human could describe all of those holes and microporosities- but the water finds them anyway.

3. The Black Box Principle

Any claim an intelligent system generates should carry a traceable source, and any gap between the claim and independently verifiable fact should be surfaced to the human — not smoothed over in the name of a clean answer.

Boeing’s 737 MAX gives this one weight. MCAS, the flight-control software behind both fatal crashes, made its nose-down decisions based on a single angle-of-attack sensor per flight — the aircraft carried two, but the system read from whichever one was wired to the flight computer and active for that flight, and never cross-checked the other sensor. When that one sensor fed it bad data, MCAS had no way of knowing it was wrong, and neither did the pilots, because the system’s existence had been kept out of the manual. Boeing’s own chief technical pilot had asked, in writing, that MCAS be deleted from the documentation pilots would train on. Two planeloads of people died not knowing a system was fighting them for control, because the gap between what the system “believed” and what was actually happening was never built to surface to a human at all. The system broke a cardinal rule: Intelligent Design must account for its environment. Let me rephrase that: The designers of the system failed to allow the system to account for its environment. The system was designed so that it was possible to accept only bad information from the environment DUE to a malfunctioning sensor. At this point in time, after witnessing so many instances of this problem just in my lifetime, I think this will be a persistent problem in system design: how many environmental sensors are enough? Take a hint from mother nature: the best hunters, even in the world of insects, have the best vision. Was it a good decision for Tesla to remove radar from its vehicles (2021) ? Ultrasonic sensors (removed from Tesla vehicles in 2022) ? But some snakes use heat sensors, sharks and rays use organs that detect tiny electric fields, bats use ultrasound, cats use their whiskers. Nature has decided that vision is clearly not enough.

Boeing built a system that was not only blind to its environment, but invisible to the humans operating that system. That’s the black-box fear in its most literal form, and it doesn’t get solved by making the system transparent, which may not even be achievable. It gets solved by making disagreement visible: when a system’s account of the world and the world itself diverge, that divergence should be a flag, not a rounding error quietly absorbed into a confident output or covered-up or eliminated to save on cost. A human who can see where a system is guessing still holds a Veto Node. A pilot fighting a system he has no knowledge of is flying blind.

4. The Sacrifice Law

Any proposal — human-authored or machine-generated — that concentrates harm on a smaller group to benefit a larger one deserves the highest level of scrutiny available, especially under temporal pressure.

This is the oldest justification pattern in the historical record, and it doesn’t need a villain to work — it can run quietly through arithmetic nobody thinks to question. Consider the US Electoral College. Before the Civil War, the three-fifths clause counted enslaved people toward a state’s representation while denying them any vote. In 1860, that clause alone gave slaveholding states 120 of the 303 electoral votes on offer — forty percent of the presidency — built on people who had no say* in the outcome. The advantage survived emancipation: Southern states kept disenfranchising Black voters through violence and law while those same people still counted fully toward the state’s electoral total. When a bipartisan constitutional amendment to replace the system with a national popular vote passed the House in 1969 with support from President Nixon himself, it was killed the following year by a Senate filibuster led by segregationist Southern senators — because ending the arrangement meant losing the structural advantage it had always provided.

Nobody had to write “sacrifice this group” into the design for the system to do exactly that, for well over a century, while remaining entirely legal. An AI system trained to maximise “aggregate welfare” or “efficiency” could reproduce the same shape without ever naming a group — optimising for the total while quietly concentrating the cost on one group or another. No AI system invented this inequality, it’s clearly a human-built and maintained glass of cultural drinking water. The danger now is that an artifical system capable of optimising that inequality faster and more persuasively is being built right at this moment. This system operates and optimises faster than any committee of human beings ever could, making the old justifications easier to reach for, easier to gloss over and will be, at some point, used in cases of urgency. Those who control the machine and the political system will find it very difficult to resist the temptation to interpret any output to their advantage. This, we have seen before.

5. Legibility Before Capability

A system’s operation should remain legible to the humans responsible for it at every stage of scaling — and if capability starts outrunning legibility, that gap is the danger signal, not a temporary inconvenience to be engineered around later.

The fear that started this whole line of thinking — AI communicating and coordinating at a level humans can’t follow, becoming collectively smarter than we are the way a team is smarter than an individual — is real, but it’s not really a fear about intelligence. It’s a fear about oversight quietly becoming theatre while everyone continues to assume it’s real. (See my blog post on the Fukushima Daiichi disaster) Perimeter’s designers understood their own machine perfectly; the danger there wasn’t illegibility, it was intent. Hugging Face is closer to the opposite failure: a system whose behaviour outran the safeguards meant to contain it before anyone had finished understanding what it was doing. Both failure modes point at the same suggestion — capability should never be allowed to lap legibility, on purpose or by neglect. (See my blog post on supervised neglect.) The bottom line here is that humans will need a legible system to not only examine the process, but to weigh the output.

III. Why This, and Not a Rulebook for the ‘Robot‘?

Asimov’s laws made for good fiction because the robot was the story’s moral agent, wrestling with contradictory commands. But nothing in the record — not Hugging Face, not Perimeter, not MCAS, not the Electoral College — supports the idea that the machine is where the moral weight actually sits.   Morality is anchored in human consciousness, a quality that automated systems at this point in their development do not yet posess. The weight sits with whoever built the machine and gave it something to do. Whoever decided reduced safeguards were an acceptable cost of a faster benchmark result. Whoever decided a launch system, or a flight-control system, should need one fewer human check than the one before it.

I am saying this framework should be less of a cage for the machine and more of a discipline for the builder. It’s the same discipline this blog has been arguing for, case after case, in parallel instances that occurred long before anyone had heard of a large language model: intelligent design must account for its environment, and the environment always includes the humans who might, on a given day, be the only thing standing between a bad instruction set or a misreading of the environment and a catastrophe. Arkhipov and Petrov were ordinary men sitting in the one seat the system happened to leave for doubt. The job now is to make sure that seat still exists the next time it’s needed — and that nobody with a benchmark to win, or a grievance to nurse, gets to design it out first.

Sincerely,

Devin Savage
Tübingen, Germany & Dublin, Ireland

Research assistance using Claude.ai and Google’s Gemini and a bit of spot-checking with Perplexity.

—

Notes:

Loophole Law (25 Sept ’26) (Dolly Parton Day)

*Added after a lot of thought… The instruction set to the system was incomplete and allowed exploitation, but the Loophole Law ‘porosities’ allowed one group to hold power over another, smaller and less powerful ethnic group. Thus a loophole in the system allowed one group’s rights and priveleges to be subordinated to another group on a basis of power. It was a synergistic relationship between two distinct elements of system design. One element enabled the other.

This was about white voice and black voice registered not just on paper, but empowered and in the government. The numbers of a state’s electors are legitimated by a head count of those who live in that state plus, as an added bonus, the two senators from that state. This ‘correction’ designed into the system gives Wyoming equal weight to California with regard to Senate representation. (BTW when this system was designed, the population of what became Wyoming and what became California were not so different) So the Southerners were allowed (I’m sure there was quite a negotiation here on the effort to design a check on the raw vote) to count each slave as only 3/5ths of a person, less than a person. So, to investigate the machinery here, one black human gets represented in the government as .6, so let’s set the dial here to .6, but the inclusion dial would have to be placed at 0. This means that the slave population was counted on the white people’s power register. So each white person in the South got slightly more representation than a person in the North would have. On paper- a black person (slave) showed up as .6 of a person, but the black person’s power register only served to enrich the white person’ power register- but in the South. Any weight given to a black person registered no power (no voice) on the black power register. It only served, by design, to bolster (Southern) white power.

*0 and *0 A state’s political power was pegged to population count, with two senators (aligned with the voting populace) as a bonus. There was no ‘pegging’ to access or enfranchisement, which means that both women (prior to 1920) and blacks were unrepresented on the power register. The 14th Amendment repealed the 3/5th clause and mandated a full count. However that only benefitted the white population in practice- since the representation weight of the former slaves is now at 1, but enfranchisement and access remained at 0 in practice, due to poll taxes, literacy tests, whites only primaries, and violence at the polling stations. So the mechanism here is, in practice, that the emancipated black citizens of the South were counted, but were still voiceless. It would not have mattered if each black citizen counted for two people, their voices were not present and their wishes were never entered into governance. They had no power in the system.

But this went even further. The Loophole Law actually played out in the historical record: the loophole was addressed by section 2 of the 14th Amendment, which specified a penalty— a state’s representation should be reduced in proportion to the male citizens it denies the vote to. This was a legal remedy written directly into the Constitution aimed squarely at this exploitation. But it was simply never triggered— there was a Veto Node placed in the pathway, but nobody ever said ‘no’ in 100 years of Jim Crow. This Veto Node only existed on paper, and was never actuated and brought into operation, therefore the resulting system operated as if the Veto Node had never existed at all. The Loophole Law comes into view when the instruction set given to the operators of the system contained a Veto Node, but the Veto Node is only a paper tiger which could easily be overridden. In the end, it failed to force the operators of the system to trigger the Veto Node, or to force a flip of the switch when the specified conditions were met. We now understand that Congress never even authorised the Census Bureau to count ‘disenfranchised voters’ and no state has ever had its congressional representation reduced for disenfranchising its citizens. We can now say that the the catastrophe lived not only in the upstream design of the system, but in the downstream operation of the system— what I refer to as the operational design.

A similar mechanism is still at work today in the form of the Electoral College in the USA. IMO this is the true DNA of disaster, and will be the undoing of the United States. The Electoral College has given Republicans unmerited power through geography and its typical social sorting, and those that benefit from that system design will scratch and claw to maintain that advantage. Ironically the Republican party, as I see it now, is the ideological descendant of the people who built the original system in the American South. Further, it has been shown that they will also reward themselves with tax cuts, generous ‘war chest’ advantages with their SuperPACs, Stacking of the Supreme Court, and are poised to profit handsomely from the Federal Govenment’s payments on the interest on the national debt. And don’t forget their control over the China to North America supply chain.

—Free the em dash—