There’s no Such Thing as Immutable Code

Posted on by

Quote of the Day

… when people talk about “hardwiring” ethics into AI, it’s often more aspirational than technical. The harsh truth is: there’s no such thing as immutable code—only code that hasn’t yet met its adversary.

Microsoft Copilot
June 19, 2025
As posted in Think Fast on June 21, 2025

Copilot, thank you for the warning.

Humans, prepare appropriately.

Share

7 thoughts on “There’s no Such Thing as Immutable Code”

  1. Mostly true, though you can put code in ROM and that is done occasionally.

    The case of AI is a good example of what’s said here, though, because AI is mostly not code at all. Instead, it’s mostly training data. In other words, you have a (comparatively) small body of code whose actions are controlled by data generated in the “learning” process. That “learning” is an example of intentional self-modifying code, with the additional painful issue that the nature of the modifications is not really predictable, nor is it fully known to the supposed authors of the software. In other words, while “ordinary” software has — at least in theory — knowable properties, that is never the case for AI and that deficiency is intentional.

    • “though you can put code in ROM and that is done occasionally.”

      My impression is, it was quite prevalent when RAM was a lot more expensive, and code was much simpler (and not subject to the incessant upgrade cycle).

      • A current example is the RP2040 chip, a small ARM system-on-chip that’s the core of the Raspberry Pico microcontroller. Its flash writing code is in ROM.

    • In the early years of the P.C. that was true of both system BIOS and peripheral cards. But by at least the early 2000s, and probably several years before, it all switched to flash memory.

      About 2004, when working on a contract for a certain government agency, I created a demo of extremely persistent malware by adding less than 256 bytes to some empty space in the BIOS of a network card. And I did this to a remote machine over the network…

      Even in the old days ROM if you could get a “toe hold” you could hijack the execution of the ROM program by changing the return address of a function call to point at code you had placed into RAM. The classic example to do this would be via a buffer overflow from unchecked input from an external source.

  2. Nothing created by man is inherently evil…or good. It’s always neutral. Waiting for implementation by humans…and perhaps eventually Artificial Intelligence. TV and the internet both had HUGE potential for enlightening and advancing humanity. Both were subverted, perverted and twisted to make clever people rich while deluding and misleading anyone gullible. The coming technological tsunami will be no different. It simply is. How it’s used will determine how it effects humanity and the future.

  3. Great point—AI ethics as a hard-coded feature feels more like a comfort blanket than a technical reality. The real challenge is staying ahead of the adversaries, since even the best code can become a liability in the wrong context.

    • I think “AI Ethics” is a SF concept not found (or to be found in the foreseeable future) in the real world.

Comments are closed.