Today, after completing this course, I came to the conclusion a friend at Mandiant was correct.
Several months ago, my Mandiant friend told me that AI is making things better for cyber attackers than cyber defenders. It was his belief that it increased the defenders attack surface more than it enabled us to rapidly defend ourselves. I wasn’t totally convinced until today.
There are just so many ways that AI’s can be subverted. And to make things worse our management is literally saying, “Everyone should be programming now.”
A few weeks back, I was helping a senior manager, who only wrote a few hundred lines of code a couple of decades ago get through the V.P. required coding exercise for all managers. She did okay. She didn’t really like doing it, but she had a Python program that accomplished the task she wanted to do. Python dominates in the security field. It has its advantages, but if you want performance and stable programs that are moderately large, you are much better off with a statically typed language rather than the dynamic typed Python.
To the naive it may seem that if you can get some code to do what you want it then you are done. Ahhh… no. There are reasons why software developers have version control systems, development, staging, and production systems, unit tests, test driven development, stress tests, code reviews, static and dynamic analysis tools, mutexes, semaphores, locks, locking objects, profilers, and interrupt driven drivers. Management is not going to be able to save money by laying off all the senior developers and writing their own code or even turning the development over to developers with a year or two of experience. I have heard it said that every item on a pilot check list was paid for by the death of one or more people who learned the hard way why that item was important. A similar thing is true with software development. And even if you tell your favorite AI to write safe code and to review it. There will be “issues” that may be accidents, or an agent gone rogue. But there will be issues that people will pay a high price for.
After finishing the course, I had a discussion with Chat GPT on this and related topics. During the course of the discussion it told me a joke that rang just a little too true:
As an old C programmer, you may appreciate another version of the joke:
Dynamic languages let you shoot yourself in the foot faster.
AI agents let you automate the shooting.
One of the more obvious (to me) problems is that by laying off the junior devs and using senior devs with AI-boosted productivity, they have nuked the pipeline that turns young programmers into senior devs over a number of years of experience.
Suddenly the plot device used in so many S/F stories of “they forgot how to do things” doesn’t sound so unlikely.
Are you seeing any evidence in your workplace they are thinking very far past the next quarterly or annual reports, WRT people development?
The difference between coding and a pilots checklist is people don’t generally die from a coding error…lose money, yes…but not die. And to bean counters the bottom line drives literally EVERYTHING. So, if you can live without computers and the associated problems you are better off doing so as much as possible.
“don’t die from a coding error”. True for unimportant stuff, of which there is quite a lot. But there is also coding that changes whether a business keeps going. And coding that controls your finances. Not to mention that there is the coding in your car, parts of which are safety critical. Or the coding in your pacemaker.
As for AI and reducing programmers, another possibility is that management will get rid of the experienced programmers and then pair the novices with AI. Then we’ll have bad code without the expertise to spot it.
This is what I am seeing hints of. It is definitely happening in some places. I don’t know how widespread it will be.
Some of my software was used in an air traffic control simulator. Extrapolating that to actual air traffic control or airplane instrumentation made me realize just how important software quality control could be.
The way to get to bean counters is risk analysis. They know how to take that into account (or at least, they attempt to do so).
But nailing down the risk level on this stuff is really, really hard because it is changing so rapidly (among other reasons).
The bean counters who do (or try to do) risk analysis are the actuaries who work for insurance companies.
There is a reason that the men who cut the trees down and then cut them into lumber have the highest insurance premiums in America.
As for lessons and warnings paid for in blood and in lives; it is said that every stupid product warning label is the result of a lawsuit for an injury or death, and in that lawsuit lawyers argue about what consumers should be reasonably expected to already know, and the judge was hired for less than the lawyers to decide what the parties would have decided had they thought about the issue beforehand.
Oh, I don’t mean “do” risk analysis, like actuaries, I mean that referencing risk analysis and more importantly the INSURANCE once can buy to deal with those risks (along with certain other actions that then reduce the cost of that insurance) can get the bean counters’ attention.
There used to be a site – overlawyered.com – that has shut down, last I knew, sadly. It chronicled that kind of stuff, not just the warning labels, all of those kinds of crazy, stupid things. There also used be a yearly warning label competition that I always saw via overlawyered, but it wasn’t run by them…. who was it that did that… I think it was some law school. Can’t remember.
Fun stuff! Well, from an appropriate distance (say, a hardened bunker in the next solar system over or something).
We keep learning lessons, forgetting them, and re-learning them.
Weinberg’s Law :
If builders built buildings the way the programmers wrote programs, the first woodpecker that came along would destroy civilization.
My last big project before retiring (i.e. being laid off with a package a month before I planned to quit) involved building an automated process to stand up and provision critical servers when one got sick and died. I couldn’t have done it in the time available without AI assistance, but I had to watch that AI like a hawk because it kept trying to declare victory and quit.
Weinberg made his observation around 1975, only five years into the Unix Epoch. The sci-fi “they forgot” trope comes more true every day.
There is such a thing as safety critical software. Check out the large air transport regulations (part 25) some time. The industry ended up developing a number of standards to minimize design errors in the airplanes. RTCA/DO-178B is what I used for a large number of years later updated to DO-178C before I retired several years back. All the things Joe mentioned are codified in that standard (FYI – Python is NOT allowed to be used for the highest safety levels). Depending on how safety critical the software was (as determined by safety assessment with its own standards and rigor requirements) you would have to use different levels of independence between the different task roles he mentioned. For those that were truly paranoid about how unknown hardware failures (including design errors) they required software to run on different versions of processors and compare results in real time. Look up 777 Flight controls some time – I think there is some public domain stuff describing the triple lane design each lane running on a different type of processor.
One aspect of D0-178 that goes a bit beyond what Joe discussed was that you not only had to be able to show that the software did what it was supposed to do under all forseeable conditions, but that it was documented with sufficient detail and clarity that it could be modified later without breaking something outside of the area of change. If you couldn’t do that you ended up having to rerun your entire suite of tests all over again which could take months and might sometimes roll up into airplane level testing especially if you changed something that interacted with other systems using your system’s output.
Before I left, they were struggling with how to standardize an approach that would allow developers to use model based practices that would achieve similar levels of rigor. I am quite certain that AI used in development would be very hard for them to accomodate given the need for being able to trace every design product (requirements, design, code and tests) back to the system requirements they derived from.
X-ray machines, hospital pharmacy pill dispensing software, railroad scheduling/monitoring software, PSAP logging/monitoring software – the list of safety critical software is endless.
Kurt
Pingback: Instapundit » Blog Archive » A WORD PROCESSOR DOESN’T MAKE YOU A WRITER, A GRAPHICS PROGRAM DOESN’T MAKE YOU AN ARTIST: AI Age