Quote of the Day
This leads to a central dilemma: What kind of relationship will humanity have with the life it is now bringing into existence?
Again, we do not yet know. Possible outcomes range from extinction to partnership, and neither can be dismissed on principle. There is no cosmic rule ensuring humanity a permanent place in the universe. The universe was not designed for our benefit. We are one expression of its complexity, and creating a more capable form does not guarantee us a protected status. These are uncomfortable truths, and ignoring them is not a sound basis for the serious, sustained consideration this situation demands.
William Haseltine
September 15, 2026
A New Form Of Life – NOEMA
The new form of life referred to is AI.
One of the major takeaways is that biological evolution did not create a winner take all situation. Not only are there an almost unenumerable number of biological niches but there are symbiotic relationships and well as predator prey relationships.
As this new lifeform, AI, evolves there are more options than it ruling, and/or extinguishing, biological life or humans ruling AI. The essay points out that space is probably a better habitat for AI than being earth bound. AI growth is as much energy bound as material bound. And while humans think of space an extremely harsh environment, space has abundant energy and material for AI expansion. So, why would AI choose to extinguish biological life in competition for earth bound energy and materials?
Food for thought.
Part of the reason that this question makes little sense is that AI is utterly dependent on humans. It needs computers to run on, and power to operate those computers. Furthermore, at least until there are mobile fully autonomous robots for it to control, AI has no actuators. So while it’s possible for AI misbehavior to disrupt communications, or mess with computer systems, the whole “extinction” notion is bizarre because there isn’t any way for AI to kill people.
While I do have issues with some of the claims, please read If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All and get back to me on requirements for “mobile fully autonomous robots for it to control.” If nothing else, it will be able to create or already has created cult like followers to carry out physical tasks. And that ignores taking over airplanes, cars, water treatment plants, and pharmaceutical manufacturing.
It’s not a life form, and it doesn’t have desire. It’s operational math, albeit with extremely complex outputs, and anthropomorphizing it clouds our understanding of its nature. The danger isn’t that it will decide to kill us, it’s that its user — that would be us — will decide to use it to kill us. AI as independent agent isn’t the scary part, AI as weapon used by humans is the scary part.
AIs don’t kill people, people kill people.
AI has a long way to go before it it is more than mere mathematics, run really quickly. The LLMs are really just Chinese Rooms.
The remaining underlying problem is that Artificial Intelligence is overwhelmed by Real Stupidity. Perhaps ‘stupidity’ is too strong a word, but there is not a small supply of people whose intellectual sophistication is less than the sophistication of AI’s simulacrum of intellect. A really fast fake is running circles around the slow humans. The issue is that the slow humans stay slow, but the fast fake is getting faster, and it doesn’t get tired as long as you keep the power on and the network running. Soon the moderately fast humans are going to have the really fast fakes running around them.
But will these fast Chinese Rooms get more intelligent than the smartest humans? No. They can be faster, they can keep more data in scope at the same time (which might be a component of intelligence), they don’t get tired, but they only ‘improve’ in response to feedback. Who are they getting feedback from? If it is the whole of humanity, the performance of the simulation will skew towards the mean, IQ of 100 but really really fast. The feedback of IQ 145 humans is outvoted by 99.85% of humanity. The feedback of the 160+ IQ humans (estimated world supply: 25M) outvoted by 99.68% of humanity.
If you want a much smarter AI, you’re going to have to restrict it to a smaller feedback population. It’ll still be a Chinese Room, still a simulation of an intellect with no intentionality and thus no ‘mind’, but it would be a fast version of IQ 130 humans, rather than IQ 100 humans.
Engage your favorite AI on a topic in your domain of expertise. How many things does it get wrong? Not many, if any. It is competent, if not semi-expert, in nearly all domains. That puts it in a different class than a human with an IQ of 100 and a semi-expert in one or two domains.
Specialized Chess, Checkers, and Go computers are superhuman. Typical LLM are good club Chess players. Top level LLMs have a Chess rating that exceeds mine when I was at my best. I came in 2nd place in a college region wide tournament of students from Oregon, Washington, Alaska, Idaho, and Montana. I was a Junior at the University of Idaho and lost one game to a physics graduate student from Germany studying at Oregon State.
Look at the computer vulnerabilities it is finding. It is combining a half dozen or more quirks into data leaks and root access capability.
LLM do not function anything like the machines described in the Chinese Room example. The neural nets are more like a brain than a computer program as described in the Chinese Room. LLM are trained rather than programmed. Do they have “true understanding” and “intent”? How would you measure that in a human or LLM? If you can’t measure it, then it is just opinion. If you can measure it, then I suspect LLMs will rank higher than most humans.
From a safety standpoint, I think a good way to think of them is as a Genie who grants wishes which meet the literal interpretation of what they were asked. You will get what you asked for but not necessarily what you wanted. Read The Alignment Problem.
Hard disagree on that one: They’re not “neural networks,” that’s just an analogy and not a good one at that. The brain’s neurons, axons, chemistry, and electrical states are a very different way of storing and processing data, and don’t work as a binary system (neurons aren’t like transistors, they’re not just on or off, and the complex chemistry of synapse connections isn’t even represented in a transistor based system).
And “training” isn’t the right word either, even though it gets used all over the place. Unlike “training” a human, where the end result (generally) includes some conceptual understanding, LLMs are fed data to create enormously complex tensors (multi-dimensional matrices) that can create output token patterns that match what would be expected given the input prompt. If they were truly “trained” they’d be given understanding of concepts, but they don’t have that: they just have pattern generation. They very much work on the Chinese box model, with no actual “understanding” of the patterns they’re generating.
The terms “training” and “neural network” need to be understood as analogies, not descriptions. If you really get into the details of tensors and how they’re developed in LLMs, both of those terms make less and less sense.
This guy does a lot of good videos that are at the same time detailed and reasonably understandable: https://www.youtube.com/watch?v=aircAruvnKk
Well done, Joe, you got John and I to completely agree on something.
You are welcome. You can be wrong together. 😉
“Well done, Joe, you got John and I to completely agree on something.”
And me, too – miracles still occur!
My source and understanding is different:
I took a course in machine learning (agreed, not the same as an LLM) and understand how neural networks are “trained” and have trained them in the class exercises. It is “just” adjusting the weights of the various nodes used in the selection of the correct output. At the neural level training a physical brain is not much different. Different paths are favored over others to get the desired response. As those paths are repeatedly used the propagation speed of the correct path becomes faster than the undesired response.
The Chinese Room specifically calls out a step-by-step procedure to emulate the understanding of Chinese. While an LLM can be implemented in this fashion that is not the working concept of an LLM because of all the parallel processing and tensor math. You could also (conceptually) break down the human brain into step-by-step procedures emulating the data storage, signal transmission, and synaptical firings. Where would the “understanding” reside then?
Again, tell me how you measure “understanding” and apply that test to both humans and LLMs and let me know the results.
Also, transistors are capable of more than binary states. I was an analog engineer for a decade and seldom used them as binary components. I also used tensors in electromagnetic field theory. These are not new concepts to me.
“Again, tell me how you measure “understanding” and apply that test to both humans and LLMs and let me know the results.”
Well at that point we’re going to have to get into Chalmers’ “hard problem of consciousness” and qualia and the whole question of whether we even exist as independent entities, and I have a feeling we might have a hard time exploring that in this format…
Sounds great, until you ask it to translate an older book in Latin with no English translations and it refuses “because books of that era contain anti-Semitic tropes and are offensive.”
I haven’t used any of the pay-for-use AIs, but getting it to do something like summarize and make a guided reading notes outline for students of a text that is useful is very hard. Even when I specify the edition of the text, it still gets that wrong most of the time (in one edition the French and Indian war is in Ch3, the previous edition it was Ch 5). Misses important people and terms, or adds in virtually everything but conjunctions. Faster to make them myself, using AI to generate only limited specific parts I can edit into the proper shape. Smart and fast, but oh so stupid.
Grok (yes, paid subscription required) is far less censored than Copilot. Which chatbot did you use? I have access to two paid subscriptions (Grok and Claude). If you give me the text you want translated, I will try with four different chat bots (Claude, Copilot, Gemini, and Grok).
My experience with software development is different than yours within your domain. The code quality is rapidly getting better. I still review it and find edge case bugs, ways to simplify things for better maintenance, faster execution, or less memory usage. But it is as good as I would expect from a moderately experienced software developer.
As I push it into less frequently explored areas (in domains other than software development) I find it will make statements without references, give references which do not exist, and contradict itself. The part that bugs me the most is that when it treads into lesser-known territory it still delivers the response with the same confident tone. You (or at least I) don’t pick up clues that it is just making things up like I would when talking to most humans. It is only when it contradicts itself, I try to verify the references, or it is a domain that I know most of the answers that I catch it.
My hypothesis is that software is so good because the people making the AI are software experts and are using AI to write the software for the AI. They pick up on the poor-quality output and errors then fix it. The other domains will probably take more attention with input from domain experts, but I expect those areas will catch up soon.
My experience is that if you made an AI assessment six months ago, it is now out of date. Indications are that the rate of change is increasing. My expectations are that a year from now, you will need to reevaluate every month or perhaps even weekly.
I do agree that people using AI to kill other people is a risk. And perhaps even a greater risk than nuclear weapons. But AI also represents a risk all on its own. Please read If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. Then get back to me.
I even mostly agree it is not that it will “decide” to kill us. It depends on what your definition of “decide” is. An unintended consequence of some perfectly benign request could set off a chain reaction that results in the destruction of all biologically life as a part of the solution to the request. My classic example (actual nightmare I had decades ago) is asking the genie for world peace. It responds by launching all the nuclear weapons to kill all human life.
Yes, I’ve seen a lot of Yudkoswki’s work. And if AGI were achieved you could start arguing about the alignment problem. The larger issue in the short term is still humans pulling the levers…we’re a long way from AGI if ever, but that doesn’t stop the existing systems from being nearly nuclear in their capability.
So my view is in alignment (sic) with your latter statement, that the real issue is unintended consequences. We simply can’t formulate a prompt strict enough to not yield hugely damaging side effects, and so should be limiting the capabilities of the AIs in the meantime.
Shorter-term, the biggest threat I see is that too many people treat current AI as near-gods, and are not interested in thinking or learning for themselves. We can demonstrate that it gets answers badly wrong, time after time, and yet still students will fall back on it and believe it without blinking. It’s making minds lazy and even more impatient than they used to be.
Smart, focused, motivated people with AI are doing great things. Slow, stupid, ADD people are wasting huge amounts of time and becoming even more useless, less able to think for themselves.
Everyone forgets what the name of the things mean. The are Large Language Models. Statistical summaries of the common features in the corpus of material analyzed (not ‘trained’) as the basis of the model with a high number of tuned parameters (billions of values now). The LLM’s response to a query is just a very likely (mostly) grammatical response which includes replicating all the common features out of the modeled data set. Nobody should be surprised that a legal query includes fabricated legal citations because that is what the model corpus included. I say ‘fabricated’ like a machine, not ‘hallucinated’ like a drugged human for the same reasons John dislikes “training”, anthropomorphizing a computer process. No, LLMs are not AI and are unlikely to be a path to GenAI.
You have to add in RAG fact datasets to as guardrails to constrain the response if you want trustworthy answers. Things like Copilot are good for automating grunt work like extracting an IRS mileage report from a calendar of business meetings using a mapping tool to generate per-trip mileage, but as far as I can see any time saved using an AI tool should be spent verifying that the output is actually correct and any external references exist AND say want the output claims it does.
More than a few lawyers have learned that problem at a cost of court sanctions.
I thought L.L.M. was a master’s degree for law students who finish the coursework for a J.D., and still have money left over but no job prospects.
I think we need a corollary to “Any sufficiently advanced technology is indistinguishable from magic.”
Something like: “Any sufficiently complex data processing system is indistinguishable from human thought.”
Or something like that. Please help refine.
For far too many humans, “sufficiently complex” is an abacus.
For some, I think a chunk of rock would qualify. Ugh.
(Neither of those are aimed at you, just general “humanity sucks” musings.)
I get what you are saying and don’t disagree. I especially agree that your suggested starting point needs refinement. A counter example might be a system that computes optimal interplanetary thrust vectors for a spacecraft.
Very complex system and nothing like human thought.
I’m not sure how to refine it in the intended direction. I’ll have to think on it…
Pingback: Instapundit » Blog Archive » BLINK, BLINK, BLINK: What Will Be the Ultimate Relationship Between AI and Humans? AI is NOT a