As a preface to this article, I’d like to state the obvious: we cannot un-ring that bell. Great minds have been theorizing the emergence of this tech for decades1, and they have been proven right.
There’s no going back — even if we wanted to. The genie has been freed, and there’s no stuffing it back in the lamp.
And while it tends to fulfill wishes “monkey paw”-style, even skeptics have to admit that it has its uses.
A couple arguments against the genie
A computer cannot be held liable
A wrong recommendation, delivered through a screen with sufficient authority (or the appearance of) can have devastating effects. Here are a couple examples:
- Generalized surveillance of the public, and presumption of guilt 2
- A woman was wrongfully jailed for 6 months due to an error in facial recognition software 3 4
- A lawyer, caught using “hallucinated” cases in court 5
Who goes to jail when things go wrong? In the case of licensed professionals, their ability to work is directly on the line, but when it comes to law enforcement, officers seldom face such harsh consequences for their actions.
If the AI mistakenly identifies you as a criminal, all the data it retained will be used to charge you, but whatever it discarded - even if it was exculpatory - will never surface. A computer will mark you for jail and a human will haul you there — possibly under the threat of lethal force.
Who gets punished then? The Law Enforcement Officers who looked no further than the brief they received? The software company’s C-suite? The programmers working for them? The ML engineers who trained the models?
And how will such mistakes be prevented in the future?
Fool me once
This is more of an emergent observation, but the conversational approach to LLMs changes the way we process the output. If you ask a colleague to explain something to you, you (usually) do so because you think you can trust the answer you will receive.
Trust that it is truthful, at least to extent to which your colleague is knowledgeable.
Trust that the explanations you get come with appropriate caveats, and that the more you discuss the topic, the better your needs are understood.
Trust that if you possess some knowledge or experience that contradicts what you are being explained, it can be re-contextualized to help you understand it better — or that your colleague may learn something from the exchange.
This trust exists in part out of respect for the knowledge and proficiency of others, but also because we value our own reputation — and we expect others do the same. We don’t like liars and cheats, and if we spot one, we actively avoid working with (let alone trusting) them.
Circling back the first point, what does an LLM care about its reputation? It does not, it cannot — no matter how hard we anthropomorphise them. That’s not to say they only produce garbage, but when they do, they face no consequences – for they cannot.
But someone needs to be blamed. So instead of wasting our efforts sternly reprimanding the computer, we blame ourselves. And we somewhat gaslight ourselves, the same way a Ponzi-scheme victim gets told by the scammer thay they “didn’t apply the method correctly” – else it would have worked. We tell ourselves didn’t prompt right, we didn’t provide enough context. In short the tool that was not defective, we simply used it wrong.
Of course we may also blame the company that provided the model, and it may not like the potential loss in market share. But so far it’s essentially been a giant game of “Musical Chairs”, with no real eliminations.
In short, we cannot trust LLMs because their is virtually no consequence for them randomly breaking that trust. Maybe if we stopped saying “trust” and instead started calling this “faith”, we might begin to understand that it is misplaced.
Teach a man how to fish
If we ask our colleague for an explanation, and he responds with an annoyed grunt and a PR (Pull Request), we may have a solution to our immediate problem, but we will not be better equipped to deal with the next.
If we kept asking vague questions and changing the topic halfway through all the time, our colleagues would soon stop responding altogether — at least until we put some effort into understanding what it is we are asking.
This is the other side of the previous coin: Aside from a loss of time (and associated API costs) there is essentially no negative consequence to us being hand-wavy and relying on the models to provide the solution. It does not immediately punish intellectual laziness. It will keep engaging, because it does not have anything else to do. It might even provide the solution we need eventually, but will we have learned anything?
A seasoned and curious professional may avoid this pitfall, but junior colleagues, faced with the pressure of delivering something — anything, as long as it counts towards “progress” — may rob themselves of a learning opportunity.
This opportunity might come back around in the guise of defects to be fixed — but by then they might be back to square one.
Experience allows us to grasp the fundamentals of ready-made solutions and apply them while remaining vigilant about their potential trade-offs. That is because we have already been through the learning process, we have already experimented with variations, and come to favor some aspects over others.
What if we get into the habit of looking at stuff instead of actually doing it? How long until we lose touch with the larger scope of the problem? With the nitty-gritty detail of our technological stack?
Those who can’t do, teach.
To be clear, I have the utmost respect for a number of my former teachers, for a variety of reasons. When it comes to IT though, there is no doubt that it was those with the most hands-on experience who taught me the most.
My ability to understand was hard-earned. So was my ability to do. But they both lose some of their edge if I don’t practise regularly. This is true for any discipline.
I do not want to jeopardize them by turning myself into a “teacher” to an oblivious, undisciplined, unruly something that is neither student nor peer.
Is it really intelligence?
Probability and the illusion of intelligence
A lot of the early progress was made by making LLMs bigger - essentially increasing the number of parameters. Here’s a look at OpenAI’s series of “GPT” model” models, for which we have readily available data6:
| Model | Year | Parameters |
|---|---|---|
| GPT-1 | 2018 | 117 M |
| GPT-2 | 2019 | 1.9 B |
| GPT-3 | 2020 | Up to 175B |
| GPT-4 | 2024 | Estimated 1T |
The sheer number of parameters is not the only thing that determines the model’s output. Training methods, the quality of training data, fine-tuning the LLM for a specific domain, “Mixture of Experts” (MoE) approach… All contribute to improving the quality of the output.
RAG (“Retrieval Augmented Generation”) lets the model query data on the fly and enrich the response accordingly, with the downside of opening it up to injections, if it fetches data from a poisoned source.
A Ralph loop (Claude.com) will help, by repeatedly “beating” the models back on track, and in turn help with the results. But these processes don’t come cheap (more on that later).
Overall, when qualifying the output, I guess I would still be among those who say “it’s a Stochastic Parrot”.
But even I have to admit that the parrot is getting pretty damn eloquent.
Is there really a difference?
Another angle hinted at above, is this: for all practical purposes, is there really a difference between “actual” human intelligence and a powerful enough tool?
“Any sufficiently advanced technology is indistinguishable from magic.”
– Arthur C. Clarke’s 3rd law (Wikipedia)
If given the same instructions, an AI accomplishes a task as reliably as a human, there’s little to do but accept that the task has indeed been “solved”.
Provided we make the models powerful enough, could be get into a Library of Babel situation, where every program is essentially just a matter entering the “correct seed” to the appropriate “statistically correct” AST (Abstract Syntax Tree)?
This begs the question of how we define a “task” and what it means for it to be “solved” or “done”.
- How complex is the task?
- Is our way of doing it efficient?
- Can it be improved?
- Will we get the necessary insights to figure out the improvements?
- How much can one deviate from the instructions?
- Under what conditions?
- With what consequences?
- What feedback would be expected from a human employee?
- How to react in the face of an unexpected situation?
- Can an AI be relied upon to hit the “emergency stop” and prevent a disaster?
- How do we translate a “gut feeling” into a set of instructions?
- Given the “seed” for a correct program, if we need to perform changes, how fare away are we from the next seed?
The point I am trying to make here is that while I believe these issues can be addressed to some degree, the work ahead is non-trivial.
But I am adamant: yes, there is a difference, for better or worse. And that difference feeds directly into the trust and liability concerns outlined above.
The ever-hungry beast inside the (black) box
We are used to dealing with black boxes all the time. Our computers, phones, cars, fridges and ovens are black boxes. Until something breaks, then we take a peek and fix what we can — that is, if the manufacturer is not actively trying to prevent us from doing so7.
If we run models locally, they become less obscure: We install the necessary GPU drivers, a tool such as Ollama or llama.cpp, download the weights from Hugging Face and off we go. There are a myriad of resources out there to learn more about their inner workings, how to tune them…
…And then we realize that they suck.
It’s fun, sure, but you won’t get anything serious done by using free models on your own hardware. And the better ones out there barely fit on regular consumer hardware anyway.
But efficient (read: paid) models are a black box, with an insidious characteristic: we interact with them through our browser or through APIs. And in doing so, we don’t realize their actual scale.
- Meta wants to build a data center roughly the size of Manhattan (NoteBookCheck.net)
- The DRAM prices have increased up to 5-fold, and gone down to “only"" ~3-fold since September 2025.8
- Lawsuits are in progress regarding the theft / “fair use” of virtually everything ever produced to train the models9
These paid models are / were being sold at a loss: unlimited $200 plans for Claude could cost several thousands of dollars of compute to the company — and restrictions are coming to how / when such plans can be used.
Side note about the FSF
If you have never heard about “Free Software”, here are a couple starting points:
Before dismissing their militant effort as them being “digital hippies”, consider how much we owe them for the current state of software. If you think “but it’s terrible!” know that it would undoubtedly be worse.
The chardet case presents a very complex question on what is “free” software - and what the GPL ought to protect.
Scaling, continued growth and power dynamics
Does it keep scaling linearly?
Will more compute always mean better outcomes? We already see issues related to “context rot” whereby the short-term memory of the model becomes so full that it starts losing track of the task at hand.
There’s little doubt in my mind that someone will come up with a way of structuring the context so that the most important instructions survive forever, and always get the highest priority — preferably with a high degree of user control. But what then? Will every extra GPU pull its own weight with regards to the total output?
Assuming this is true, there’s no reason for the demand for electrical power and hardware to ever abate — imagine a game of Risk where one seizes Data Centers instead of fertile land. This would also likely mean that the “state of the art” would forever be out of reach of individuals.
If the scaling hits dismissing returns, then we could expect the demand to slow, leading to improved availability of the technology — ultimately placing some of the control back in the hands of the users. Perhaps networks similar to “peer 2 peer” could appear, distributing inference load about several cooperating systems. That’s probably wishful thinking though.
Can the beast starve itself?
Assuming the AI effectively displaces anyone working with a keyboard (except for CEOs of course ;) ) then it’s not a long jump to assume that software innovation at the hands of humans will essentially stop.
We create new tools to solve problems or become more efficient. Be it making deployments and ops more reliable, improving the “developer experience”, making our programs memory/type safe…
If we are not the ones wrestling with the limits of our current tools, where will the drive for innovation in this sector come from? Will “good enough” become all that ever is?
If that happens, can the models keep improving? They have essentially been made and improved on human-generated works. But if you’ve ever seen the result of training an image-generation model on its own output, you know it is disastrous. Can the same happen to code?
Autonomy and sovereignty
Assuming AI-augmented systems become prevalent in every aspect of life, both personal and professional, then the tool itself becomes a liability.
If you cannot do payroll, enter a building, fix a leaking faucet or pay your
groceries without some API call being made to a server on the other side of the
country (or overseas), then he who controls the AI controls the universe you.
I don’t want to sound like a paranoid scaredy-cat, but we’ve seen the damage unintentional outage10 of cloud(-related) services do. What if a directed, politically-motivated shutdown (at the behest of governments or any other bad actor) became a viable and tempting option?
Such exposure is undesirable, as would be a single point of failure for a country’s power grid or water supply, whether or not ill-intentioned actors exist.
And even on a much smaller scale, what we currently know as XSS, ClickJacking, CSRF, would pale in comparison of a prompt injection that could spread itself: Imagine an assistant that reads incoming emails and gives you a TL;DR version of them. A poisoned email could be enough to hijack it, potentially using your email to spread to your contacts – like a spam email chain, except it steals all the secrets on your device.
If this is hosted, and the recovery mechanisms are exposed to such tools, one might end up entirely locked out of their own system, unable to pull the proverbial plug, because it is somewhere they cannot access.
The cynic in me thinks that it’ll get (way) worse before it gets better – the financial incentives are simply too great for the companies and countries to stop the arms race anytime soon.
Passing laws that impede the march of your own AI companies is suicide if you cannot force others to obey them as well. And I know these are not nuclear weapons we are talking about — but they might as well be. If most of the world’s data goes through your servers, and your API powers major electronic systems, your tokens essentially become the new oil.
If these tokens become necessary to operate other energy-providing systems (electrical, oil, gas…) then how do you ever break free?
How do you find out what is real and what is not, when search engines become so polluted by fabricated content, that you essentially need to rely on 3rd-party processing to separate the wheat from the chaff?
The return to in-person
Another aspect I have not touched on is the social engineering angle.
I’m positive that in the not-so-distant future, photographic evidence will no longer exist. When images can be faked in real-time, how do you know what is real and what is not? Who will you turn to for reliable information?
A single picture of you is enough to produce fakes that can do harm before they are debunked. A voice sample obtained via spam calls is enough to replicate your voice and impersonate you. The more of you circulates, the better the fakes.
With the amount of personal data floating around, it’d be easy to find out your phone number, get a sample of your voice, then call your boss and have you quit with extreme prejudice. Or call the police on “yourself” confessing to the most heinous crimes.
Cryptography, and digitally-signed media might help, but how do you make that both widespread and reliable? What if your keys get compromised? Once we have removed humans from the processes, how do we fix them?
Perhaps we’ll see a radical shift towards in-person communication, and a gradual desertion of online spaces.
All of us outside, literally touching grass.
Footnotes
-
70 years, if we take Dartmouth’s 1956 workshop as a starting point. Wikipedia ↩
-
We’re All So F’d | NVIDIA x Palantir, Global Surveillance, “Pre-Crime” Arrests, & AI (YouTube) ↩
-
“Grandmother jailed for 6 months after AI error linked her to a crime in a state she had never even visited, lawyers say” - The Independent ↩
-
Louis Rossman - Innocent grandma imprisoned for 6 months due to AI mistake: enough is enough. (YouTube) ↩
-
See the “Consumer Rights Wiki” https://consumerrights.wiki/ ↩
-
Gamers Nexus: RAM: WTF? (YouTube), The DRAM Cartel | Price Fixing, Anti-Consumer Collusion, & Corporate Conspiracy (YouTube) ↩
-
Timeline of the lawsuits (sustainabletechpartner.com) ↩
-
AWS Outage Forbes / CrowdStrike 2025 incident TechTarget.com ↩