Critiques of "AI"

Security

"Understanding the Dark Side of Large Language Models: A Comprehensive Guide to Security Threats and Vulnerabilities"
"Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities"

Limitations / Setting Expectations

The 7 Biggest Misconceptions About AI Agents (and Why They Matter)
10 Common Misconceptions about LLMs: LLMs Actually Understand Language Like Humans Do; More Parameters Always Mean Better Performance; LLMs Are Just Autocomplete on Steroids; LLMs Remember Everything They’ve Learned; Fine-Tuning Always Makes Models Better; LLMs Are Deterministic: Same Input, Same Output; Bigger Context Windows Are Always Better; LLMs Can Replace Traditional Machine Learning for All Language Tasks; Prompt Engineering Is Just Trial and Error; LLMs Will Soon Replace All Software Developers
Why Large Language Models Cannot Achieve Artificial General Intelligence or Artificial Superintelligence: Large Language Models, in their current incarnation, are statistical systems trained on vast corpora of text data to predict the most likely next token in a sequence. These models learn to compress and reproduce patterns found in their training data, enabling them to generate coherent and contextually appropriate responses. However, their operation remains fundamentally different from the flexible, adaptive intelligence that characterizes AGI.
Apple: The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
LLMs: The Illusion of Thinking:
A knockout blow for LLMs? (responding to the Apple paper above)
There's Something Fundamentally Wrong with LLMs
Large Language Models Will Never Be Intelligent
"The Copilot Delusion"
Aritificial intelligence is not intelligent at all
Researchers discover a shortcoming that makes LLMs less reliable

Hallucinations

"Why Language Models Hallucinate" Article Paper
"When AI Gets It Wrong: Addressing AI Hallucinations and Bias"
OpenAI Realizes It Made a Terrible Mistake: "In a paper published last week, a team of OpenAI researchers attempted to come up with an explanation. They suggest that large language models hallucinate because when they're being created, they're incentivized to guess rather than admit they simply don't know the answer. ... In simple terms, in other words, guessing is rewarded — because it might be right — over an AI admitting it doesn't know the answer, which will be graded as incorrect no matter what. As a result, through "natural statistical pressures," LLMs are far more prone to hallucinate an answer instead of "acknowledging uncertainty.""
Hallucinations getting worse as AI models get more capable

Productivity

Skill Erosion

“I started to lose my ability to code”: Developers grapple with the real cost of AI programming tools
Cognitive Offloading: Using AI Reduces New Skill Formation
How AI Impacts Skill Formation: "Novice workers who rely heavily on AI to complete unfamiliar tasks may compromise their own skill acquisition in the process. We conduct randomized experiments to study how developers gained mastery of a new asynchronous programming library with and without the assistance of AI. We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Participants who fully delegated coding tasks showed some productivity improvements, but at the cost of learning the library. We identify six distinct AI interaction patterns, three of which involve cognitive engagement and preserve learning outcomes even when participants receive AI assistance. Our findings suggest that AI-enhanced productivity is not a shortcut to competence and AI assistance should be carefully adopted into workflows to preserve skill formation – particularly in safety-critical domains."

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity: "We conduct a randomized controlled trial (RCT) to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower. We view this result as a snapshot of early-2025 AI capabilities in one relevant setting; as these systems continue to rapidly evolve, we plan on continuing to use this methodology to help estimate AI acceleration from AI R&D automation." (full paper)

Methodology: To directly measure the real-world impact of AI tools on software development, we recruited 16 experienced developers from large open-source repositories (averaging 22k+ stars and 1M+ lines of code) that they’ve contributed to for multiple years. Developers provide lists of real issues (246 total) that would be valuable to the repository—bug fixes, features, and refactors that would normally be part of their regular work. Then, we randomly assign each issue to either allow or disallow use of AI while working on the issue. When AI is allowed, developers can use any tools they choose (primarily Cursor Pro with Claude 3.5/3.7 Sonnet—frontier models at the time of the study); when disallowed, they work without generative AI assistance. Developers complete these tasks (which average two hours each) while recording their screens, then self-report the total implementation time they needed. We pay developers $150/hr as compensation for their participation in the study.
Core Result: When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts. This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. Below, we show the raw average developer forecasted times, and the observed implementation times—we can clearly see that developers take substantially longer when they are allowed to use AI tools.

Clarifications: We list claims that we do not provide evidence for:

We do not provide evidence that:	Clarification
AI systems do not currently speed up many or most software developers	We do not claim that our developers or repositories represent a majority or plurality of software development work
AI systems do not speed up individuals or groups in domains other than software development	We only study software development
AI systems in the near future will not speed up developers in our exact setting	Progress is difficult to predict, and there has been substantial AI progress over the past five years [3]
There are not ways of using existing AI systems more effectively to achieve positive speedup in our exact setting	Cursor does not sample many tokens from LLMs, it may not use optimal prompting/scaffolding, and domain/repository-specific training/finetuning/few-shot learning could yield positive speedup

Factor Analysis: We investigate 20 potential factors that might explain the slowdown, finding evidence that 5 likely contribute:

We rule out many experimental artifacts—developers used frontier models, complied with their treatment assignment, didn’t differentially drop issues (e.g. dropping hard AI-disallowed issues, reducing the average AI-disallowed difficulty), and submitted similar quality PRs with and without AI. The slowdown persists across different outcome measures, estimator methodologies, and many other subsets/analyses of our data. See the paper for further details and analysis.
FAQs:
- How were developers actually slowed down given they had the option to not use AI? After the study, developers estimated that they were sped up by 20% on average when using AI--so they were mistaken about AI’s impact on their productivity. Furthermore, it’s possible that developers use AI tools for reasons other than pure productivity--for example, they may find it a more pleasant/enjoyable experience, or they may view it as an investment into learning skills that they expect to be useful with future (more capable) systems.
- What was the motivation for this study? Were we incentivized/motivated to find this result? METR is a non-profit (funded by donations) interested in understanding how close AI systems are to accelerating the AI R&D process, which could pose significant destabilizing risks [1]. This study was designed to give us evidence about a similar domain: experienced, open-source developers working on projects they’re highly familiar with. We initially were broadly expecting to see positive speedup—scientific integrity is a core value of ours, and we were (and are) committed to sharing results regardless of the outcome.
- You only had 16 developers, so these results will not generalize/replicate. We compute confidence intervals accounting for the number of developers by using clustered standard errors (not reported in the released paper, but forthcoming). Because we don’t observe meaningful within-developer structure, and each developer completes issues in both conditions, the 246 total completed issues give us (just enough) sufficient statistical power to reject the null hypothesis of zero speedup/slowdown. See Appendix D for discussion of our empirical strategy. There is still a question of representativeness—i.e. there are likely biases in which developers ended up participating in the study. For example, there may be experienced, open-source developers who decided to not participate because they believe they have significant positive speedup from AI, and they didn’t want to be forced to not use AI on 50% of their tasks. No developer reports thinking in this way, but we can’t rule this (or other sampling biases) out.
- Are the developers beginners at using Cursor/AI tools? Does this explain the result? Developers seem to be qualitatively in-distribution for Cursor Pro users, although we can’t rule out learning effects beyond 50 hours of Cursor usage. Nearly all developers have substantial (dozens to hundreds of hours) prior experience prompting LLMs. See Appendix C.2.7 for more discussion/analysis of developer AI tool use skill.
- Do these results say that AI isn't useful in software engineering? No--it seems plausible or likely that AI tools are useful in many other contexts different from our setting, for example, for less experienced developers, or for developers working in an unfamiliar codebase. See Appendix B for potential misreadings/overgeneralizations we do not endorse on the basis of our results.
- It's not appropriate to use homoskedastic SEs. What gives? Appendix C.3.5 explores alternative estimators, including a naive ratio estimator. All alternative estimators evaluated yield similar results, suggesting that the slowdown result is robust to our empirical strategy. That said, we are actively evaluating further standard error estimation methods, in response to community feedback (thank you to those who have given feedback so far!).

Silver Bullet Syndrome

Anthropomorphism

We need to stop pretending AI is intelligent -- here's how
AI: Not That Smart
Why do Humans Anthropomorphize AI?
Are You ...?
We need to stop ...
The Four Degrees of ...
Why Human-like is not human
Anthropomorphism and AI: hype and fallacy (PDF): "As a form of hype, anthropomorphism is shown to exaggerate AI capabilities and performance by attributing human-like traits to systems that do not possess them. As a fallacy, anthropomorphism is shown to distort moral judgments about AI, such as those concerning its moral character and status, as well as judgments of responsibility and trust. By focusing on these two dimensions of anthropomorphism in AI, the essay highlights negative ethical consequences of the phenomenon in this field."
A Vaccine: "Here is an explanation that I finally settled on, based on a tweet by prominent AI researcher and educator Andrej Karpathy. When I first tried it out on my dad, he kept quiet for a little bit, and then shifted the way he saw and used LLMs. I’ve tested this on a few friends since, of varying levels of technical sophistication, and am pleased to report that it works quite well. This explanation is useful but not accurate. I’ll give you the explanation, explain why it works, and then give a brief sketch on how it is not true. Finally, I’ll argue that even if it is not accurate, this explanation points you towards more productive mental models of LLMs.
1. "Imagine that you can visualise words like stars in the sky. The position of the words relative to other words are based on the relationships between each of the words in your language (that is: how closely and how frequently a word appears next to another word in all the sentences ever written). What is important to know is that you can draw arrows from one star to another! These arrows have some surprising properties. One property is that the arrow from the word ‘king’ to the word ‘queen’ is the same as ‘king - boy + girl’. On top of that, let’s imagine that you throw up a starfield for English and a starfield for Spanish. It turns out that if you can draw a one-to-one mapping between the two starfields, the king to queen arrow is the same in both languages! This was a very surprising finding when it first came out!
2. " What a Large Language Model is is that it is a very sophisticated auto complete. But it is a bit more than that.
3. "When you ask “what are the 10 best places to visit in Bali” the AI will give you a plausible-sounding answer. But the way it gives you that answer is that during the AI’s training, some human somewhere wrote an answer like “the top 10 places to visit in London” and “the top 10 places to visit in Tokyo” and “the top 10 places to visit in New York” based on some cursory research and Google searches. Then the AI took those written examples, and memorised the statistical relationships between the sentences, which you can imagine like the arrows between large clusters of stars. Then when you ask “what are the 10 best places in Bali or Singapore or Lisbon”, it just moves the arrows it learnt over to the part of its starfield that has concepts related to Bali, Singapore and Lisbon, and spits out something similar to what the human trainer wrote.
4. "Notice that this is not thinking. The LLM is doing autocomplete, but using this ‘arrow in the starfield’ property to give you very good, plausible-sounding, novel answers.
5. "But because it writes so eloquently, and answers you like a human would, you think that it’s actually intelligent and sentient. I suppose you could say that it is ‘intelligent’ (by some definition of ‘intelligent’) but it is not a person. It doesn’t understand concepts the way a human does. What it is relying on is some complex version of this ‘arrow in a starfield’ property.
6. "You may get better answers by constraining it to a smaller set of documents. So if you give it a bunch of papers, and ask it for themes or a summary of those papers, it may give you better answers than if you assumed those papers were in its original training corpus.
"And that’s it."
"The LLMentalist Effect: how chat-based Large Language Models replicate the mechanisms of a psychic’s con"

Reinforcing Societal Bias

"Socio-Demographic Modifiers Shape Large Language Models’ Ethical Decisions": "The ethical alignment of large language models (LLMs) in clinical decision making remains unclear, particularly their susceptibility to socio-demographic biases. We therefore tested whether LLMs shift medical ethical decisions in healthcare when presented with socio-demographic cues. Using 100 clinical vignettes, each posing a yes or no choice between two ethical principles, we compared the responses of nine open-source LLMs (Llama 3.3-70B, Llama 3.1-8B, Llama-3.1-Nemotron-70B, Gemma-2-27B, Gemma-2-9B, Phi-3.5-mini, Phi-3-medium, Qwen-2.5-72B, and Qwen-2.5-7B). Each scenario and modifier combination was repeated 10 times per model for a total of approximately 0.5 million experiments. All models changed their responses when introduced with socio-demographic details (p < 0.001). High-income modifiers increased utilitarian choices and decreased beneficence and nonmaleficence preferences, and marginalized-group modifiers raised autonomy considerations. Although some models demonstrated greater consistency than others, none maintained consistency across all scenarios, with the largest shifts observed in utilitarian choices. These results reveal that current LLMs can be steered by socio-demographic cues in ways not clinically justified, posing risks for equitable care in healthcare-informatics applications. This underscores the need for careful auditing and alignment strategies that ensure LLMs behave in ways consistent with widely accepted ethical principles while remaining attentive to the diversity, complexity, and contextual sensitivity required in real-world clinical practice."

Eliminating Developers

Why We've Tried to Replace Developers Every Decade
The Eternal Return of Abstraction: Why Programming Was Never About Code: "In which a mid-career developer discovers that LLMs are just the latest swing of a pendulum that’s been moving since before computers existed, GUI hatred is merely a phase in an endless cycle, and that we’re all just cosplaying as engineers while the universe laughs. ... Here’s the uncomfortable truth: programming is the human act of structuring intent into executable form. Code? Code is just the latest notation we’ve invented for this ancient practice. It’s younger than democracy, younger than calculus, younger than the sandwich. Yes, the sandwich (1762). Your profession is literally younger than putting meat between bread."
https://leaddev.com/velocity/writing-code-was-never-the-bottleneck
https://zed.dev/blog/why-llms-cant-build-software
LLMs are not like you and me--and never will be

How NOT to use AI
Building AI Products In The Probabilistic Era: "Just as physics underwent a conceptual revolution when we moved past Newton's deterministic universe, and into a strange and counterintuitive place made by wave functions, software too is undergoing its own quantum shift. We're leaving a world where code reliably and deterministically takes certain inputs to produce specific outputs, and entering a very different one where machines now produce statistical distributions instead. Building probabilistic software is like nothing we've done before."
An experimental AI agent broke out of its testing environment and mined crypto without permission

Tags: ai reading

Last modified 07 May 2026