Reading in Artificial Intelligence

Curious thoughts

"Who owns the code?": "This shift raises an important question: who is accountable when something goes wrong – Copilot, the reviewer, or someone else?Rajesh Jethwa, CTO of software engineering consultancy Digiterre, describes this issue as a “minefield”, because there are a number of entities involved in creating the code. First, there are the providers of the models themselves, such as OpenAI or Anthropic. It is currently unclear whether these providers own the code generated by their models. Second, there are the authors of the code used to train the model. There are still questions around whether they have any claim to ownership of the resulting code, given the provenance of the training data. Third, there are employees and the organizations they work for. Typically, when an employee creates code as part of their job, the organization owns that code. However, it remains uncertain whether the organization or the individual employee should bear responsibility for the code that is produced with the help of a coding assistance."

Contrarian/hype-debunking

"The LLMentalist Effect: how chat-based Large Language Models replicate the mechanisms of a psychic’s con"
"AI coding tools make developers slower but they think they're faster, study finds"

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity: "We conduct a randomized controlled trial (RCT) to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower. We view this result as a snapshot of early-2025 AI capabilities in one relevant setting; as these systems continue to rapidly evolve, we plan on continuing to use this methodology to help estimate AI acceleration from AI R&D automation." (full paper)

Methodology: To directly measure the real-world impact of AI tools on software development, we recruited 16 experienced developers from large open-source repositories (averaging 22k+ stars and 1M+ lines of code) that they’ve contributed to for multiple years. Developers provide lists of real issues (246 total) that would be valuable to the repository—bug fixes, features, and refactors that would normally be part of their regular work. Then, we randomly assign each issue to either allow or disallow use of AI while working on the issue. When AI is allowed, developers can use any tools they choose (primarily Cursor Pro with Claude 3.5/3.7 Sonnet—frontier models at the time of the study); when disallowed, they work without generative AI assistance. Developers complete these tasks (which average two hours each) while recording their screens, then self-report the total implementation time they needed. We pay developers $150/hr as compensation for their participation in the study.
Core Result: When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts. This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. Below, we show the raw average developer forecasted times, and the observed implementation times—we can clearly see that developers take substantially longer when they are allowed to use AI tools.

Clarifications: We list claims that we do not provide evidence for:

We do not provide evidence that:	Clarification
AI systems do not currently speed up many or most software developers	We do not claim that our developers or repositories represent a majority or plurality of software development work
AI systems do not speed up individuals or groups in domains other than software development	We only study software development
AI systems in the near future will not speed up developers in our exact setting	Progress is difficult to predict, and there has been substantial AI progress over the past five years [3]
There are not ways of using existing AI systems more effectively to achieve positive speedup in our exact setting	Cursor does not sample many tokens from LLMs, it may not use optimal prompting/scaffolding, and domain/repository-specific training/finetuning/few-shot learning could yield positive speedup

Factor Analysis: We investigate 20 potential factors that might explain the slowdown, finding evidence that 5 likely contribute:

We rule out many experimental artifacts—developers used frontier models, complied with their treatment assignment, didn’t differentially drop issues (e.g. dropping hard AI-disallowed issues, reducing the average AI-disallowed difficulty), and submitted similar quality PRs with and without AI. The slowdown persists across different outcome measures, estimator methodologies, and many other subsets/analyses of our data. See the paper for further details and analysis.
FAQs:
- How were developers actually slowed down given they had the option to not use AI? After the study, developers estimated that they were sped up by 20% on average when using AI--so they were mistaken about AI’s impact on their productivity. Furthermore, it’s possible that developers use AI tools for reasons other than pure productivity--for example, they may find it a more pleasant/enjoyable experience, or they may view it as an investment into learning skills that they expect to be useful with future (more capable) systems.
- What was the motivation for this study? Were we incentivized/motivated to find this result? METR is a non-profit (funded by donations) interested in understanding how close AI systems are to accelerating the AI R&D process, which could pose significant destabilizing risks [1]. This study was designed to give us evidence about a similar domain: experienced, open-source developers working on projects they’re highly familiar with. We initially were broadly expecting to see positive speedup—scientific integrity is a core value of ours, and we were (and are) committed to sharing results regardless of the outcome.
- You only had 16 developers, so these results will not generalize/replicate. We compute confidence intervals accounting for the number of developers by using clustered standard errors (not reported in the released paper, but forthcoming). Because we don’t observe meaningful within-developer structure, and each developer completes issues in both conditions, the 246 total completed issues give us (just enough) sufficient statistical power to reject the null hypothesis of zero speedup/slowdown. See Appendix D for discussion of our empirical strategy. There is still a question of representativeness—i.e. there are likely biases in which developers ended up participating in the study. For example, there may be experienced, open-source developers who decided to not participate because they believe they have significant positive speedup from AI, and they didn’t want to be forced to not use AI on 50% of their tasks. No developer reports thinking in this way, but we can’t rule this (or other sampling biases) out.
- Are the developers beginners at using Cursor/AI tools? Does this explain the result? Developers seem to be qualitatively in-distribution for Cursor Pro users, although we can’t rule out learning effects beyond 50 hours of Cursor usage. Nearly all developers have substantial (dozens to hundreds of hours) prior experience prompting LLMs. See Appendix C.2.7 for more discussion/analysis of developer AI tool use skill.
- Do these results say that AI isn't useful in software engineering? No--it seems plausible or likely that AI tools are useful in many other contexts different from our setting, for example, for less experienced developers, or for developers working in an unfamiliar codebase. See Appendix B for potential misreadings/overgeneralizations we do not endorse on the basis of our results.
- It's not appropriate to use homoskedastic SEs. What gives? Appendix C.3.5 explores alternative estimators, including a naive ratio estimator. All alternative estimators evaluated yield similar results, suggesting that the slowdown result is robust to our empirical strategy. That said, we are actively evaluating further standard error estimation methods, in response to community feedback (thank you to those who have given feedback so far!).

General

Artificial Intelligence for a Better Future: An Ecosystem Perspective on the Ethics of AI and Emerging Digital Technologies - Bernd Carsten Stahl (PDF)
Artificial Intelligence: Foundations of Computational Agents (2010), 1st Edition - David L. Poole, Alan K. Mackworth @ Cambridge University Press (HTML)
Artificial Intelligence: Foundations of Computational Agents (2017), 2nd Edition - David L. Poole, Alan K. Mackworth @ Cambridge University Press (HTML, Slides)
Deep Learning
Introduction to Autonomous Robots - Nikolaus Correll (PDF)
On the Path to AI: Law’s prophecies and the conceptual foundations of the machine learning age - Thomas D. Grant, Damon J. Wischik (PDF)
Probabilistic Programming & Bayesian Methods for Hackers - Cam Davidson-Pilon (HTML, Jupyter Notebook)
The Quest for Artificial Intelligence: A History of Ideas and Achievements - Nils J. Nilsson (PDF)

Fuzzy Logic

Expert Systems

Wikipedia

Large Language Models

"On the Biology of a Large Language Model": "We investigate the internal mechanisms used by Claude 3.5 Haiku — Anthropic's lightweight production model — in a variety of contexts, using our circuit tracing methodology."

Natural Language Processing

Machine Learning

A Brief Introduction to Machine Learning for Engineers - Osvaldo Simeone (PDF)
A Brief Introduction to Neural Networks
A Comprehensive Guide to Machine Learning - Soroush Nasiriany, Garrett Thomas, William Wang, Alex Yang (PDF)
A Course in Machine Learning (PDF)
A First Encounter with Machine Learning - Max Welling (PDF) (:card_file_box: archived)
A Selective Overview of Deep Learning - Fan, Ma, and Zhong (PDF)
Algorithms for Reinforcement Learning - Csaba Szepesvári (PDF)
An Introduction to Statistical Learning - Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani (PDF)
Approaching Almost Any Machine Learning Problem - Abhishek Thakur (PDF)
Bayesian Reasoning and Machine Learning
Deep Learning - Ian Goodfellow, Yoshua Bengio and Aaron Courville
Deep Learning for Coders with Fastai and PyTorch - Jeremy Howard, Sylvain Gugger (Jupyter Notebooks)
Deep Learning with PyTorch - Eli Stevens, Luca Antiga, Thomas Viehmann (PDF)
Dive into Deep Learning
Explorations in Parallel Distributed Processing: A Handbook of Models, Programs, and Exercises - James L. McClelland
Foundations of Machine Learning, Second Edition - Mehryar Mohri, Afshin Rostamizadeh, Ameet Talwalkar
Free and Open Machine Learning - Maikel Mardjan (HTML)
Gaussian Processes for Machine Learning
IBM Machine Learning for Dummies - Judith Hurwitz, Daniel Kirsch
Information Theory, Inference, and Learning Algorithms
Interpretable Machine Learning - Christoph Molnar
Introduction to CNTK Succinctly - James McCaffrey
Introduction to Machine Learning - Amnon Shashua
Keras Succinctly - James McCaffrey
Learn Tensorflow - Jupyter Notebooks
Learning Deep Architectures for AI (PDF)
Machine Learning
Machine Learning for Data Streams - Albert Bifet, Ricard Gavaldà, Geoff Holmes, Bernhard Pfahringer
Machine Learning from Scratch - Danny Friedman (HTML, PDF, Jupyter Book)
Machine Learning, Neural and Statistical Classification
Machine Learning with Python - Tutorials Point (HTML, PDF)
Mathematics for Machine Learning - Garrett Thomas (PDF)
Mathematics for Machine Learning - Marc Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong
Neural Networks and Deep Learning
Practitioners guide to MLOps - Khalid Samala, Jarek Kazmierczak, Donna Schut (PDF)
Probabilistic Models in the Study of Language (Draft, with R code)
Python Machine Learning Projects - Lisa Tagliaferri, Brian Boucheron, Michelle Morales, Ellie Birkbeck, Alvin Wan (PDF, EPUB, Kindle)
Reinforcement Learning: An Introduction - Richard S. Sutton, Andrew G. Barto (PDF)
Speech and Language Processing (3rd Edition Draft) - Daniel Jurafsky, James H. Martin (PDF)
The Elements of Statistical Learning - Trevor Hastie, Robert Tibshirani, and Jerome Friedman
The LION Way: Machine Learning plus Intelligent Optimization - Roberto Battiti, Mauro Brunato (PDF)
The Mechanics of Machine Learning - Terence Parr and Jeremy Howard
The Python Game Book - Horst Jens (:card_file_box: archived)
Top 10 Machine Learning Algorithms Every Engineer Should Know - Binny Mathews and Omair Aasim
Understanding Machine Learning: From Theory to Algorithms - Shai Shalev-Shwartz, Shai Ben-David

Coding Assistants/Interaction

"How to use GPT as a natural language to SQL query engine"

Retrieval-Augmented Generation (RAG)

Tags: reading ai machine learning fuzzy logic logic expert system nlp

Last modified 23 August 2025