The Fundamental Misunderstanding Fueling the AI Coding Hype

This week, science communicator and YouTuber Hank Green published this video where he reflects on the dropping "cost of code". Critically, Hank is not a software developer, and while he makes several interesting points, the main thesis of his video (the dropping cost of code is to programming what Guntherberg was to printing) immediately surfaced a familiar intuition: "This isn't how this works!".

I am a Software Developer by trade, with eight years of professional experience in Backend/DevOps/SRE development. And I would like to think I'm at least decent at my job, and like many of my peers, I have had a strong intuition that LLM-based so-called "AI Agents" are incapable of replacing me; not "not yet there", ontologically incapable.

What I did not have was good dialectic tools to express the intuitive reasons behind this belief. When someone suggested I read Peter Naur's 1985 Programming as Theory Building, I found them, the beautiful, shiny, dialectic tools necessary to express those ideas. The tools are a bit crusty – they are from the 1980s, using a very academic dialect, and such turns of phrase as "program text" instead of "source code".

I will therefore aim to do two things: firstly, to rephrase the relevant bits of Programming as Theory Building, as I understand them, for the 21^st century; and secondly, to recontextualize some of those points as they pertain to modern discourse around AI-generated code.

What this is deliberately not is an essay on the ethical or practical relevance of AI coding in software development workflows. Much ink has already been spilled on the subject, and I do not believe I have any worthwhile thought to change anyone's mind on the matter.

Programming as Theory Building

The paper Programming as Theory Building by Peter Naur (1985) is a seminal work of Computer Science. It is easily found online and not particularly long, but I will attempt to succinctly summarize the core points in 21^st century, decidedly non-Academic English. If you are already familiar with it, feel free to skip this section.

Its main thesis is that Developers don't write Code, they build a Theory about the problem domain which they apply through Code. The Theory is not the Spec, the Documentation, the Code, or any combination of the three; it is the purely intellectual, higher order understanding of the problem domain, in much the same way that a music sheet is not the Experience of listening to and performing said music.
The Code and the Documentation, as thorough as they may be, are still just partial manifestations of the Theory that the Developer seldom possesses. Useful tools, but not the whole truth.

This initial intuition is immediately supported by an enecdotal evidence; in a large enough software project no amount of documentation can make up for the need of a trained, competent "L3" team made up of the original developer(s) for troubleshooting complex issues.

This thesis is supported by the arguments as outlined below.

Developer Responsibilities

These are the three main responsibilities of the Developer, as outlined by Naur in his paper.

The Code must abide by Real World demands

A Project Manager would say "Guys, for the fifteenth time, please follow the spec". A Vibe Coder would plead, "Claude, that's not what I asked". A Customer might complain, "this is nice and all – but I wanted an omelet, not a chicken coop".

It is the Developer's responsibility to ensure that the Code they produce works as expected and solves real problems, by translating their Theory of the system into practical programming and design decisions.

As a personal side-note, I will point out that "following the Spec" and "abiding by Real World demands" are not quite the same thing; the Spec is merely the shadows on the wall of Plato's Cave. As detailed as it may be, it will never perfectly capture reality. A good Developer keeps that in mind and challenges the Spec where they believe it to be incompletely or incorrectly addressing Real World demands.

The Code must be justifiable

Here is a shameless tautology for you: There is Good Code, and there is Bad Code.

... Which means there is Code which strikes a balance of flexibility, simplicity, robustness, and cost; and there is Code that does not, or fails to.

A Developer must be able to justify Design and Programming decisions come review time. It is proper engineering practice to engage in back-and-forth direct challenges; "Why is this the right decision", "Wouldn't Y have been better", "Isn't Z simpler", "What about this edge-case". To an outsider, it may look like Developers tearing each other down, but the goal is actually to use challenges as a form of consensus building.

That is the process by which Developers leverage rational, analytical thinking to justify how their design and programming decisions relate to the Theory.

Changes Requests must be addressed intelligently

Virtually all Code is bound to change. Customers come back with notes; use-cases change; hardware and software stacks change; security gaps are found.

Many a long-term project has devolved into a mountain of patches, each change precariously sitting on top of all others, as seemingly endless strata of functions, modules, and services built on top of existing functionality no-one dares touch. The end result is Software which is outrageously expensive to change, and virtually impossible to test.

Developers who have come to understand the Theory have the necessary understanding to make informed decisions about how to make changes which optimally leverage existing facilities, or which strategically refactor the code to add flexibility to avoid redundant and potentially conflicting behaviors.

The boldest claim of all is then that it is the Developer's responsibility to implement Code changes in a way that best leverages existing facilities and minimizes technical debt.

Change Management

As outlined previously, it is not a question of if your Code will change, but when it will change.

Changing Code is capital-H Hard. Always has been, always will be; it is easier to add a Jenga piece to the top of the tower than it is to add it to the middle. Raw LOC output is not a useful metric here; in fact, the most valuable Developer for your project's long-term reliability may be the one who removes more code than they add.

Possession of the Theory allows Developers to identify existing facilities, established design decisions, and relevant architectural details to properly confront the Code with the Real World changes in order to arrive at an optimal solution with minimal technical debt.

Developers faced with an unknown codebase and tasked with a code change do not possess such Theory; they will therefore inevitably make naive changes, failing to leverage existing facilities and adding new or modified behaviors as patches, layering them until the Jenga tower collapses as soon as a fly lands on it.

Code Lifecycle

Any given project or codebase is likely to go through the following cycle;

1. At first, the project is scoped, customers are involved, Developers are actively designing and implementing the software. Having shipped their first version to production, they keep actively iterating on the project, implementing features and squashing bugs.
   This whole time, each developer is actively constructing and building off of the Theory.
2. Down the line, the project has matured, the business has moved on, and the Code is dead. It may still be in active production use, but it is not being actively worked on anymore. The Developers still have the Theory but have moved on to other things.
   That is not a value judgment; there is no need to keep working on something that is (sufficiently) feature-complete.
3. Optionally, the Code needs to be revived for some ulterior project.
   Ideally, the original Developers are still around and can advise the new Developers on how to proceed; the mentoring process allows them to gracefully transfer their understanding of the Theory.
   Unfortunately, the original Developers are not always available. In that case, the Theory has to be rebuilt anew. Reading the Documentation and the Code only goes so far; the new Developers still have to spend a lot of time rebuilding an intuitive understanding of the Theory in order to effectively fulfill their Responsibilities as outlined above.
   (Side note: This part is where Naur argues that, as a consequence, taking over a codebase without its original developers is a nonsensical endeavor and a rewrite makes more sense. This is, in my opinion, an insufficient basis for such a radical approach in many cases.)

Developer Status

Traditional management techniques (e.g. Taylorism, Fordism, etc.) view Developers as nothing more than assembly line workers. Their job, in that view, is to mindlessly apply a given set of rules; here's a feature request, here's the spec, implement, review, rinse, repeat.

Such a naive approach to Software Development is doomed to fail as the real fruit of the Developer's labor is not Lines of Code or User Stories, but their built understanding of the Theory. The Theory is how they add value to the project, for without it they cannot fulfill their Responsibilities described above.

Workers treated as mindless drones will deliver code. But that code will fail on one or several of the Developer's Responsibilities, which will lead to sub-par or even non-functional Code in the long run.

Therefore, the Developer must be held Responsible for their contributions, and for that, they must be held as a peer and an equal to any Engineer, Lawyer, or Project Manager in the company.

Building on the Theory in the 21^st century

The most important takeaway when this 1985 paper was released was the "Developer Status" section, which was – for its time – revolutionary. Giving the keyboard monkeys agency! How novel!
Today, this idea has properly permeated the tech world, and only particularly uptight and – let's face it – backwards companies still fail to see the value in giving Developers proper agency.

However, with us having gained a common understanding of Programming as Theory Building, the following statement should now come to have a clear meaning:

AI Agents may be programmers, but they are not Developers for they do not have the Theory.

Sure, for simple enough problems, the cross-product of a large context window and trillions of parameters may be an adequate solution. However, LLMs are still just fundamentally prediction machines and do not possess real reasoning capabilities.

In particular, going over the Developer Responsibilities once again:

1. LLMs do not consistently abide by Real World demands[1]. To their credit though, they approximate a passable answer to what is queried of them often enough that they have co-opted to the world economy, so credit where credit is due.
   Still, there is no guarantee that any given output actually matches the prompt. And even if it did (which it doesn't), properly condensing Real World information into a proper prompt is already doing most of the Developer's work – remember, the real work is building the Theory, not spitting out the Code. That's the ancillary part.
2. LLMs cannot justify their decisions according to any consistent Theory. They make arbitrary design decisions, and if pressed, they may just as well sycophanctly change thousands of lines of code or gaslight their operator into believing their technical "decision" is technically sound. However none of that is based on any real reasoning.
   The Developer still shoulders the responsibility for any and all technical "decisions" made by the LLM.
3. LLMs are incapable of intelligently addressing Change Requests. They can make changes; but much like a Junior Developer unleashed on an unknown codebase, an LLM cannot reason itself into using existing facilities, avoiding code duplication, and making thoughtful refactors[1][2]. Yes, even if you add no technical debt please to CLAUDE.md[2]!
   Once again, the Developer personally shoulders the responsibility of minimizing technical debt.

Another parallel that can be drawn is that large swaths of Code generated by an LLM are analogous to going through the "Revival" part of the Code Lifecycle, but in reverse; the LLM generated a bunch of Code with no Theory, and the Developer who holds the Theory must compare the generated Code to it.

Lastly, the status of the Developer is well above that of the "AI Agent". The idea of an "AI Agent" appeals most to people who like traditional management techniques; the AI Agent executes blindly and doesn't complain. Consequently, they are also subject to the same pitfalls that were decried by Naur back in 1985.

Conclusion

So where does that leave us? Well, the best case scenario for an LLM is that it spits out code which accurately reflects the prompt, according to techical and design decisions made by a human Developer in accordance with the Theory. Which is to say the Developer has not been replaced, and their most important responsibility – building the Theory – has never been put into question.

The immediate corollary is that the efficiency gains of delegating programming work to an LLM are bounded to how much time a developer actually spends writing code – which is usually much less than half of their time. You can't be 200% more efficient by automating a task that takes less than 50% of your time – it sounds obvious when I see it like that, but evidently many people in the industry who are paid a lot more than I am don't seem willing to understand that Developers do more than clacking lines of code at their keyboard all day.

At risk of repeating myself, I'm not making a particular claim towards the general (non-)usefulness of AI here. I am simply reframing the discussion in terms of responsibilities; those which are solely the Developer's, and the ancillary byproduct (Code) which may or may not be delegated to an AI Agent, or a random contractor half the world away (replace "AI Agent"/"LLM" with "low-skill contractor" in this essay and the point still stands).

I understand there is a tremendous amount of money to be made by selling investment pitches that ignore all the above and pretend that Software Development is a Solved Problem™. Hopefully, this essay gives at least one other person some necessary dialectic tools to fight against disingenuous attempts at discrediting the actual work of Software Developers.