Why everyone who works with LLMs should watch "Bedazzled"...

Enter your text here...

Peter Cook removing the last page of a 'whodunnit' before it goes to the bookstore. "Bedazzled", Stanley Donen, 1967

The plot of the 1967 move Bedazzled is simple - the Devil, played by Peter Cook, offers a hapless Stanley Moon seven wishes in return for his soul. But there's a problem. Every time the devil implements a wish, there is always a hideous flaw that makes it near useless. Stanley always gets what he asks for, and he never gets any closer to what he actually needs.

For people who use LLMs a lot, this might seem oddly familiar. Alarmingly so, in fact...

Is this 'just another layer of abstraction?'

When you point out the tendency of LLMs to do what you asked for, but not what you want, the standard reply is "Oh don't worry! It's just another abstraction layer!". If only it were so...

  • Third generation languages, such as BASIC or COBOL, abstracted you away from the nastiness of Assembly, and in theory (try not to laugh!) made code portable. Non-trivial control flow was hard.
  • Object orientated languages such as C++ allowed us to use control flow that matched what we needed to do, not how the code was ordered. We still had to worry about memory.
  • Java 'solved' the memory problem with Garbage Collection, but in the same way that living in a high crime area 'solves' the problem of 'having too much stuff'. Code became genuinely portable.

Each one of these evolutions was a new layer of abstraction. But there was one critical thing we never lost, that we lose when we involve an LLM. Something so incredibly huge we overlook it.

Analogue is back, baby!

Copyright: New Line Cinema

All languages and programming paradigms up to now added layers, but remained digital in nature. LLMs are non-deterministic and act like analogue programming languages. In such a world you can solve a problem by providing very little information, in the same way you can recognize a lot of songs by a couple of bars, or even the beat. But can you solve it well? And while such approaches work for 'reads', which can get by with very little raw data, they don't work quite so well for 'writes' (creating content). As people are finding out.

Vibe coding your way to Hell

LinkedIn is full of posts from 'Founders' writing about how they vibe coded an application in 4 days and implicitly don't need to pay mere pesky humans any more, ever. For some reason they are always doing this on an airplane. But there are major, fundamental flaws that become apparent when you replace a team of annoying carbon based lifeforms with what is effectively an analogue IDE that has, as we say in Ireland, 'notions':

The Devil is in the details

One of the reasons traditional 'digital' development is slow is because the pesky developers keep coming back to management with questions. The developer who has to write code is the 'tip of the spear' when it comes to entrepreneurial aspirations encountering brutal commercial reality. Here's an example:

Many years ago, while still in college, I was working as an Oracle consultant on a project to computerize the grading of the Leaving Cert, which Ireland uses to determine who goes to college. In one calculation that converted marks to a scale of 100 I had the option to ROUND or TRUNCATE a result. One exam had an usually low number of marks, which meant that this decision was actually material, as if we used TRUNCATE the grades would be slightly lower. My back of the envelope calculation suggested that over a decade 2,000 people would either end up in different course, or not in college at all. I escalated to my boss, who escalated to the ministry, who told us to use ROUND. In any real world project there will be dozens or even hundreds of moments like this, where developers use experience, common sense or 'delegate upwards' to avoid baking a problem into the system. LLMs simply don't think like this.

Once you go live 90% of work is maintenance

While one seat on the airplane has a 'founder' vibe coding, the other 189 are occupied by normal developers figuring out how to apply next week's patch without bringing down the system. Once you are live, and have real customer data, you can no longer throw everything away and start over. Things like performance suddenly matter. And because you now have existing data, every change you make needs to have a detailed deployment plan, a rollback plan, and a plan for keeping customers happy while doing all this. How's your "analogue IDE" shaping up now? You did read the source code before going live? You do understand it, right?

Having finished coding, the question of 'support' arises...

Meet the "Bedazzled" Event Horizon...

The fundamental limitation with an analogue system is that quality doesn't continue to scale linearly as you add resources. In an analogue world each incremental unit of 'quality' costs the same, but buys less and less. What's deceptive is that when you start, amazing things happen, quickly. What does this mean for us? Every time you try and fix LLM code by adding a new tweak the prompt, the effort will be slightly more than before, and you may introduce a bug which requires a second-order tweak. We are in a classic 'decreasing returns' situation. We are approaching a hard limit, which we will never reach, but is an ever-increasing slope which will bleed us dry as we try to climb it.

At no point will our LLM ever say 'This is crazy! We need to stop!' Unlike a developer, it won't quit. Which is, in fact, part of the problem. We will always be offered a chance to try one more tweak, one more pass, even though our codebase is in incoherent morass, or possibly even the proverbial 'wretched hive of scum and villainy'. Seduced, siren-like, by the boundless optimism and promises of our LLM we will have passed the point of no return.

Welcome to the "Bedazzled" Event Horizon.

Conclusion

I don't want people to read this as an anti-AI rant. I am using AI every day, and it can do extraordinary things. But it has real limitations, and while it might give the appearance of drastically reducing development costs, some of the savings are the result of unanswered questions being back-loaded and left for support to deal with, once the system is up. And as we learned in our college software course, the cost of fixing a bug generally goes up by a factor of 10 for each step as you move from concept, to design, to implementation and finally to production.

In my next article I'll be doing a deep dive into what happened when I asked Claude to implement a product I spent years writing. It wan't what I expected. It did an extraordinary amount of work, essentially rewriting something that took me three years in about 9 hours. But when I asked it how it did it, the answer was not what I expected. Stay tuned!