“Nobody is in charge. It’s a headless blunder operating under the illusion of a master plan.”
- Worth, in “Cube” (1997)
I think this is a good article and accurate. I’ve tried to block the rising tide of this within my own company, but my management is pressing forward with plans to implement an AI harness. I pushed forward a design that at least put questions and decisions in the hands of developers, but I got my hand slapped.
We are announcing 10x dev speeds based on pocs that haven’t been team reviewed or deployed in test. The code looks fine tbh, but I know it’s an illusion. The moment we start signing contracts based on 10x, the writing is on the fucking wall.
AI can be an accelerator — my teams have seen 30% productivity gains without sacrificing quality, and I think I could push that higher, perhaps as much as 2x, though I’d rather just target continuous progress than predict the benchmark — but we can’t be happy with that and so we are turning an incredible success into an epic failure.
Does that 30% increase include extra review time? I’m finding that my small team is producing a lot more code, but I’m also spending way more time reviewing, correcting stupid LLMisms, and keeping complexity in check. The last point is the one that concerns me the most because I just can’t keep up and complexity is starting to balloon.
Yes. They are rebalancing to more review/orchestration then writing code. That is measurable in story points over the last 8 months that we have been using AI heavily.
The trick (and I’m not saying I have it solved but I work hard on this with some success) is making sure the AI doesn’t bloat the documentation to the point where it can’t be understood. Unambiguous diagrams and meta languages are easier to understand and validate for correctness by developers and AI alike. The more we rely on the ambiguous natural language to explain everything, the less understandable it is to anyone.
Thanks! That’s really helpful. I’ll have to keep pushing back on the team to make sure they get better at their own review instead of just dumping it all on me. I still think it’s important to have an understanding of the system, and more review will help.
The code looks fine tbh, but I know it’s an illusion.
In my experience, it isn’t. At least as long as you use a very recent model, a language with strong checks (basically Rust), and you have a good test set, or a specification that AI can derive tests from.
2x
Rookie numbers. Once you accept that you aren’t going to be able to closely review all the code, and you switch to testing, 2x is laughable. AI can produce working products hundreds of times faster than humans. And yes I do mean working.
Curious how that works out for you. Especially when you’re troubleshooting unexpected behaviors. Because AI is pretty garbage at understanding why things go wrong. I do prod support every day and it correctly explains the problem <30% of the time.
Also, I catch subtle things AI code does wrong at least once a week. That’s just what I catch. Imagine how much I must miss at these volumes. No, software is going to be a dumpster fire for the foreseeable future and then we will emerge from the ashes as we always do.
Because AI is pretty garbage at understanding why things go wrong.
Lol what? That’s probably the thing AI is best at.
I do prod support every day and it correctly explains the problem <30% of the time.
I suspect you’re holding it wrong. What kind of product is it, and how do you use AI? I can probably help if you want?
Also, I catch subtle things AI code does wrong at least once a week.
Me too, but I catch subtle things that humans do wrong more than that.
I suspect you’re holding it wrong. What kind of product is it, and how do you use AI? I can probably help if you want?
Well, my prod support toolkit pulls data from anywhere from 6 to 16 different systems to local jsonl files with PII obfuscated so it doesn’t get sent to the AI. Then it has to examine where data integrity was violated, assign it to the system or user that mishandled the data, discover the root cause — because if it’s an automated system then either a business rule went wrong, data was previously polluted, or an agent mis-entered data in some upstream system. Lastly, it has to identify how to fix the data and request permission to resolve the issue, or provide instructions to other operators with permissions in systems I can’t reach into to fix, such as prod database or Salesforce, which is the domain of other teams.
It diagnoses anything from user error to system instability to networking issues to user error to business requirements that work cross-purposes. It queries three different data stores, two different log providers, and ten or so APIs.
It has several playbooks for processing “standard” problems, which include instructions to attempt to falsify its conclusions. It also has documentation of each system and specific conventions/apis/etc. And to the degree possible, it executes everything deterministically with small python scripts for each system, “flow” bash scripts for executing standard sets of calls, and transforming and validation. It also executes multiple adversarial agents that attempt to disprove its conclusions or rewrite unclear wording and remove all the little one-time notes it leaves for itself.
I don’t want to suggest this toolkit isn’t extraordinarily helpful, because it has saved me countless hours, but it gets conclusions wrong all the time and I have to continually ask it whether it performed this check or that one, or why it decided this was the root cause when there isn’t enough information to make that determination or there is clearly evidence that points in another direction.
It is written pretty much exactly as I envisioned and when I need to add another tool for another system, it does that itself in minutes. There are likely some areas that could be improved.
I recently added the PII obfuscation because someone would push back hard on that in a security review once we explain what we’re doing. It would work better if every system wasn’t written by separate teams to separate standards, but it does really well. I can compare tokens to see if they match without decoding them, and I also have a script for turning tokens back to clear text so that I can see the values without sharing them with the cloud.
Let me know if you see any gaps there other than operational because I don’t get to dictate business requirements. I think it’s extraordinarily robust, but that doesn’t help it diagnose the right problem.
Point is: if it has all of that and can’t reliable reason about how systems work together, what are the chances it’s misunderstanding how the code needs to go together?
Software hasn’t been working too well lately, has it?
It hasn’t for a number of decades, if we’re being honest. The difference is that the stakes nowadays are much, much, much higher than, say, in 1999
About ten years ago, two small revolutions happened in our industry, whilst the rest of the world paid little attention. They were cloud computing, and open source libraries. (…) tools like NPM hit the scene in 2014, it really felt like a gamechanger. You could just add a line to your package.json, hit npm install and - whoosh - now you could reuse code written by any stranger on the internet.
Python’s pip and RubyGems have been around for many years prior, fyi.
Open source was great because you had a small number of passionate, unpaid developers, building code that everyone else depended on. (…) But the cost was that everyone stopped knowing the details.
It’s kinda ironic how one of the takes from that can be “this is why programmers should continue reinventing the wheel”
I heard: “The cost of open source is that everyone stopped knowing the details, therefore everyone should use shady blobs made by big tech instead.”
It’s kinda ironic how one of the takes from that can be “this is why programmers should continue reinventing the wheel”
It’s a bit more nuanced.
- Software Engineering is about tradeoffs, since it’s a form of engineering
- There is more than one topic to address here from a best practices standpoint
- There is more than one way to use package managers, and it’s up to you to implement a correct solution for the use case.
Software Engineering is about tradeoffs, since it’s a form of engineering
You and your team need to choose where the tradeoff between re-implementing functionalities and re-writing existing things that you have control need to be made. Do you really need to use Left-Pad, or can you write a function and leave it in a JS library for your software to call? Do you have the time to dedicate to reviewing the dependencies and how they function? Is that available for your target language? Is it worth it to use a different language if you don’t get the source code of your libraries?
There are several other factors to consider these are just some examples.
There is more than one topic to address here from a best practices standpoint
In this instance, it’s largely two ideas from the Pragmatic Programmer :
- DRY : Don’t repeat yourself/don’t reinvent the wheel (NB : the original idea is to refactor code so you’re not copy pasting edge cases all over the place, but similar ideas fit into this perspective)
- Cargo cult programming : Know what your software is doing.
Yes, you should not repeat yourself. But also, you should know what it is the dependencies you have are doing and not just blindly trust them. Someone, at some point, should be responsable for vetting the security and reliability of the packages you are choosing to make use of in your software. There is a bedrock level where you need to trust that at least the language or compiler or equivalent is going to work correctly, but the most recent package that you are using that is on version 0.0.1 that is going to make the backbone of your business might need some vetting. Perhaps choose to re-implement that one so you are using something more dependable that isn’t going to be at risk due to external factors.
There is more than one way to use package managers, and it’s up to you to implement a correct solution for the use case.
Certain package managers allow you to host your own package store on a network. You can use this functionality to secure a certain version, and ensure that your systems will depend on a vetted, secure version of that library. This way, if a package gets compromised or changed to not include your necessities, you can still continue to function without being affected.
If you, the reader, found this useful, I am currently looking for work.
Eternally relevant Gregor Kiczales: Why Black Boxes Are So Hard To Reuse
“Reinventing the wheel” is part of how programming is taught. Not every “wheel” on NPM is free of un-foreseen consequence either; Not even looking at the borrowed wheel, let alone understanding it, should never have been rewarded.
Python’s pip and RubyGems have been around for many years prior, fyi.
And PHP’s PEAR, and Perl’s CPAN
No. Just people are starting to look at the tower of abstractions upon which they built their world.
You only need to embrace understanding to realize that the stacks could be much smaller and much more stable and more likely to be correctly secured.
But getting people to think in formal proofs, state machines and boring technologies is quite difficult.
It was curious to me how he takes the PHP vulnerability as an example. You can have your stack of one component: PHP, and you’ll be vulnerable.
I know! I’ll drink a bunch of red bull stay up for 4 days straight and give Anthropic 10000 dollars!! And in a mania induced frenzy rebuild everything! That will fix it! /s
I like your characterization of security as a statistical outcome, if not the fact its effectively an anomally leftover from … other … times.



