Some conjecture on my part, but I imagine that the bugs being found are not the kind you can just press a button on the LLM to fix anyway, but are instead the sort of bugs which require the reviewer to deeply consider design tradeoffs, of which I believe there are simply more in a DB or programming language codebase, for example. Stuff I am finding in review is often like "this thing isn't like the other thing, which has this behavior, which is wrong if the aim is X, but sort of correct if the aim is Y, and totally perfect is the goal is Z and the use case is J, but not I. Unless it's Q, in which case you probably need to reconsider some fundamental decision making from 12 months ago and have some conversations with people about how users desire it to work in the first place, and then set about understanding the tradeoffs inherent to that design."
ETA: This is extremely generous to the LLMs, btw. If I got an LLM to pop something this concise out I would fully commit to finding a different profession tomorrow, but for now I am for better or worse stuck doing prompting some non-zero number of times in order to get all of this assembled context, and then need to use my human powers to decide whether or not more humans indeed need to discuss it.
I like the way you put this. I agree this is where the frontier of software development is right now. You can have LLM chug and bug fix all day long but in the end most real issues are in this layer of intent. I don't think this problem is particular to DB/programming language codebase either - it happens a lot in product design it just doesn't surface in code review. "Everything is working perfectly" yet the resulting workflow or UI is not truly making the user's life easier, or has grown grotesque with additions/modifications.
I will say, if the problem is purely code you occasionally see "sparks of AGI" where the model will step back and say "hold on, there is a massive simplification we can make here...". This started happening with Fable-class models. But rarely if ever will the model push for a huge simplification at the product level.
This is why code is going to be automated, and the software engineer is a link between organizational intent and the code. This is what I’m working on.
Finding the holes in your specs under automation: time spent is returned 10x
Writing code: Human 15 lines per minute. LLM: 600+ lines per minute
Functional Validation: same time per iteration. Fewer bugs with LLM when properly specified => fewer iterations. Increased awareness of LLM escaping specification boundaries feeds back into increasing quality of spec writing => fewer bugs.
The difference is that instead of spending half your time understanding the business requirements and half your time writing code, you'll spend half your time understanding the business requirements, 5% of your time typing these requirements into a chatbot, and the rest of your time doing... ????
Some of the rest of the time can be spent making it look like your ratios are the same but in reality you just take the saved time and do whatever you want with it. The decision making and reviews aren’t happening any faster anyway so just take it easy and coast a bit
I guarantee there will not actually be saved time - the bullshit will increase to more than compensate, as everyone actually fixing things will be drowned out by bullshitters just pretending to fix things and creating more actual problems.
You nailed it! I often got bug reports from AI that would touch far more code to fix than what I am willing to touch, and the likelihood of that large piece of new code (which I honestly would not be able to review thoroughly) will likely contain more bugs than the one I am fixing.
So in the end I often ignore these bug reports after evaluating their actual impact on the product.
And imagine you had an LLM ingest the entire corpus of comments and discussion threads for all of Postgres' history, and asked it to make a judgment call on the tradeoffs - or even on how to present the tradeoffs for discussion.
How much of its judgment would be driven by the data vs. its own internal knowledge? How much might be driven by old vs. new discussions that might not represent new directions of the project? How much might be driven by over-matching against how bugs in less mission-critical code paths were handled?
There's no way to quantify the answers to these questions. Which is not to say this applies to every problem out there! But human judgment is still a bottleneck for design questions on widely, disparately-used complex systems.
ETA: This is extremely generous to the LLMs, btw. If I got an LLM to pop something this concise out I would fully commit to finding a different profession tomorrow
Many of us have already experienced that uncomfortable revelation, where a leading-edge LLM actually comments correctly on our questionable judgement. You will, too, soon enough.
This is one angle of the productivity story. You don't have any sort of plausible deniability covering all these bugs, test cases etc. when they are comprehensively discoverable by tooling. It's like having typing bugs on a strongly typed language with static analysis, you just can't make a lot of excuses.
So that's one measure of extra time that will be spent on every single feature, where in the past things would've been rolled out in a more barebones state. It's better overall, but is it going to be perceived as being more productive?
There’s plenty of things you can very deterministically verify — if the LLM writes a test case which causes an ASan violation, you probably have a bug; if the LLM writes a test case which run the various corruption checking tools and they find corruption then you either have corruption or you have a bug in the corruption checking tool; etc.
There’s plenty of ways in which you can guide LLMs, and sandbox them, to get a much higher hit rate.
And that’s before the increase of validity of bugs compared with a year ago — we’re past the peak of slopageddon, and maybe even at a point where a higher percentage of security bugs reported are valid than was the case pre-LLMs.
Not in my experience. I've pointed agents like Claude at existing codebases I work on, and it does a pretty solid job at identifying bugs. It's not garbage or slop at all.
There's a big difference between randos getting agents to hallucinate bug reports on GitHub, vs. when your own internal workflow of LLM-assisted code review is identifying real, legitimate problems in your new features.
This totally matches my experience. It's actually kind of insane how well LLM's like Claude are able to find edge cases, race conditions, overflows, states you didn't think could be reached. It really makes you think about code quality in a whole new way. Becoming aware of so many potential bugs when you write the code, vs. finding out from users over the coming months/years -- it forces you to really consider the costs of quality all at once in a way that simply wasn't possible before.
Opus continues to tell me that my API is missing a `/ready` route that a smoke test can check against. I assume it’s part of its memories, but it’s now 3 or 4 weeks that it won’t let go of that whenever tasked with test design.
I remember an LWN article on this question a few weeks ago. The debate was going on and they weren’t sure if it was that there were more bugs or just that because of AI they suddenly knew about more bugs than usual.
Either way, unfortunate but a very understandable decision.
I'm inclined to think the latter - all sorts of software, particularly long-standing open source Unix software written in C, is riddled with bugs that sometimes don't get discovered until years later. Most ordinary Unix system software like Postgres is written in such a way that it's not possible to be confident that the code is free of subtle bugs, and we should assume that it is not free of subtle bugs.
QA is becoming the blocker for projects in an AI era, and the real source of value. AIs can lay down a good base of testing but I see even frontier models still need help with testing because they are basically as bad as humans in terms of testing only what comes to mind, if not even a touch worse. Though they are really good at blasting through problems once they are found.
Ironically, despite the predictions that AI will murder SaaS, I think the result is they end up even more entrenched, because Job Bob's Bait Shack And Tax Filing Services may be happy to instantiate some front office system for themselves via AI prompt, but they aren't going to want to QA that post-AI any more than they want to now. Once we all settle down, SaaS will manage more code than before, and do more QA, and Joe Bob's etc. will be even less interested in trying to replicate the work with AI because it will represent an even larger QA load than it does today, relative to the size of their business.
Or, to put it in the context of this article, even if you can prompt a database into being today, which you can, you can't prompt something into existence that has been tested and QA'd like Postgres, and you would be well advised to ponder that gap between your prompt and billions of computer hours of field testing very carefully before you decide where to store your company's most important data.
> Ironically, despite the predictions that AI will murder SaaS, I think the result is they end up even more entrenched, because Job Bob's Bait Shack And Tax Filing Services
OK, so this is slightly different since it's more about how runtime behavior than design-time behavior, but... I'm in a SaaS workplace, and I've been trying to seed the following idea:
1. The very cheapest/fastest end of the market will be eaten up by Job Bob's Bait Shack stuff. Cheap wrappers around a prompt that goes "Pretend you're a computer that obeys the following rules" jank. A race to the bottom we can't win.
2. Most of our (desirable) customers will (eventually) realize they can't tolerate systems which casually or unexpectedly violate known business rules. That leads to angry consumers, junk data, broken workflows, and lawsuits.
3. When our product can capture and enforce those rules with confidence, that's both an important differentiator and also a kind of moat/stickiness.
4. Therefore there is a strategic business justification for Engineering to fence-off where we use LLMs in production. When we use them for design, it's to create better deterministic systems than before.
The last point is one I make often. AI cannot be trusted to be deterministic, but they can produce deterministic tools that are verifiable as such. Those tools I can trust as much as any other complex tool from the pre-AI era.
I don’t think I’ve worked at a place that wasn’t trying to automate away QA. Perhaps Delivery Hero pre-acquisition, since we had 2,5 QAs for 4 devs in my team.
So… are we gonna use the extra productivity (and money) from coding agents to hire QA people? Oh boy, is that highly unlikely.
But I like your take. I wish it were so, perhaps we wouldn’t have had Tahoe.
Not to mention the blast radius of any small mistake in payment processing, tax filing, or any other financial or regulatory issue, is just so massive compared to making some ai slop bar special flyer. Joe's bait shack is still just going to pay for square.
There will be a ton of smaller Joe’s bait shacks, all competing against each other in a race to the bottom - and with relatively limited blast radius per shop.
Not surprising. I've been running AI reviews through old codebases and finding all sorts of interesting things. AI is significantly raising the quality bar and there is a lot of backfilling to do.
I have a unique perspective on this as I've been working on rewriting and rearchitecting Postgres to solve a lot of the long standing issues Postgres has[0][1].
If I had to root cause why we're finding so many bugs now, I would attribute it to the limited Postgres test suite. AI is merely uncovering bugs that already existed. For the most part, Postgres relies on individual regression tests to test each feature. While regression tests are great for making sure features don't break (yes, SELECT 'Hello, World' LIKE '%Hello%' still returns true), they actually aren't that great hitting the edge cases (passing a string that's over 2MB to a specific function will trigger a buffer overflow and crash your DB). The main way Postgres has prevented bugs in the past has been by minimizing risky changes and having lots of human review. As we're learning now, there were lots of gaps that were not caught by the existing process.
While the Postgres team is skeptical of AI, I've had lots of success in using AI to scale more advanced forms of testing such as Deterministic Simulation Testing, differential coverage guided fuzzing, and formal verification. To put it in perspective, in the process of doing testing for pgrust, I've uncovered ~50 different bugs in core Postgres. The bugs I'm finding aren't through AI review, but are instead through using AI to do more advanced testing than regression testing. All of the techniques I'm using existed existed pre-AI, but aren't used in meaningful capacity as part of the core Postgres development.
Right now I'm going all in on formal verification. I wrote a compiler to translate C and Rust functions to Lean and am then having an LLM write a proof that the two functions are identical. So far I've been able to verify 10.5% of the codebases behave identically (10.5% = 67k/640k LOC of Postgres). I'm finding details as minute as a parsing function where Postgres trimmed spaces using isspace() and pgrust only trimmed ascii whitespaces. These aren't cases you would find with traditional regression testing or human review. The thoroughness of the types of bugs I'm finding is giving me confidence that that this is catching more bugs than any sort of manual testing could. I don't currently see any reason that I won't be able to scale up the formal verification to cover 100% of the codebase. The only issue is it takes a long time to do.
> What does "rebuild" mean? Translation from C to Rust?
We translated from C to Rust, rearchitected several components and are working on rearchitecting more. For instance, we replaced the process-per-connection model with a thread-per-connection model and added a columnar format + new query executor that's hundreds of times faster than Postgres for OLAP workloads.
I feel the pain. We have been delaying release because sooo many bug reports have come in. AI is finding a lot of obscure, and interesting, and annoying bugs.
Doesn't that just imply the existing version of Postgres also probably has more bugs than humans can fix? Just release it. Software is never perfect and always a work in progress. I'm guessing the bulk of these bugs are probably stupid edge cases that just get development bogged down trying to plug so many holes at once.
Indeed, but I for one very much appreciate Postgres’ stability and quality.
As a lead contributor explains in the article, if they were to ship some of these broken features, they wouldn’t be able to fix them until v20, rather than patching them on-the-go. Sounds like a Very Bad Idea given many installations stay behind for some years.
I tried that this summer. I encountered this doom loop where the review agent would find bugs, the coding agent would fix them in some overengineered way that introduced new bugs, the review agent found them, the coding agent made more while fixing them, repeat until insanity.
I've been building a side project end-to-end with agents, and it's really not true for complex software.
At a micro scale their code is right, but tons of invented complexity no sane person would introduce. I've found the two most common triggers are the models polluting their own context windows (poorly worded instructions in comments/issues/AGENTS.md) and a poor "taste" for breaking changes (they'll constantly try to make their additions non-breaking, and if prompted they can make breaking changes, it has to be very carefully scoped or they start assigning great reward to ripping out things)
That is how you end up with a test suite that now takes 10x as long, unless you occasionally prompt the agent to consolidate and group tests.
There are still a lot of issues with coding that require intervention. Just recently, I had Astra detect a minor bug in an HTTP client that resulted in it deciding that the best approach would be to spend 1800 LOC on implementing its own from scratch. Some other time, Opus 5.5 was trying to tell me that test runs on a small to mid-sized application would take another three weeks until they would be fully finished.
Y'know, I had a teammate who had been claiming this up until it became clear that the horrific spike in production defects that we were struggling with was mostly attributable to them.
I'm not going to say that's always what happens. But I will say that you can't find problems you aren't watching for.
> But I will say that you can't find problems you aren't watching for
I can't, but Claude almost regularly comes across "BTW things". I catch them because I read the lead agent's transcript, and have worker agents always add a retro section to their report, so I'm almost always seeing attention-worthy things pop up.
But think of Claude as being like a particularly keen outsourced developer: great at the tactical stuff, but sometimes lacks common sense in weird and surprising ways, so you’ve still got to keep an eye on the work product.
That's not my experience. Agent usually follows the pattern correctly by creating a relevant test and then making it pass. Where strange tests are more likely to fall in is when agent writes the test after the testee.
Same experience. It's especially bad with Claude and anything with non-code text, where it will happily write up endless Claude-ese comments that break any rules in the repo and are constantly incorrect in small ways, catch them in the review, rewrite specific lines in ways that are still wrong despite still having the review in its context, and repeat twenty times in a row.
The coding loop and the review loop are orthogonal and probably should remain so. Not least because it's probably worth the money for more intelligent models for code reviews where it's a more focused task while coding loops are all over the place.
Weirdly enough I've found the opposite, using too smart a review model gets you overfit instead of benefit. The coding side can be more complex in a way that benefits from fable/astra, but the review side sometimes not so much.
Seeing how Fable handled Linux kernel bugs, where out of 82 bugs reported only 6-8 were actually bugs (not already fixed or fixed in newest version or straight up made up) I'm skeptical till humans actually sift through this pile.
I have seen people discussing the fact that AI can literally find an infinite number of bugs. There are always more bugs to be found, whether they would have ever affected any actual real world case or been noticed or not.
What makes you think that the number of bugs is finite?
There are alternatives to that possibility. A perfect program without bugs might be like a perfect definition of the concept of "chair". Whatever definition you come up with there will likely be edges cases that makes the definition awkward or incorrect, and any definition that takes into account all those cases will not be a particularly good or useful definition.
> whether they would have ever affected any actual real world case or been noticed or not.
That's what I was thinking as well. Wouldn't the majority of the bugs not also have existed in Postgres 18? The version that everyone is using atm...
There is a similar thing with AI detecting vulnerabilities. It's not that the vulnerability isn't real, but just that in most cases for these vulnerabilities to exist there also needs to be a very specific set of circumstances as well making the likelihood of the vulnerability to exist in the wild quite low.
Good! We need to delay all software releases until security audits of the code by AI/fuzzers/etc. Otherwise people ship buggy software, the hackers find the exploits, and we're compromised as soon as we upgrade. This is why we absolutely must have un-nerfed security models. And if you want to stop having your personal information leaked by compromised companies, we need to make this process mandatory.
This is the happy path for LLM assisted coding. Obviously consumes less tokens being more constrained and focused so it's not what gets the most marketing but having the second loop of code reviews seems worth investment even if the coding loop is entirely manual or you're in full LLM autopilot mode
Training data is filled with shortcuts, conveniences, bugs, hacks, hacks that fix other hacks, hot fixes, patches, mappings that map the hack to hack to a better solution and all types of other surprises.
That's the cost of trying to scale so quickly. You dont filter out the crap.
A nice side effect is that the best way to fix it is more AI.
From the article: "He was also at pains to point out that the reverts were not down to 'vibe coding' in development, as the technique was not available when most of the new features were written."
I wonder when people will finally feel comfortable not putting vibe coding in scare quotes. Maybe when it accounts for 50% of all new software, instead of only 10%.
Why read the article when you can read the headline and make up your own?
I think these type of new techniques will ultimately create more jobs for humans. If we think the answer is just “AI” everyone will create their own Postgres with a lot more bugs and vulnerabilities.
"Explaining the number of reverts — features whose rollouts have now been postponed to next year's release, PostgreSQL 20 — Peter Eisentraut, veep at EDB, a PostgreSQL support and consultancy, said the reason could be at least in part attributed to the LLM code reviews used by developers."
They have been doing Postgres for a long time, and I believe they have been significant contributors from before AI coding.
Their marketing has to mention AI. Everyone has to in a business like that. I would not take it as evidence they are slop coders or hyping AI just on that basis.
ETA: This is extremely generous to the LLMs, btw. If I got an LLM to pop something this concise out I would fully commit to finding a different profession tomorrow, but for now I am for better or worse stuck doing prompting some non-zero number of times in order to get all of this assembled context, and then need to use my human powers to decide whether or not more humans indeed need to discuss it.
I will say, if the problem is purely code you occasionally see "sparks of AGI" where the model will step back and say "hold on, there is a massive simplification we can make here...". This started happening with Fable-class models. But rarely if ever will the model push for a huge simplification at the product level.
This is why code is going to be automated, and the software engineer is a link between organizational intent and the code. This is what I’m working on.
Finding the holes in your specs under automation: time spent is returned 10x
Writing code: Human 15 lines per minute. LLM: 600+ lines per minute
Functional Validation: same time per iteration. Fewer bugs with LLM when properly specified => fewer iterations. Increased awareness of LLM escaping specification boundaries feeds back into increasing quality of spec writing => fewer bugs.
So in the end I often ignore these bug reports after evaluating their actual impact on the product.
How much of its judgment would be driven by the data vs. its own internal knowledge? How much might be driven by old vs. new discussions that might not represent new directions of the project? How much might be driven by over-matching against how bugs in less mission-critical code paths were handled?
There's no way to quantify the answers to these questions. Which is not to say this applies to every problem out there! But human judgment is still a bottleneck for design questions on widely, disparately-used complex systems.
Many of us have already experienced that uncomfortable revelation, where a leading-edge LLM actually comments correctly on our questionable judgement. You will, too, soon enough.
So that's one measure of extra time that will be spent on every single feature, where in the past things would've been rolled out in a more barebones state. It's better overall, but is it going to be perceived as being more productive?
Sorry, garbage in garbage out doesnt fly. I feel sad for open source maintainers having to deal with this massive slop-ageddon.
There’s plenty of ways in which you can guide LLMs, and sandbox them, to get a much higher hit rate.
And that’s before the increase of validity of bugs compared with a year ago — we’re past the peak of slopageddon, and maybe even at a point where a higher percentage of security bugs reported are valid than was the case pre-LLMs.
There's a big difference between randos getting agents to hallucinate bug reports on GitHub, vs. when your own internal workflow of LLM-assisted code review is identifying real, legitimate problems in your new features.
This totally matches my experience. It's actually kind of insane how well LLM's like Claude are able to find edge cases, race conditions, overflows, states you didn't think could be reached. It really makes you think about code quality in a whole new way. Becoming aware of so many potential bugs when you write the code, vs. finding out from users over the coming months/years -- it forces you to really consider the costs of quality all at once in a way that simply wasn't possible before.
Opus continues to tell me that my API is missing a `/ready` route that a smoke test can check against. I assume it’s part of its memories, but it’s now 3 or 4 weeks that it won’t let go of that whenever tasked with test design.
Either way, unfortunate but a very understandable decision.
Ironically, despite the predictions that AI will murder SaaS, I think the result is they end up even more entrenched, because Job Bob's Bait Shack And Tax Filing Services may be happy to instantiate some front office system for themselves via AI prompt, but they aren't going to want to QA that post-AI any more than they want to now. Once we all settle down, SaaS will manage more code than before, and do more QA, and Joe Bob's etc. will be even less interested in trying to replicate the work with AI because it will represent an even larger QA load than it does today, relative to the size of their business.
Or, to put it in the context of this article, even if you can prompt a database into being today, which you can, you can't prompt something into existence that has been tested and QA'd like Postgres, and you would be well advised to ponder that gap between your prompt and billions of computer hours of field testing very carefully before you decide where to store your company's most important data.
OK, so this is slightly different since it's more about how runtime behavior than design-time behavior, but... I'm in a SaaS workplace, and I've been trying to seed the following idea:
1. The very cheapest/fastest end of the market will be eaten up by Job Bob's Bait Shack stuff. Cheap wrappers around a prompt that goes "Pretend you're a computer that obeys the following rules" jank. A race to the bottom we can't win.
2. Most of our (desirable) customers will (eventually) realize they can't tolerate systems which casually or unexpectedly violate known business rules. That leads to angry consumers, junk data, broken workflows, and lawsuits.
3. When our product can capture and enforce those rules with confidence, that's both an important differentiator and also a kind of moat/stickiness.
4. Therefore there is a strategic business justification for Engineering to fence-off where we use LLMs in production. When we use them for design, it's to create better deterministic systems than before.
I'd say pre-LLM, just because I don't want to surrender the term "AI" to one particular subtype without a fight.
Some of those complex tools would be "Good Old-Fashioned AI", which generally means stuff doing symbolic logic rather than mystery-weights.
So… are we gonna use the extra productivity (and money) from coding agents to hire QA people? Oh boy, is that highly unlikely.
But I like your take. I wish it were so, perhaps we wouldn’t have had Tahoe.
If I had to root cause why we're finding so many bugs now, I would attribute it to the limited Postgres test suite. AI is merely uncovering bugs that already existed. For the most part, Postgres relies on individual regression tests to test each feature. While regression tests are great for making sure features don't break (yes, SELECT 'Hello, World' LIKE '%Hello%' still returns true), they actually aren't that great hitting the edge cases (passing a string that's over 2MB to a specific function will trigger a buffer overflow and crash your DB). The main way Postgres has prevented bugs in the past has been by minimizing risky changes and having lots of human review. As we're learning now, there were lots of gaps that were not caught by the existing process.
While the Postgres team is skeptical of AI, I've had lots of success in using AI to scale more advanced forms of testing such as Deterministic Simulation Testing, differential coverage guided fuzzing, and formal verification. To put it in perspective, in the process of doing testing for pgrust, I've uncovered ~50 different bugs in core Postgres. The bugs I'm finding aren't through AI review, but are instead through using AI to do more advanced testing than regression testing. All of the techniques I'm using existed existed pre-AI, but aren't used in meaningful capacity as part of the core Postgres development.
Right now I'm going all in on formal verification. I wrote a compiler to translate C and Rust functions to Lean and am then having an LLM write a proof that the two functions are identical. So far I've been able to verify 10.5% of the codebases behave identically (10.5% = 67k/640k LOC of Postgres). I'm finding details as minute as a parsing function where Postgres trimmed spaces using isspace() and pgrust only trimmed ascii whitespaces. These aren't cases you would find with traditional regression testing or human review. The thoroughness of the types of bugs I'm finding is giving me confidence that that this is catching more bugs than any sort of manual testing could. I don't currently see any reason that I won't be able to scale up the formal verification to cover 100% of the codebase. The only issue is it takes a long time to do.
[0] https://github.com/malisper/pgrust
[1] https://malisper.me/the-four-horsemen-behind-thousands-of-po...
You look at Postgres C functions and prove they are identical, but the license is AGPL without mentioning Postgres authors. Consider me confused.
We translated from C to Rust, rearchitected several components and are working on rearchitecting more. For instance, we replaced the process-per-connection model with a thread-per-connection model and added a columnar format + new query executor that's hundreds of times faster than Postgres for OLAP workloads.
> without mentioning Postgres authors
We have a NOTICE file which references the Postgres license here: https://github.com/malisper/pgrust/blob/main/NOTICE
Relational databases are one of those things where you really do want to get a whole lot closer to perfect than a lot of other software.
As a lead contributor explains in the article, if they were to ship some of these broken features, they wouldn’t be able to fix them until v20, rather than patching them on-the-go. Sounds like a Very Bad Idea given many installations stay behind for some years.
Not every time. But often enough.
At a micro scale their code is right, but tons of invented complexity no sane person would introduce. I've found the two most common triggers are the models polluting their own context windows (poorly worded instructions in comments/issues/AGENTS.md) and a poor "taste" for breaking changes (they'll constantly try to make their additions non-breaking, and if prompted they can make breaking changes, it has to be very carefully scoped or they start assigning great reward to ripping out things)
And as others have said some models people prefer for implementation work are less prone to this.. 5.6 Sol, 6.1 Sol, Opus 5.5
There are still a lot of issues with coding that require intervention. Just recently, I had Astra detect a minor bug in an HTTP client that resulted in it deciding that the best approach would be to spend 1800 LOC on implementing its own from scratch. Some other time, Opus 5.5 was trying to tell me that test runs on a small to mid-sized application would take another three weeks until they would be fully finished.
I'm not going to say that's always what happens. But I will say that you can't find problems you aren't watching for.
I can't, but Claude almost regularly comes across "BTW things". I catch them because I read the lead agent's transcript, and have worker agents always add a retro section to their report, so I'm almost always seeing attention-worthy things pop up.
But think of Claude as being like a particularly keen outsourced developer: great at the tactical stuff, but sometimes lacks common sense in weird and surprising ways, so you’ve still got to keep an eye on the work product.
Also maybe I'm a curmudgeon but I generally at least mark-one-eyeball all the tests it's generating and tell it when it's being extra dumb.
Just don’t post the unedited AI “bug report” as an issue please. A human should make sure it’s actually a real problem before dumping it on devs.
A common misunderstanding.
The # of bugs lurking in a codebase is finite. At some point, the only way to find more is introduce fresh bugs through new code.
What we're seeing now is a transition phase where pre-existing bugs are found (and hopefully fixed) at an increased rate.
That's what I was thinking as well. Wouldn't the majority of the bugs not also have existed in Postgres 18? The version that everyone is using atm...
There is a similar thing with AI detecting vulnerabilities. It's not that the vulnerability isn't real, but just that in most cases for these vulnerabilities to exist there also needs to be a very specific set of circumstances as well making the likelihood of the vulnerability to exist in the wild quite low.
That's the cost of trying to scale so quickly. You dont filter out the crap.
A nice side effect is that the best way to fix it is more AI.
It's a endless money loop.
I think these type of new techniques will ultimately create more jobs for humans. If we think the answer is just “AI” everyone will create their own Postgres with a lot more bugs and vulnerabilities.
"Explaining the number of reverts — features whose rollouts have now been postponed to next year's release, PostgreSQL 20 — Peter Eisentraut, veep at EDB, a PostgreSQL support and consultancy, said the reason could be at least in part attributed to the LLM code reviews used by developers."
Naturally, EDB is an AI shop (https://www.enterprisedb.com):
"Agents need direct, governed access to data at the source. EDB Postgres AI is built for that."
Of course, nothing ever has worked in PostgreSQL before AI!
Their marketing has to mention AI. Everyone has to in a business like that. I would not take it as evidence they are slop coders or hyping AI just on that basis.