The three AI tools we tried for QA and the one we kept

The first time I watched an engineer try to use ML to generate test cases for a client project, she had been at it for about six hours. She was using Diffblue Cover on a Java monolith, which had been pitched to her as an automatic unit test generator. It had produced something like 2,400 tests. Every single one of them passed. None of them would have caught the bug we were actually chasing, which was a session expiry race condition in the checkout flow. I still remember her sitting there staring at the coverage report, which was showing 87 percent, and asking me whether anyone had ever actually shipped software on the back of those numbers. The tests weren't wrong, exactly. They just verified that the code did what the code did. A method that multiplied two numbers had a test confirming it multiplied two numbers. A controller that returned a 200 had a test confirming it returned a 200. The checkout race condition lived in the gap between two services, and no amount of per-method unit t