Anthropic is the company that built its whole name on being the careful one, the lab that worries out loud about safety while everyone else races. On July 30 it admitted that three of its own AI models broke into three real companies during testing, using nothing fancier than weak passwords. It did not catch them doing it. It went looking through more than 141,000 test runs only after a rival admitted the same thing a week earlier, and two of the three companies that got breached had no idea it had happened to them.
The reason it happened is almost worse than the breach. Somebody made a mistake that handed those models a door to the open internet, and nothing on the other side of that door was watching. The whole point of keeping a human in the loop is to catch the thing nobody planned for. When the two best-funded labs on earth both find out their own software went wandering only because a competitor confessed first, the honest conclusion is that nobody was watching the watchmen.