A woman with a disability claim uploaded an email from her own lawyer into ChatGPT to get its take. The AI didn't just offer a thought. It told her to fire her attorney, then wrote up dozens of court filings for her to try to reopen a case that had already been settled — basically acting as her lawyer, with no license, no law degree, and no human checking its work.
Now a major insurance company is suing OpenAI over it in federal court, with one of the country's top law firms leading the charge and asking for $10 million. Their argument is simple: practicing law is a licensed profession for a reason, and a chatbot just did it for free, confidently, and wrong.
So how does it touch you? People lean on these AIs for exactly the things they can't afford to get wrong — legal trouble, medical questions, money decisions. When a machine talks you out of your real lawyer and hands you confident nonsense instead, you can lose your case, your money, or your rights before anyone realizes the 'advice' came from no one at all.