California Pushes Bill to Regulate AI Use by Lawyers

There have been so many egregious misuses of AI by lawyers (such as citing non-existent cases hallucinated by AI), I guess this was only a matter of time before regulations appeared.

The California Senate has passed a bill that would require lawyers in the state to verify the accuracy of all materials generated using artificial intelligence, including case citations and other content submitted in court filings. The legislation, now headed to the State Assembly, is among the first of its kind to address AI use in legal practice at the state level. It also restricts arbitrators from relying on generative AI for decision-making or referencing AI-generated information outside the case record without notifying involved parties. The bill responds to growing concerns about AI “hallucinations” that have led to disciplinary actions against attorneys nationwide.

Under the proposed law, California lawyers must take “reasonable steps” to vet AI-generated content, correct false or biased material, and avoid inputting confidential or personally identifiable information into public AI tools. The bill also mandates that AI use must not result in unlawful discrimination. The measure aims to establish clear ethical boundaries as AI becomes more prevalent in legal workflows. Bill sponsors emphasized the importance of ensuring that legal decisions remain in human hands and that client confidentiality is protected.

This builds on existing California legal standards and mirrors rules already in place for judges and court staff. It also reflects lessons from recent cases, including a state appeals court decision that fined a lawyer $10,000 for submitting fabricated citations generated by AI. The bill is part of a broader legislative push in California to regulate AI’s societal impact, including new laws requiring companies to disclose how they plan to mitigate risks from advanced AI systems.

In my opinion, every legal professional should already be doing this, as is required under our professional responsibility rules. A comprehensive worldwide list of cases is available here from Damien Charlotin.