The writer sits on the Civil Justice Council and the UK Ministry of Justice’s senior data governance panel

ChatGPT, the technology that launched a thousand quandaries, has now reached the attention of the most senior judges in the UK. This month, the House of Lords constitution committee heard evidence from the president of the Supreme Court. He described the experience of a litigant in Manchester who relied on ChatGPT for legal advice and inadvertently made an entirely fictitious submission to the court. Similar experiences have been reported in the US.

Concerns about generative AI models returning accurate legal information have led even ardent proponents of the technology to conclude that it should only be used to augment, rather than replace, the advice of lawyers. This is a pragmatic approach, albeit one that undermines its potential to expand access to legal advice to the growing numbers of people who are unable to afford it. However, in the rush to this solution, those in charge of the legal system are missing an opportunity to interrogate and address why the tool is returning inaccurate advice in the first place.