Can ChatGPT pass the Bar Exam?

We'll make a lawyer out of you yet ChatGPT

Shaun Conroy

Share this article...

• Follow us

A recent research study has shown that ChatGPT-3.5 was able to successfully clear the torts and evidence sections of a multistate bar exam with multiple-choice questions. The upgraded version of the AI model, GPT-4, also demonstrated high scores on the bar exam. In this article, we will explore the details of the study and answer the question How Did ChatGPT Perform in the Bar Exam?

ChatGPT and the Bar Exam

The bar exam is a rigorous test that assesses knowledge and reasoning ability through essays, performance tests, and multiple-choice questions. It is a standardized test used by many U.S. jurisdictions as part of their bar admission process. It consists of three parts that assess a candidate’s knowledge of common law principles, ability to apply legal principles to specific fact patterns, and ability to complete legal tasks.

This exam is designed to allow attorneys to be licensed in multiple states without having to retake the bar exam for each state. The passing score is set by each jurisdiction, and exam scores can be transferred to other states’ jurisdictions, subject to each jurisdiction’s rules and requirements.

GPT-3.5 and GPT-4, the OpenAI large language models that ChatGPT is built on, have the ability to accomplish difficult tasks that necessitate extensive legal knowledge, reading comprehension, and writing proficiency, to the point where they can meet the standard of human lawyers in almost all U.S. jurisdictions. The National Conference of Bar Examiners, which created the multiple-choice section, has noted that lawyers possess unique abilities that are acquired through schooling and practice, which artificial intelligence currently cannot equal.

How did ChatGPT perform in the Bar Exam?

Earlier this year research conducted by professors from Illinois Tech and Michigan State College of Law, ChatGPT 3.5 was able to achieve a passing rate in two categories of the multistate bar exam (MBE) and performed similarly to human test-takers in another category, demonstrating its general understanding of the legal domain. In another research conducted by OpenAI researchers itself. ChatGPT performed well on the bar exam and scored in the 90th percentile in that research, It got almost 76% of the multiple-choice questions correct, up from ChatGPT’s 50%, and outperformed the average human test-taker by more than 7%.

ChatGPT was capable of generating essay and performance test responses that were mostly pertinent and logically connected. The threshold for passing the bar exam varies from state to state, and in New York, a score of 266, around the 50th percentile, is required to pass. ChatGPT’s results suggest that the capabilities of large language models extend to complex tasks that demand significant legal knowledge, comprehension, and writing ability. These models meet the benchmark that applies to human lawyers in almost all jurisdictions in the US.