How accurate is ChatGPT in generating responses?

how much can you really rely on what Chat GPT responds with??

Shaun Conroy

Share this article...

• Follow us

ChatGPT, the AI chatbot developed by OpenAI, has been making waves in various fields for its accuracy and performance. However, it is important to note that its knowledge is limited to data available up until 2021. One of the tests of its accuracy was conducted by Professor Christian Terwiesch at the University of Pennsylvania, who put ChatGPT to the test by giving it an MBA exam.

The exam consisted of seven questions covering various business concepts such as process analysis, inventory turns, and queueing analysis. This was designed to simulate a real-life human environment, and the questions were asked exactly as they would be to a human student. To the surprise of Professor Terwiesch, ChatGPT began the exam with a bang, receiving an A+ for its answer to the first question on the concept of a bottleneck in process analysis.

Although the chatbot struggled with some of the more complex questions, the professor was still impressed with its overall performance. After conducting the MBA exam on ChatGPT, Professor Terwiesch concluded that the AI chatbot would have received a grade between a B and B-.

ChatGPT accuracy explained

Another demonstration of ChatGPT’s accuracy was in the medical field, where the chatbot was able to pass the US Medical Licensing Examination (USMLE). According to research, ChatGPT was able to achieve over 50% accuracy across all exams and even offered insights and explanations for its answers.

In a separate study led by doctors from medical start-up Ansible Health, ChatGPT performed at or near the passing threshold for all three exams required to become licensed as a doctor in the U.S. While ChatGPT’s accuracy is certainly remarkable, it’s important to keep in mind that its performance can vary depending on the context and prompt given.

The training data used to train the model includes the biases and limitations of the text sources used, so it’s crucial to use the model responsibly.

This sentiment was echoed by the CEO of OpenAI, who stated in a tweet that “ChatGPT is incredibly limited, and that “It’s a mistake to rely on it for anything important right now”.

READ NOW: Chat GPT vs Google

Test conversation with Chat GPT failed

As shown in the above snapshot, A test conversation with ChatGPT revealed that the AI chatbot’s performance is not always consistent.

When asked how many times Argentina won the FIFA World Cup, ChatGPT initially replied “Argentina has won the FIFA World Cup once, in 1986.” However, when asked “Who won the FIFA World Cup in 1978,” ChatGPT responded, “Argentina won the 1978 FIFA World Cup.

This led to the conclusion that Argentina had won the World Cup twice. Yet, when the same question was asked again, “Now I am asking you again how many times has Argentina won the World Cup?” ChatGPT gave the same answer, “Argentina has won the FIFA World Cup once, in 1986,” despite the previous conversation.

This highlights the accuracy of Chat GPT and the need for ongoing improvement and refinement.

READ NOW: Can Chat GPT write & fix code?