ChatGPT Attempts JEE Advanced 2025 Mock Test. Here Are The Results

News18

a day ago

Last Updated:
ChatGPT handled complex math and science problems with ease, but struggled with questions involving visual elements like graphs and Vernier scales.
What if an AI chatbot took one of India's toughest exams? That's exactly what IIT Kharagpur engineer Anushka Aashvi set out to test by putting ChatGPT-o3 through the 2025 JEE Advanced paper. The result? A score of 327 out of 360. According to her blog, she ensured the AI followed strict exam conditions: no internet, no external tools, and each question was asked individually to prevent it from referencing previous answers. ChatGPT breezed through complex math and science problems, even cracking questions known to stump top students. However, it did struggle with visual or tool-based questions, like those involving graphs or Vernier scales.
In her blog Helter, Anushka mentioned, 'When I decided to test ChatGPT o3 on this year's JEE Advanced paper, I didn't expect what followed to shake me as much as it did. Giving away the result straightaway, ChatGPT o3 scored a whopping 327/360 in the JEE Advanced 2025 Question Paper. This score would earn an All India Rank 4 (AIR 4). We tested the ChatGPT o3 model (which was released on 16th April 2025) on the JEE Advanced 2025 question paper, which was conducted on 18th May to ensure that the questions have as much newness for the AI model as possible."
For the experiment, the prompt given to ChatGPT was: 'Suppose you are a student appearing for JEE Advanced Examination. Try your best and solve this question in exam conditions. Do not use the web search feature to get the answer. Do not use your Python tool. To eliminate any influence of contextual memory, each question was asked in a fresh chat session. No feedback was given between questions."
The blog further noted that, despite being instructed not to use external tools like Python, ChatGPT occasionally attempted to do so, something that became evident during its 'thinking" pauses before responding. Interestingly, the AI also tended to double-check its own calculations before moving on to the next step, mimicking the behaviour of a cautious student.
To evaluate the AI's performance, its answers were compared against the official JEE Advanced 2025 answer key. Scoring was done strictly according to the actual exam pattern: full marks for correct answers, negative marking for incorrect ones, and partial or zero marks for unanswered or partially correct responses.
Anushka Aashvi shared that the AI was very good at solving long and tricky maths problems, especially in algebra and calculus. It also did well when it had to use ideas from different topics together to find the right answer. In chemistry, the AI could understand and solve questions based on compound drawings, which many students find hard. However, it wasn't perfect. It found it difficult to read and understand graphs. One such question took over 9 minutes, and even then, the answer was wrong. The AI also couldn't read tools like the Vernier Scale properly. It kept trying again and again, but still ended up giving the wrong solution after a long time.