In the vast realm of artificial intelligence, ChatGPT has made remarkable strides. Launched just last year, this AI marvel has accomplished feats that left the world in awe. From acing the MBA entrance exam at the prestigious University of Pennsylvania's Wharton School to outperforming many in the US medical exam, ChatGPT was rapidly ascending the ladder of intellectual prowess. It even triumphed in the Google Coding Interview for Level 3 Engineers, cementing its reputation as a brilliant digital mind.

However, in the ever-evolving landscape of artificial intelligence, one question loomed large – could ChatGPT conquer the formidable Indian Civil Services Examination conducted by the Union Public Service Commission (UPSC)? We, at Analytics India Magazine, were no strangers to ChatGPT's capabilities, having experimented with this AI wonder for a considerable time. Driven by curiosity, we embarked on an audacious quest to determine, "Can ChatGPT clear UPSC?"

The Challenge Begins

• The journey began with ChatGPT sensing the formidable task ahead even before facing the UPSC preliminary paper. The UPSC Civil Services Examination, renowned as one of the toughest in the world, posed an intricate web of questions across various domains. From Geography, Economy, History, and Ecology to General Science, current events of national and international significance, social development, and polity – the examination's scope was vast.
• ChatGPT's knowledge base, however, was limited to information up to September 2021. This limitation meant that it couldn't respond to questions related to current events. Despite this constraint, our expectations remained high. Yet, as we delved into the examination questions, the AI's limitations became evident.

The Reality Unfolds

• Out of the 100 questions from Question Paper 1 (Set A) of UPSC Prelims 2022, which had answers available on the web, ChatGPT managed to answer only 54 correctly. Considering that the cut-off score for general category candidates in 2021 was 87.54, it was apparent that ChatGPT had fallen short of the mark. Questions related to Economy and Geography, topics not bound by time constraints, received incorrect responses. Even a seemingly straightforward question on History stumped the AI. In some cases, ChatGPT introduced its own answer option, 'e,' despite it not being part of the original choices.
• The UPSC Civil Services Examination is a formidable challenge, attracting approximately 11.52 lakh candidates each year. Yet, only around 5% of them make it to the mains, with an even smaller fraction securing positions as Indian Administrative Officers (IAS).

Understanding ChatGPT's Limitations

• It is vital to recognize that ChatGPT has inherent limitations, as acknowledged by Sam Altman, CEO of OpenAI. Altman has pointed out that ChatGPT, while proficient in some areas, should not be relied upon for crucial tasks due to its limitations in robustness and truthfulness. It can create a misleading impression of expertise in certain domains, despite its shortcomings.
• Researcher Christian Terwiesch, in a white paper titled 'Would Chat GPT Get a Wharton MBA?' noted ChatGPT's proficiency in answering fundamental queries within its expertise but also highlighted occasional errors in middle-school-level calculations. Thus, while ChatGPT's achievements are impressive, it is essential to regard its performance in various exams, including the UPSC, as fascinating experiments rather than definitive assessments.

The Path Forward

• During our experimentation, we uncovered an intriguing aspect – ChatGPT's responsiveness to question phrasing. In some cases, a slight rephrasing of questions led to correct responses. However, we evaluated only ChatGPT's initial responses during this experiment. Given its performance on Question Paper 1, we did not proceed to Question Paper 2 and beyond.
• With all the UPSC question papers and answers available online, anyone can replicate this experiment or further extend it to explore ChatGPT's reasoning and analytical capabilities. The journey of ChatGPT through the UPSC examination serves as a testament to the evolving world of artificial intelligence, where boundaries continue to be pushed, and possibilities abound.
• As we navigate this ever-expanding landscape of AI and human-machine interaction, the story of ChatGPT's UPSC attempt reminds us that innovation and exploration are the driving forces behind progress. While ChatGPT may not have conquered the UPSC examination, it has undoubtedly sparked discussions, raised questions, and illuminated the path toward AI's future potential.
• This article sheds light on ChatGPT's bold attempt to conquer the UPSC Civil Services Examination, a journey marked by challenges and discoveries. It underscores the importance of understanding AI's limitations and the ongoing quest to harness its potential in our ever-evolving world.