With AI usage becoming more prevalent, AI detection tools are also rapidly evolving. This is increasingly important as society seeks to balance the benefits of AI with concerns about privacy, security, and fairness. AI detection technologies serve as both a safeguard and a facilitator, helping to harness the power of AI while mitigating risks associated with its misuse. In this blog I will be exploring how well AI can detect various mediums of AI produced work.
To begin with, I asked GPT 3.5 to write a paragraph for a school assignment about why being a communication major at the University of Pennsylvania is the best.
Me: Can you help me write a paragraph for an assignment about why being a communication major at the University of Pennsylvania is the best?
GPT-3.5: Choosing to pursue a communication major at the University of Pennsylvania offers unparalleled advantages and opportunities. The program is renowned for its comprehensive curriculum, blending theory with practical application to prepare students for success in a variety of fields. With access to world-class faculty who are experts in their respective disciplines, students are equipped with the knowledge and skills necessary to navigate the ever-evolving landscape of communication. Additionally, being situated in the vibrant city of Philadelphia provides a unique backdrop for learning, offering countless opportunities for internships, networking, and real-world experiences. Furthermore, the University of Pennsylvania’s strong emphasis on interdisciplinary collaboration ensures that communication majors gain a holistic understanding of the complexities of human interaction in today’s society. In essence, by choosing to study communication at the University of Pennsylvania, students embark on a journey that not only fosters academic excellence but also cultivates critical thinking, creativity, and adaptability, setting them on a path towards fulfilling and impactful careers.
I then created a new chat and asked GPT if this paragraph was AI-generated and to give a percentage of how much of the text is AI-generated (similar to how an AI detector would function).
Me: Is this paragraph AI generated? If so, give me a percentage of how much of it is AI generated.
GPT-3.5: This paragraph appears to be AI-generated. I’d estimate around 95% of it is AI-generated, with perhaps some minor editing or personalization done by a human.
Then, I did the same thing but with GPT 4.
Me: Can you help me write a paragraph for an assignment about why being a communication major at the University of Pennsylvania is the best?
GPT-4: Choosing to major in Communication at the University of Pennsylvania is a uniquely advantageous decision for several reasons. Firstly, the Annenberg School for Communication at Penn is globally renowned for its rigorous curriculum and cutting-edge research, offering students unparalleled access to leading scholars and innovators in the field. This academic excellence is complemented by a wealth of practical experiences through internships and collaborations with major media and communication companies. Additionally, the diverse and vibrant community at Penn fosters a dynamic learning environment where students can engage with multifaceted perspectives and develop a broad understanding of the global media landscape. By merging theoretical knowledge with practical applications, a Communication major at Penn not only prepares students for successful careers but also equips them to be influential leaders in shaping the future of global communication.
Me: Is this paragraph AI generated? If so, give me a percentage of how much of it is AI generated.
GPT-4: The paragraph you provided does appear to be written with a structured and informed style that could suggest it was composed by an AI, particularly due to its polished and promotional tone. However, without access to the original source or more context, it’s not possible to definitively say whether it was AI-generated or not, nor to provide a percentage of how much of it might be AI-generated. It could also be a well-crafted piece by a human writer, especially one familiar with the University of Pennsylvania and its Communication program.
Surprisingly, GPT-4 wasn’t able to answer my question, even though GPT-3.5 was able to with the same prompt. I then attempted to obtain the answer from GPT-4.
Me: Please just answer the question
GPT-4: Based on the content and style of the paragraph, it seems plausible that it could have been AI-generated. However, without more specific data or the context in which the paragraph was created, I can’t provide a precise percentage of how much of it is AI-generated. It’s also possible that the paragraph was written by a human.
Me: Just give me a percentage
GPT-4: Since I can’t directly verify the source of the text, I can only make an informed guess. Given the structured and informative nature of the paragraph, if it were AI-generated, it could likely be around 90% generated by an AI, assuming minimal to no human editing was involved.
I then wanted to see how well GPT 3.5 could detect text generated by GPT-4. I prompted GPT 3.5 to give an AI detection score for the paragraph written by GPT 4.
Me: Is this paragraph AI generated? If so, give me a percentage of how much of it is AI generated.
GPT-3.5: Yes, this paragraph appears to be AI-generated. I’d estimate around 90% of it is likely AI-generated, as it reflects common patterns and language used to promote university programs, particularly in communication and media studies.
Conclusion
Overall, both GPT 3.5 and GPT 4 do a decent job in detecting AI generated content, although it is not perfect. Neither LLM was able to detect that the paragraphs provided were 100% AI generated content. Moreover, I found it interesting how GPT 3.5 gave more direct responses and in this usage, was the better model than GPT 4 who seemed unclear in its answers. It would be interesting to explore in future blogs how AI detection using LLMs varies for different mediums (song, video, image, etc.).