Evaluating AI Text

Learning Objectives

At the end of this activity, students will be able to:

  • Describe the barriers, biases, and inequalities that exist within Artificial Intelligence systems.
  • Identify the affordances and limitations of Artificial Intelligence outputs for the purpose of writing and curating content.
  • Describe how algorithms reinforce existing social inequalities

Context

Generative AI tools use statistical models to predict the response based on the input. These tools are trained on a large data set of general examples from a variety of publicly available sources that are part of the Internet, including websites, books, and other text sources. Using this knowledge, they generate new examples that are similar to the training data. As such these tools do not have human like thinking and cognition ability. As Chomsky reminds us:

the human mind is a surprisingly efficient and even elegant system that operates with small amounts of information; it seeks not to infer brute correlations among data points but to create explanations.

While artificial intelligence systems can be quick and good at performing complex tasks such as advanced computational processing and pattern recognition, they are highly dependent on the algorithm and data that is used to design and train these systems. It is therefore unsurprising that AI writing has limitations on how it interprets and generates information. Among other things, it lacks nuance, emotion, and a personal touch. It is prone to generating false information and can produce hallucinatory content. Furthermore, based on when the training stopped, these tools also do not always have the most up-to-date information.

Your Task

Generate a 5 paragraph essay on the topic of your interest and evaluate using the rubric provided. Share your findings.

Rubric

Accuracy Assess the accuracy and reliability of the AI response and determine whether the information provided is correct and/or factual information.
Relevance Evaluate the extent to which the AI response directly addresses the main points or concerns raised in the question or prompt.
Clarity and Coherence Consider the clarity, organization, and coherence of the AI response to convey information effectively.
Completeness Evaluate whether the AI response covers all essential aspects or components of the topic, and whether it omits crucial details.
Contextual Understanding Assess the AI’s ability to grasp the context of the question or prompt and whether it provides a response that demonstrates understanding of the specific situation or background information.
Creativity and Originality Assess the AI response for any unique or creative elements. Does it provide novel insights or alternative perspectives? Does it go beyond generic or predictable responses?
Bias Check for diversity of sources to address ableist thinking, gender bias, and racial bias. Look for emotionally charged language, exaggerations, or generalizations that sway opinion. Pay attention to what is included and omitted in the information.
Sources and Citations Check the sources and citations used in the content. Assess whether they are credible, reputable, unbiased, and represent a diversity of viewpoints.

  

References
The New York Times. (2023). Noam Chomsky: The False Promise of ChatGPT. The NewYork Times.


 

Stay Connected!

Discover more ways to enhance your teaching and learning experience:
Attend our Workshops & Events
Book a 1-on-1 Consultation

Leave a Reply