Generative AI for content generation and automated scoring: no-code and low-code solutions

With the advancement of generative AI technology, such as GPT, test development organizations and language testing researchers have started to explore the potential of leveraging generative AI to automate aspects of test development. One example application is the automated generation of test contents, such as reading comprehension passages at target CEFR levels, multiple-choice questions, short-answer questions, and graphics. A second application is using prompt-engineering so that the large language model directly generates scores and feedback for essays. This bypasses the usual automated essay scoring approach wherein expert human raters provide ratings for a sample of essays and models are developed to predict human scores.

The purpose of this workshop is to provide an introduction to the large language models (LLMs) that are available for automating content generation and automated scoring of writing. We will introduce the best practices for writing prompts in ChatGPT, refining prompts iteratively to get the intended outcomes, and building scalable production solutions through API interactions. This means that we will (a) use freely available online interfaces such as ChatGPT and Claude.ai to try out prompts, and (b) use Python code, which we will provide, so that we can instruct large language models to complete tasks at scale, e.g., for hundreds of essays, rather than one at a time.

We will focus on applying Generative AI technology for the following tasks:

• Generate test content and test items
• Score student essays
• Provide qualitative feedback on student essays

Participants will be asked to apply different prompt engineering approaches (e.g. few-shot and zero-shot learning) to:

• Write reading comprehension passages at different CEFR levels
• Generate multiple-choice, cloze, and short-answer questions for reading comprehension passages
• Generate graphics as a visual support for reading comprehension passages
• Assign scores to student essays that are aligned to writing rubrics
• Discuss the best approaches to prompt engineering
• Critique each other’s prompts
• Compare the pros and cons of different LLMs
• Discuss the ethical implications of using LLM in test development

This workshop will be suitable for participants who have played with ChatGPT but who want to take their usage to a more professional level, and who have little or no experience of using Python or APIs. In advance of the workshop, participants will be expected to set up an account and payment method for OpenAI’s API access (instructions will be provided).

Speakers: Alistair Van Moere and Jing Wei

Alistair Van Moere

Jing Wei