HIEST seminar 8 May 2024: Tetiana Krushynska

Algorithmizing the development of multiple-choice questions

In RUU E314 Isa; to attend online please register here.

Presentation summary:
Tests with selected-response or multiple-choice questions (MCQs) are widely used nowadays, including national and international exams, because they are resource-efficient for measuring the performance of large groups of students. Properly composed MCQs share similar diagnostic characteristics with constructed-response questions. Because MCQs are organized in structured item banks, they provide precise curriculum coverage. Item banks require constant replenishment: MCQs that have been used in exams, should be replaced with new tasks, and progress in the knowledge area requires that test items be updated.

The process of MCQ development for replenishing a standardized item bank could be divided into constructing (or selecting) prototype tasks that meet learning outcomes, and then transforming the prototype into numerous variations. This is not a problem for fields such as mathematics, where “formula questions” and “word tasks” are common but generating test tasks in natural sciences is much more challenging. The development of multiple-choice questions could be facilitated by algorithmizing – implementing the sequence of defined actions that lead to the desired result of creating a necessary number of MCQs that have the required characteristics. In natural sciences, MCQs are usually textual. We already have quite sufficient AI resources for text analysis and transformation. Algorithmizing MCQ development aims to facilitate the interaction between task developers and AI tools.

The main research question is: What is the association between algorithmization in developing test tasks and quantitative and qualitative characteristics of produced test tasks for assessment in natural sciences education programs?

The method of the Realist Synthesis is applied. In this type of literature review, new knowledge is derived from previously published studies through theory building by collecting evidence of successful and failed configurations within the Mechanism-Context-Outcome system. The Mechanism is considered as the process of MCQ development. It includes the cooperation of different groups of developers (teachers, technical and subject experts), implementing guides and instructions, and the use of PC and AI. Learning subjects, educational programs, types of HE institutions, exam format, state regulation of exams, etc. are the components of Context where the Mechanism works. The Outcomes are expected to be a high-quality item bank created via productive AI-assisted work of task-developers, but limitations and side effects will also be defined.

Before analysis of relevant articles can begin, the concept of "algorithmization" in its application to task generation should be clarified, and a "necessary number" and "satisfactory quality" of items required for exam integrity should be defined. The Realist Synthesis is an iterative process that implies reasonable changes to the initial theory under the influence of revealed data and communication with experts. Therefore, all suggestions, comments, and critical remarks from peers would be valuable.

Presenter: Tetiana Krushynska

Presenter background: PhD in pedagogy, Grant Researcher at the Finnish Institute for Educational Research