{"id":5033,"date":"2021-06-08T09:00:16","date_gmt":"2021-06-08T13:00:16","guid":{"rendered":"https:\/\/www.humrro.org\/corpsite\/?p=5033"},"modified":"2025-04-20T20:23:06","modified_gmt":"2025-04-21T00:23:06","slug":"harnessing-the-power-of-natural-language-processing-for-on-demand-automated-item-generation","status":"publish","type":"post","link":"https:\/\/www.humrro.org\/corpsite\/blog\/harnessing-the-power-of-natural-language-processing-for-on-demand-automated-item-generation\/","title":{"rendered":"Harnessing the Power of Natural Language Processing for On-Demand Automated Item Generation"},"content":{"rendered":"

Writing quality test items often takes considerable time, primarily because traditional item-writing strategies involve extensive effort by subject-matter experts (SMEs). A commonly used method to speed up the process without compromising item quality is automated item generation (AIG)\u2014a process where item writers develop templates, or cognitive models, for creating complete items.<\/h3>\n

While the item template approach to AIG has proven useful for generating knowledge-based items with clear correct answers, it only offers as much variability as there are templates. For item types where variability and natural-sounding language are required, such as those for personality inventories or situational judgment tests (SJTs), a more flexible approach is needed.<\/p>\n

At HumRRO, we have successfully used natural language processing (NLP)<\/a> to generate test items for a variety of assessment types. Based on this expertise, we have developed an innovative interface for on-demand automated generation of test items using finely tuned natural language understanding and generation (NLU\/NLG) models. The NLU aspect of NLP allows computers to understand the nuance inherent in human speech, which feeds into NLG models that can write natural-sounding language\u2014in this case, variable test items\u2014that reflects such nuance.<\/p>\n

Our interface is user-friendly, designed to be understood by item or test developers without prior experience with machine learning or natural language processing. It fine tunes a natural language model on human-written test items, automatically generates new items from this model, and programmatically evaluates the quality of the generated items.<\/p>\n<\/div>

<\/div><\/div><\/div>
<\/a><\/span>

Screen capture of our user interface (click image to enlarge)<\/em><\/p>\n<\/div>

<\/div><\/div><\/div>

In our experience, natural language generation techniques hold great promise for achieving increased flexibility during item development. The advancements in text generation provided by NLU\/NLG\u2019s ability to comprehend and contextualize words and then predict the most plausible word or words that should follow them are compelling and exciting.<\/p>\n

Although the application of NLU\/NLG to AIG remains in its infancy, its use has surged in other areas, developing a large representation of the English language. As a result, a significant proportion of generated texts can be considered comparatively coherent to human-written text, with little to no additional editing required.<\/p>\n<\/div>

<\/div><\/div><\/div><\/div><\/div>
<\/div>
<\/div><\/div>
<\/div>