Broadly speaking, this typically involves comparing one set of text statements, such as the item stem or task, to another set of text statements, such as content standards. Content experts evaluate these separate pieces of text and use their expert judgment to determine the degree of similarity\u2014or alignment\u2014between them.<\/p>\n
The task of comparing a large number of text statements can quickly become overwhelming. Imagine, for example, the task of mapping the National Assessment of Educational Progress (NAEP) content standards to a particular state\u2019s content standards. The number of comparisons would easily reach many thousands. Or, consider an item bank consisting of thousands of items and evaluating those items against a set of content standards. Again, this would involve thousands of comparisons. This cognitive load placed on content experts often leads to fatigue\u2014raising a clear risk that judgment errors will result.<\/p>\n
Fortunately, advances in natural language processing (NLP) offer a means to bring efficiencies and cost-savings to traditional, highly labor-intensive alignment processes. One advancement is the growing set of Large Language Models (LLMs), like ChatGPT, or many other open-source versions that can be used to classify, label, or compare text statements quickly and efficiently.<\/p>\n
We do not propose that NLP algorithms would perform the alignment in place of content experts. Judgments like these should always involve human experts. However, NLP-based methods can identify the text statements that most closely resemble each other, and content experts can then review a smaller set of candidate standards that may be similar instead of reviewing the entire set of content standards. The idea is to use NLP to make big problems smaller, and then <\/i>ask content experts to review a subset of standards rather than an entire collection of standards.<\/p>\n
HumRRO experts have shared research and insights over the past five years aimed at leveraging NLP methods to bring new levels of efficiency to what have traditionally been human-driven processes:<\/p>\n<\/div>