Criterion-referencing compares people to an objective standard of performance or knowledge regardless of test form, time, and location by explicitly linking the passing standard to the purpose of the exam. Criterion-referenced standard setting is not strictly data-driven. Rather, it is based on the sound professional judgment of subject matter experts (SMEs).
Before beginning the standard-setting activity, HumRRO has SME participants take the test, so they can read the items in a context similar to test candidates. Next, SMEs think about a hypothetical person who performs just well enough on the job or in the course to be considered successful. Then, SMEs describe the performance level required to be able to just pass the test (i.e., just good enough to be certified or move onto the next level). This is the minimum standard required to be certified, licensed, considered for selection/promotion, or placed into an academic or professional level or development program. Test candidates that meet that criterion are traditionally referred to as Just Sufficiently Qualified (JSQ) Candidates, Borderline Candidates, or Minimally Qualified Candidates.
Once the performance level is defined, SMEs review the test content and make multiple independent rounds of judgments about what type of test score constitutes a JSQ level.
Between rounds, SMEs share their first judgments with each other and facilitators provide impact data, such as the percentage of all candidates who answered a selected-response test item correctly, the score of an essay (or constructed response item), or the estimated percent of students classified into the different
performance levels. When making judgments at the item-level, HumRRO consultants find that it is helpful to stop and share first judgments and impact data after every 10 to 15 items, to reduce the cognitive burden on SMEs who have to go back and consider why they judged each item the way they did.
The discussion and impact data are important to ensure that SMEs have a shared understanding of the JSQ level, which enhances their level of agreement. After the discussions are complete, the SMEs independently make a final judgment without further discussion. The analyst calculates a cut-score later and provides the recommendation to the policy-making body.