The Flight Plan tool was designed to help test administrators overcome common obstacles to implementing CATs. For example, the process of tailoring ability estimates for each test taker often requires large item pools, including items that span a wide range of difficultly—especially when the testing population is expected to vary considerably on the targeted ability. A lack of item coverage at specific ability levels also increases the risk of item overexposure, which is exacerbated when those deficits occur at ability levels common among test takers.
Another challenge is, when a bank contains items with marginal psychometric properties, the CAT engine can take considerable time to reach an acceptable level of measurement precision.
Finally, some CATs adapt on both the examinee ability and on content constraints typically defined in a test blueprint. In these cases, the algorithm must pay attention to the psychometric properties of the items, and it also must use the metadata associated with the items to find the “best” items to meet both psychometric and test blueprint considerations.
Assessing the likelihood of such challenges occurring typically involves developing and running custom simulations via analytic packages such as R or SAS, requiring complex programming skills.