HumRRO-O*NET Study Reviews Methods for Measuring AI’s Impact on Work

“Indexing the Impact of AI within the O*NET System: A Review of Methods and Development of Recommendations” argues that current AI impact research would benefit from giving more consideration to well-established job analysis and job performance research. Co-authored by HumRRO’s Dan Putka, Ph.D., and Nathaniel Voss, Ph.D., and O*NET’s Phil Lewis, the report offers recommendations for developing AI impact indicators within O*NET, the nation’s primary source of occupational information.

Over the last several years, economists, computer scientists, and technology firms have produced a rapidly growing body of research on AI and work. Yet the authors found relatively little integration of the job analysis and job performance literature developed within I-O psychology—the scientific discipline that has spent decades studying what people do at work and the knowledge, skills, abilities, and behaviors required for effective job performance. The report reflects the authors’ broader belief that understanding AI’s impact will require greater collaboration across disciplines.

Over the last several years, economists, computer scientists, and technology firms have produced a rapidly growing body of research on AI and work. Yet the authors found relatively little integration of the job analysis and job performance literature developed within I-O psychology—the scientific discipline that has spent decades studying what people do at work and the knowledge, skills, abilities, and behaviors required for effective job performance. The report reflects the authors’ broader belief that understanding AI’s impact will require greater collaboration across disciplines.

“Our goal was not to challenge the value of existing approaches, but to examine them through an I-O lens and identify opportunities to build a more complete understanding of AI’s impact on work,” noted Putka.

The researchers reviewed 19 prominent AI impact studies and found that most focused heavily on tasks while giving comparatively less attention to broader models of job performance. According to the report, this task-centric focus may provide an incomplete picture of AI’s potential impact because effective job performance encompasses more than task execution alone. Contextual performance behaviors such as collaboration, helping others, organizational citizenship, as well as adaptive performance also play important roles in workplace success.

Building on the review, the authors proposed a suite of 16 AI impact indices that would allow O*NET users to examine AI’s relationship to work from multiple perspectives. The proposed framework evaluates AI’s impact on both the precursors of job performance (i.e., job knowledge and skills), as well as major elements of job performance (i.e., task and contextual performance), while also distinguishing between AI’s automation and augmentation potential.

“O*NET has long served as a trusted source of occupational information for workers, employers, educators, and policymakers,” said Putka. “As AI continues to reshape the workplace, it is essential that we explore rigorous, transparent, and scalable approaches for helping users understand where and how AI may influence work. This study contributes to a new perspective on the rapidly expanding body of literature in this area.”

The authors emphasize that their recommendations are intended to complement—not replace—existing AI impact research. Ultimately, the greatest benefit may come from blending the I-O psychology perspectives offered in this report with insights gained from labor economics and computer science reflected in the extant AI impact research.

Read the full paper: Indexing the Impact of AI within the O*NET System: A Review of Methods and Development of Recommend…