ATF Job Analysis/Competency Studies and the Development of Promotion and Performance Evaluation Processes

Since the Fall of 1998, HumRRO has been working with the Bureau of Alcohol, Tobacco, Firearms, and Explosives (ATF) to design, develop, and implement several human capital programs. Many efforts were aimed to help ATF respond to a U.S. District Court directive which required that ATF implement a number of systemic changes in its human resource practices (e.g., promotions, training, details and other special assignments, awards, evaluations and discipline). Our work began with a comprehensive job analysis study of the GS/GM 1811 criminal investigator (special agent) positions and continued with the design and development of an assessment processes for promoting special agents to supervisory and managerial special agent positions and the design and development of a performance evaluation system for all special agents. Fairly recently, we also conducted an update of the supervisory and managerial special agent job analysis results. Our work on these efforts has led to several other activities including the development of an ATF-wide leadership competency model, a job analysis of the industry operations investigators, and a structured interview to select applicants to the SES candidate development program.

 

The special agent job analysis work involved job observations, incumbent interviews, and workshops to generate lists of tasks and knowledge, skills, abilities, and other characteristics (KSAOs). Separate task and KSAO questionnaires were administered and the results were analyzed to determine the critical job tasks, KSAOs, and competencies for each grade level. HumRRO conducted additional workshops to collect task/KSAO linkage data and critical incidents of job performance.

 

The job analysis results served as the foundation for the design and development of a performance appraisal system for all special agents (i.e., 26 positions) within ATF. HumRRO worked with subject matter experts to develop behaviorally-based rating scales for each grade level and job specialty. Then, HumRRO developed and administered comprehensive supervisor and train-the-trainer sessions.

 

The job analysis data also served as the foundation for the processes to evaluate candidates for promotion to the field supervisor (grade 14) and field manager (grade 15) special agent positions. We designed and developed comprehensive assessment center simulations to evaluate the KSAOs required in the target positions. Using job analysis data and working with subject matter experts, a content validation strategy was used to develop four simulations (e.g., role play, in-basket, analysis exercise) for each promotion process. Behaviorally-anchored rating scales were developed for each competency measured in the job simulations. Then, HumRRO prepared for and delivered comprehensive assessor training. HumRRO also managed and administered the implementation of both assessment centers, processed all candidate data and conducted extensive data analysis. In addition, candidate scores were computed and feedback reports were generated. All aspects of this effort were reviewed and approved by a court-appointed oversight committee.

 

In addition to the initial assessment center work, HumRRO developed new job simulations and administered and analyzed both processes on multiple occasions. This work is on-going.

 

Additional work for ATF has included (1) using the job analysis results as the framework for a leadership competency model applicable to all supervisors, managers, and executives throughout the agency, (2) updating the original grade 14 and grade 15 special agent job analysis results, (3) developing a structured interview to evaluate applicants to ATF’s SES candidate development program, and (4) conducting a comprehensive job analysis of the industry operations investigator (IOI) positions. HumRRO is currently involved in a new effort to identify the IOI skill gaps.