Multi-Rater Assessment and Feedback
Multi-rater feedback can provide individuals with the information and support needed to make positive behavior change in the workplace. The first major component of a multi-rater (also known as 360°) feedback system is an assessment instrument that collects ratings on behavioral items from multiple observers. For managers, raters typically include direct reports, peers, and upper management. The manager also conducts a self-assessment. The assessment instrument requires raters to evaluate observable behaviors that tap important competencies. The second major component is a feedback process that summarizes the input from different raters and helps the ratee interpret and act upon the feedback.
HumRRO has worked for over 10 years to develop and implement effective assessment and feedback processes. We employ a rigorous approach to the development of the assessment instrument, the feedback process, implementation policies, and organizational communication strategies.
HumRRO's approach builds on our work in the areas of competency modeling, assessment instrument development, and implementation. We customize the materials and approach to match the culture and priorities of the client organization. If a competency model is not already in place, HumRRO professionals work with the organization to develop a job-related, valid model that can serve as the basis for a wide array of human resource management programs. We have a pool of generic items, matched to competencies and organizational level, that can serve as a starting point for instrument development. This pool contains items appropriate to three different levels of management - supervisor, middle manager, and executive.
HumRRO's work on team leadership has also yielded competency models and item pools for team leaders. We are currently expanding that work to include team members. Such level-specific assessments allow for more fine-grained and role-appropriate feedback to ratees.
Candid performance feedback from peers, direct reports, and supervisors is generally difficult to obtain because sharing such feedback (particularly if it is not positive) can be risky. Because this is a highly sensitive process, the question of who owns the feedback is of critical importance. The feedback process must take into account the goals of the organization and the risks to the individuals involved. HumRRO explores these issues in detail with the client organization at the start of each project. If the feedback is to be used purely for development purposes, the individual owns the feedback and the ratings are typically not shared with the organization. If the purpose is performance appraisal, then the results must be shared with the organization in some fashion. The psychometric properties (e.g., reliability and validity) of the assessment instrument take on special importance when assessment results support personnel decisions such as awarding promotions or bonuses.
Our analyses suggest that multi-rater instruments often do not meet the psychometric standards required to make decisions based on an overall score, or based on a change in ratings from one year to the next. One way to hold managers accountable for behavior change without focusing on numerical assessment results is to build expectations around the manager's commitment to development after receiving the feedback.
The manner in which feedback is provided is also important. Written feedback reports are a standard part of the process, but it is often difficult for individuals to accept the feedback and embark on a meaningful development path without assistance. We strongly recommend that participants have the opportunity to work with an experienced coach to help them interpret the feedback and begin the action planning process. Other implementation decisions (many of which we do not have room to discuss here) involve how the assessment tool will be administered. HumRRO has the capability for paper-based and Web-based administration. In either mode, HumRRO provides information to raters to ensure that they understand the purpose of the ratings and how to avoid common rater errors.
Implementation of a multi-rater assessment and feedback process is a major undertaking for most organizations, and open communication among those involved is key to success. This should start at the beginning (e.g., identifying the goals of the process) and continue throughout development and implementation (e.g., designing the instrument and associated policies/procedures). Regular communication and involvement of employees who will participate in the process are necessary ingredients to successful implementation.
For more information, contact:
Dr. Beverly Dugan or Dr. Deirdre Knapp or Research Notes