Two Teams Work With Maryland State Technology Grant

HumRRO used two areas of expertise in its work with the Maryland State Department of Education: (a) development of performance assessments and (b) program evaluation.

Two HumRRO teams worked with the Maryland State Department of Education and its Consortium of approximately 15 partners on its catalyst grant, which is part of the U.S. Department of Education's program, Preparing Tomorrow's Teachers to Use Technology (PT3). The purpose of the federal grant program is to ensure that teacher candidates are prepared to use technology in the classrooms for teaching and learning.

One HumRRO team actively participated in the Consortium and assisted members in the development of performance assessment tools based on the Maryland Teacher Technology Standards. This involved identifying and structuring appropriate performance assessment tasks, scoring rubrics, and benchmark performance samples. Three of the seven standards were refined during the first year and pilot tested during the second in the first semester of the 2000-2001 school year. The remaining four standards were completed in the second year and piloted during the third year of the grant. HumRRO handled preparation of all pilot test materials including brief paper-and-pencil surveys to collect data. A critical component of the work to ensure reliability of the performance tasks and their scoring was conducting shadow scoring sessions after each pilot test phase. HumRRO prepared all the materials and conducted both of these sessions, and then reported results back to the working groups for revisions. Changes to the final tasks prevented the Consortium from having a complete range of benchmark student products for each of the Maryland Teacher Technology Standards. HumRRO worked with the project director and manager and will work after the conclusion of the grant to assemble at least one exemplary product for each standard. The electronic portfolio component of the grant was completed in the second year through the work of individual Consortium members. A retreat each year of the project provided a concentrated working period to complete and polish the various undertakings.

The second HumRRO team conducted both formative and summative evaluation efforts on the grant activities and outcomes. The evaluation team attended all Consortium meetings. The formative evaluation assessed the implementation process: how the program was implemented, problems that were encountered and lessons that were learned. The evaluating team examined the documentation that had been kept from all meetings and retreats describing the processes that operated in this project and the consistency of participation and effort of the Consortium members. The first year's development activities certainly presented a challenging learning curve for the Consortium members, but it established a solid structure and format for the group to follow in the subsequent development. The process that worked so strongly for Maryland included monthly meetings scheduled well in advance, dedicated meeting times at a site away from any individual member's workplace, and an annual retreat that allowed for concentrated attention to completing one year's tasks and planning for the next.

The summative evaluation assessed the impact of the intervention with indicators of program effectiveness. An online survey was administered each year of the grant. Three versions were used to include all Consortium members, teachers receiving professional development in technology, and college students participating in courses that pilot tested the technology standards. All Consortium members completed at least 90 percent of their individual goals, which in turn supported Maryland's overall grant goals. The biggest payoff for the Consortium was seeing the adoption of the Maryland Teacher Technology Standards as part of the standard program approval process for each institution of higher education in the state.

Maryland was unique among the PT3 grantees in that it was the only state-level project that focused on institutionalizing the use of the technology standards across all the Maryland higher education institutions. Thus, its success in getting the standards adopted into the program approval process was of interest to several other states. The lead evaluator from HumRRO along with the Maryland grant project director and manager presented results from the three years of the grant at the annual PT3 grantees meeting, held in July 2002 in Washington, DC: Implementing and Sustaining the Maryland Teacher Technology Standards. To see the Maryland Teacher Technology Standards and other information from the grant, visit www.smcm.edu/msde-pt3.

For more information regarding the program evaluation, contact:
CATHY? or Research Notes

For more information regarding the performance assessment development, contact:
Dr. Sunny Sipes or Research Notes