EVALUATION DESIGN

The innovation skills impact study was designed to gather as much useful information about the learning and application of the skills within the context of the Athena project (see West, 1999 for complete details). Assessment tools and processes were developed to be as non-intrusive as possible, to avoid interfering with product development schedules and to permit naturalistic observation of the flow of work. All team members were informed of the purpose of the impact study and were asked permission to be surveyed and interviewed. All responses were compiled in strict confidentiality. Twenty-eight individuals, consisting of the initial core Athena group, were followed for this case study.

FRAMEWORK FOR EVALUATING TRAINING

The Kirkpatrick (1998) 4-level program evaluation model was selected for this study based on its ease of use and acceptance in the assessment field. This model was supplemented with Phillips (1997) `return-on-investment' model to add a financial dimension to the evaluation. The five levels of evaluation used in this study are defined as:

Level 1 - Reaction A measure of participant satisfaction and intention to apply the skills.
Level 2 - Learning A measure of transfer of knowledge or development of skill or change in attitude.
Level 3 - Behavior A measure of change in on-the-job activity or application of the learning in real situations.
Level 4 - Results The business impact achieved by program participants as they successfully apply the skills (changes in output, quality, costs, time, customer satisfaction etc.).
Level 5 - Return-on-Investment Compares the monetary benefits of the program with the program costs.

TRAINING OBJECTIVES

Based on interviews with senior Athena project management staff, behavioral objectives were set for the impact study. These were based on observed shortcomings of previous design projects and indicated in what areas the team was expected to improve:

Design Review Meetings

• Less adversarial atmosphere.
  
  
• Increase in potential solutions generated for all design concerns raised.
  
  
• Greater group satisfaction with process.

Design Documentation

• Clearer documentation of group work and thinking.
  
  
• Clearer description of thinking process for pre-meeting distribution.

ASSESSMENT TOOLS AND PROCESS

The assessment tools for this study were custom-designed to generate data to determine if the training objectives were met and to fit with each of Kirkpatrick's and Phillips' five levels of evaluation. Assessment was done prior to training, immediately after training, throughout the development of the product, and finally as product development wound down.

1. PRE-TRAINING ASSESSMENT [PTA] FORM

In order to capture the group's impressions of the design review process as it stood prior to the innovation skills program, team members were asked to rate design review meetings they had attended in terms of 20 meeting elements including meeting management issues, interpersonal communication and climate issues, generation and analysis of ideas, and quality of outcome.

2. SIX THINKING HATS TRAINING ASSESSMENT [6HTA] FORM

Immediately following their training, team members were asked to assess the training and trainers, indicate whether they had retained the content of the training, and give their impression of the value of these skills to their work at MDS SCIEX, as well as a prediction of where and when they might use them.

3. SIX THINKING HATS MEETING OBSERVATION [6HMOB] FORM

During the five months following their training, the Athena team was required to use the new approach in their design review meetings. Approximately 50% of design review meetings were observed in terms of structure and adherence to the agenda, use of new terminology, tone and content of comments.

4. FIVE MONTH FOLLOW-UP INTERVIEW PROTOCOL

As the design review phase of the study wound down, each team member participated in a 20-minute interview about their attitudes toward and use of the innovation skills since their training. They were asked to rate the Athena design review meetings in the same way they had rated previous meetings prior to their training.

RESULTS AND INTERPRETATION:

Organizational learning experts suggest that in order to create lasting change within an organization, the change should not be implemented from the top, but rather introduced from the middle or lower levels, in projects where the new approaches are most valuable and relevant to staff (Senge, 1999). The changes should be cultivated carefully, allowing self-reinforcing processes free reign and clearing away potential change barriers. The innovation skills program successes may be due in large part to an attempt to take such an approach to introducing change: supporting one cross-functional group's efforts to improve its own effectiveness and efficiency.

Originally, senior Product Development staff introduced the Six Thinking Hats program to improve perceived shortcomings in the design review process. Of note was that this desire to improve was echoed by the remaining Product Development staff. Measured before the skills training, their average rating of 20 design review meeting elements ranged from "could use improvement" to "adequate". Both management and staff tended to agree on which were the problem elements (e.g., tolerance for all viewpoints, exploration of ideas, building constructively on each others' ideas, quality of decisions).

A shared commitment to the Six Thinking Hats program as a potential remedy was also clear by the end of the training days: 76 % of the group felt the techniques would improve their work and 86 % agreed with senior staff that the tools would improve the design review process. Management and staff consensus of both the need to improve and the relevance of the improvement strategy no doubt fueled the success of the innovation skills program at MDS SCIEX.

Organizational learning experts have found that deep and lasting change occurs only with personal growth, and the commitment of all group members to self-generated change as opposed to compliance to top-down demanded change. The Athena project development team began to manifest this by actively using the Six Thinking Hats skills over the five (5) months of intensive design review meetings observed for the impact study. During the first design review meetings, adherence to the Six Thinking Hats agendas was strictly enforced by the project manager, but soon most participants were also using the terminology to preface their contributions and to regulate each other's comments, often with good-natured humour. They would draw attention to violations of the agenda colour in use ("Hey, you're red hatting my green hat!"), ask to revisit an agenda colour already completed ("I just thought of another white hat piece of information we forgot to consider"), and not hesitate to move the pre-set agenda along ("I think we've exhausted the black hats for this one").

After five months of use on the Athena project, average ratings of the 20 design review meeting elements measured before the skills training increased substantially, often an entire satisfaction level. Selected pre-and post-program ratings are shown below in (Exhibit 2 (Meeting Efficiency) and Exhibit 3 (Atmosphere for Innovation)X). A full 90% agreed that their work had improved because of the new approach, 90 % thought their co-workers' work had improved, and 95% predicted they would use the skills in the next four months.

Exhibit 2. Group Ratings of Design Review Meeting Efficiency Before and After Innovation Skills Program

EXHIBIT X (two):Group Ratings of Design Review Atmosphere for Innovation Before and After Innovation Skills Program

Exhibit 3. Group Ratings of Design Review Atmosphere for Innovation Before and After Innovation Skills Program