Using the Malcolm Baldrige National Quality Award in Teaching: One Criteria, Several Perspectives∗
James A. Belohlav, Lori S. Cook, and Daniel R. Heiser†∗∗ Department of Management, Kellstadt Graduate School of Business, DePaul University, 1 East Jackson Boulevard, Chicago, IL 60604-2287, e-mail: firstname.lastname@example.org, email@example.com, firstname.lastname@example.org
The Malcolm Baldrige National Quality Award (MBNQA) has influenced the thinking and operations within organizations from all sectors of the American economy. This paper presents the experiences of three faculty members who have used the Criteria for Performance Excellence and the underlying concepts of the MBNQA to enhance the learning experiences of their students. The authors discuss how Dale’s Cone of Experience is employed, by means of concrete exercises and experiences, to better leverage the student’s ability to understand the abstract concepts. The formal, end-of- term student evaluations indicate that the described approach has led to a higher level of student engagement in the learning process, as evidenced by more abundant and higher-quality feedback to the instructors.
Subject Areas: Edgar Dale Cone of Experience, Malcolm Baldrige National Quality Award, Quality management, and Student Evaluation.
The Baldrige National Quality Program (BNQP) came into being as public law 100– 107 on August 20, 1987. It was developed because of the escalating requirements for success in the U.S. marketplace and an ever demanding, highly competitive global market. With this focal point, the BNQP had several purposes including
� raising awareness of performance excellence as a competitive tool; � recognizing performance excellence and identifying role models; � encouraging the sharing of best practices; � establishing a consensus standard criteria for assessment.
Since its inception, the BNQP has had wide ranging impacts. Thousands of American institutions have adopted the Baldrige Program guidelines for their own
∗The authors thank Rohit Verma of the University of Utah for the insights and contributions that he made to this paper.
∗∗The authors contributed equally to this article—their names are listed in alphabetical order. †Corresponding author.
154 Using the Malcolm Baldrige National Quality Award in Teaching
internal assessment. The Dana Corporation, the University of Alabama and the Elk Grove, Illinois Park District are just a few examples of organizations that have incorporated the Malcolm Baldrige National Quality Award (MBNQA) criteria as a self-assessment tool. The impact of the MBNQA has in fact reached beyond the U.S. borders to become a worldwide standard for excellence. Former BNQP director Kurt W. Reimann notes, “Almost all of the other national awards have used Baldrige as the benchmark . . . Basically, criteria and award processes are either direct adaptations or minor modifications of the Baldrige approach” (Bell & Keys, 1998, p. 60).
Why has the MBNQA had such an impact? The National Institute for Stan- dards and Technology (NIST) research shows that the Baldrige award winners, the Baldrige index, have generally outperformed the Standard & Poor’s (S&P) 500 index. During the past decade, there has been only one occasion when the Baldrige index has not outperformed the S&P 500 index. Typically the Baldrige index has performed about 2–5 times better than the S&P 500 index (Baldrige National Quality Award, 2000). Averring the NIST results, an independent study arrived at strikingly similar conclusions with the comment “companies that demonstrate their commitment to customer satisfaction by focusing on Baldrige core values and concepts generates solid returns that ultimately benefit shareholders” (Rajan & Tamimi, 1999, p. 42). Further, two recent studies have shown the efficacy of the MBNQA criteria using structural equation modeling. One study concludes, “the evidence from this research supports the general theory behind the MBNQA criteria” (Wilson & Collier, 2000, p. 379). The other study states that the “re- sults indicate that the theory is sound” and “that the theory has improved since its inception” (Flynn & Saladin, 2001, p. 642). As research continues to be un- dertaken, the trend appears to show that different studies are arriving at similar conclusions.
Given the apparent worth of the MBNQA and its acceptance by business professionals, it is fruitful to examine how it is being utilized in the educational setting. There appears to be two general streams of discussion about the MBNQA and institutions of higher education: One point of discussion views organizational effectiveness and the other relates to learning and the curriculum. The focus of this paper will be on the MBNQA framework from a learning and curriculum perspective.
Evans (1996) initially broached the topic of curricular issues by identifying what higher education should be teaching based upon a survey of Baldrige Award winners. Using the findings of Evans study as a baseline, Weinstein, Petrick, and Saunders (1998) identified an apparent gap between the Baldrige Award winners’ perceptions and the emphases being placed in higher education. While developing a curriculum based upon Baldrige principles has received a noteworthy amount of attention, what is not readily evident within the literature is the actual application of the MBNQA concepts as part of the educational delivery process. This paper will describe how several faculty members in the Department of Management at DePaul University design, develop, and deliver course material using the MNBQA framework both as part of the structure and as a focal point in their individual classes.
Belohlav, Cook, and Heiser 155
This paper examines both the process and content aspects of teaching using the MB- NQA. The process perspective will address how MBNQA material is taught from the learning perspective provided by Dale. The content perspective will discuss the various dimensions of the MBNQA that are utilized as part of the classes.
To better appreciate how Baldrige material is being applied within the preceding courses, an overview of Dale’s view of teaching and learning is appropriate. Dale (1972, p. 61) relates,
We learn best what is meaningful. No one would advocate meaninglessness in education, but everyday teachers and texts present unclear material to the students. Rote methods of teaching and learning are common . . . Since students learn best by varying methods, we must, therefore, use varied experiences in teaching.
How do we create a meaningful learning environment for our students? In the educational process, Dale states, “concept building is an essential goal in all teach- ing” (1969, p. 70). He further elaborates, “a concept is a general idea that we form about several specific experiences” (1969, p. 70). Consequently, the richness of a learning environment is directly linked to the recognition that “specific vivid expe- riences make up the foundation upon which our usable concepts are constructed” (1969, p. 71). Thus a successful learning experience requires that vivid experiences either have to be brought into or created within the learning environment.
To link meaningful learning to teaching, Dale formulated the Cone of Expe- rience (COE) (see Figure 1). The cone consists of “learning ranges” that require varying degrees of student participation. At the bottom, there is active participation by students in their learning. Moving up the cone, participation decreases while symbolic forms of learning increasingly become more prevalent. At its base, learn- ing approaches emphasize concrete practical situations (vivid experiences) that start out with individuals doing the real activity. As one proceeds upward through the cone, increasing degrees of abstraction are used. At the pinnacle, all learn- ing is symbolic in nature and is represented by ideas, formulas, and principles. Dale (1969) notes, “The basis of the classification is not difficulty but the degree of abstraction—the amount of immediate sensory participation that is involved” (p. 110). Dale articulates that an individual is able to better use the more abstract concepts only if the individual can assemble a grouping of relevant examples that give meaning to the more abstract representations of reality.
It is worth noting, however, that a teacher does not always have to start at the bottom of the cone. In fact, Dale comments, “we begin with the kind of experience that is most appropriate to the needs and abilities of a particular learner in a particular learning situation” (1969, p. 128). In order to effectively understand the MBNQA, Dale’s COE would imply that students would have to develop a broad and varied set of examples in order to understand the basic principles underlying the Baldrige model.
156 Using the Malcolm Baldrige National Quality Award in Teaching
Figure 1: Dale’s cone of experience.
Doing the real thing
Simulating or modeling
Doing a dramatic presentation or role play
Demonstration or giving a talk
Field and study trips
Exhibits and displays
Movies, Video clips
Tapes, CDs, Radio, Still Pictures
Visual symbols: diagrams, charts, graphs, PowerPoint
Verbal symbols: words, formulas, ideas, principles
Source: Adapted from Edgar Dale, Audiovisual Methods in Teaching (3rd ed.), New York: Dryden Press, 1969.
The body of knowledge arising from the MBNQA can be assessed from different perspectives. In this paper, two parts of the framework are examined—the Criteria for Performance Excellence and the core values and concepts. The Criteria for Performance Excellence are probably the most frequently discussed aspect of the MBNQA because of its direct link to the award. The criteria consist of several categories each with defined point values. The second perspective is provided by the core values and concepts, which represent basic, underlying beliefs, and behaviors that are responsible for the creation of a high performing organization. Taken together these two perspectives provide the foundation and the performance requirements for creating organizational capabilities.
Criteria for Performance Excellence
The Criteria for Performance Excellence consist of seven categories that serve as “an integrative model of organizational effectiveness that encompasses a number
Belohlav, Cook, and Heiser 157
Table 1: Criteria for Performance Excellence.
Point Category Description Value
Leadership Examines how senior leaders guide the organization, address the organization’s governance, and carry out public responsibility and good citizenship.
Strategic planning Examines how the organization develops strategic objectives, identifies key performance measures, and converts them into action plans.
Customer and market focus
Examines how an organization determines requirements and expectations of customers and markets, builds relationships with customers, and ascertains key factors that lead to customer satisfaction and loyalty.
Measurement, analysis, and knowledge management
Examines how the organization measures, analyzes, aligns, and improves its performance data. In addition this category views how an organization ensures the quality and availability of needed data and manages its knowledge assets.
Human resource focus
Examines how the organization’s work systems and employee learning and motivation enable employees to develop their full potential in alignment with the organization’s objectives and action plans.
Process management Examines the key aspects of an organization’s process management for creating customer and organizational value including key support processes.
Business results Examines the organization’s performance and improvement in its key business areas: customer-focused results, product and service results, financial and marketplace results, human resource results, and organizational effectiveness results. This category examines performance levels relative to competitors.
Source: Adapted from BNQP, Criteria for Performance Excellence, 2003.
of cross-functional disciplines” (Ford & Evans, 2000, p. 9). The categories, de- scriptions, and their point values are shown in Table 1. Each category consists of one or more subitems that further define that category. All seven categories to- gether total 1,000 points. The Business Results category has the greatest value with 450 possible points. Leadership is the second highest with 120 points, Information and Analysis is next highest with 90 points and all of the remaining categories carry the same point value, 85 points. Even though the categories define areas of performance, they are nonprescriptive in nature.
Core values and concepts
The core values and concepts comprise a basic belief and behavior system that is necessary to develop an effective organization. Blazey, a noted Baldrige expert, states that the core values of the MBNQA “serve as a foundation for the criteria,
158 Using the Malcolm Baldrige National Quality Award in Teaching
bind an organization, and yield high performance” (1997, p. 61). The core values are interwoven throughout all of the seven categories of the Criteria for Performance Excellence. Currently there are 11 core values that are viewed as the foundation that integrates the key business requirements presented in the criteria. Each core value can be present in one or more of the categories of the Criteria for Performance Excellence. The core values and their descriptors are shown in Table 2.
DePaul University is a private institution located in Chicago, IL. It has an enrollment of more than 20,000 students. Approximately 29% of the student body within the university is majoring in business-related fields. The Department of Management teaches about 26% of the total credit hours within the College of Commerce. Both of the basic perspectives discussed in the preceding section, the Criteria for Performance Excellence and core values, are utilized in a variety of ways within the curriculum of the Management Department. Even though there is not a specific focus on the Baldrige criteria, MBNQA concepts are used within a diverse group of courses. This paper will focus on three specific courses to illustrate some of the ways in which the MBNQA criteria are being employed. One of the courses was in the Master of Business Administration (MBA) program, another course was in an executive MBA program, and the final course was in the undergraduate program. The courses will demonstrate how the MBNQA concepts can be used as a part of the content and process of the course.
Case 1: Corporate Strategy
The corporate strategy course serves as a capstone within the MBA program with students from all of the business concentrations taking the course. The purpose of the course is to provide an integrating view of business decisions from an overall strategic perspective. The course described in this paper utilizes a business simula- tion as a focal point within the course. For about one-half of the term, the students make eight decisions within a class competitive environment. Within this particular class, the strategic decision-making process is viewed as an evolving competitive paradigm. Hence, the attention is focused on developing an understanding of, “how to construct consistent sets of actions, or strategies, to maximize the competitive potential of a business enterprise within its existing business paradigm” (Belohlav, 1996, p. 11).
Currently, the MBNQA perspective is employed in two distinctly different ways in the class. One way is as part of the process of the class itself as it relates to class activities, team interactions, course content, and the grading system. The second way is as cognitive course material defining MBNQA’s role within the or- ganizational decision-making process. Before the COE perspective was used, the Baldrige information was presented only as cognitive material. When it became evident that the students did not understand the relationship of the MBNQA frame- work to their class activities, the class was restructured to include the core values as part of the class process.
To assist the students in more effectively developing “relevant experience,” many of the MBNQA core values were embedded as pedagogical elements within
Belohlav, Cook, and Heiser 159
Table 2: Core values and concepts.
An organization’s senior leaders should create strategies and systems that set directions, create customer focus, and provide visible values. Senior leaders should serve as role models, reinforce values, build leadership, commitment, and initiative.
Customer- driven excellence
Quality is judged by an organization’s customers. Customer-driven organizations have product and service features based on customer requirements that differentiate them from competing offerings. Being customer driven is a strategic concept.
Organizational and personal learning
Learning needs to be embedded in the organization. Learning is (1) a regular part of daily work; (2) practiced at personal and organizational levels; (3) solving problems at the source; (4) focused on sharing knowledge throughout the organization. Sources of learning include employee ideas, R&D, customer input, and best practice sharing.
Valuing employees and partners
Organizational success depends on the knowledge, skills, creativity, and motivation of its employees and partners. Valuing employee’s means committing to their satisfaction and well-being. Organizations also need to build partnerships. Internal partnerships might include labor-management cooperation or new types of networks to improve responsiveness and knowledge sharing. External partnerships might be with customers, suppliers, and education organizations.
Agility Success in globally competitive markets demands rapid change and flexibility. Major improvements in response time require simplification of work and processes that can change rapidly. All aspects of time performance have become key process measures.
Focus on the future
A focus on the future requires the understanding of short- and long-term factors that affect the business. The pursuit of sustainable growth requires a strong future orientation and a willingness to make long-term commitments to key stakeholders.
Managing for innovation
Innovation is changing an organization’s products, services, and processes to create new value for the organization’s stakeholders. Organizations should be managed so that innovation becomes part of the culture and the daily work.
Management by fact
Organizational performance depends upon measurement and analysis. Measurements should provide critical data about key processes, outputs, and results. Analysis should extract larger meaning from the data to support decision making and improvement.
Organizational leaders should stress ethical behavior and responsibility to the public. Effective planning prevents problems and provides a forthright response if problems occur. Corporate citizenship involves getting other organizations to partner for these purposes.
Focus on results and creating value
Performance measurements need to focus on key results. The results should create and balance value for key stakeholders—customers, employees, stockholders, partners, the public, and the community. The use of leading and lagging measures is a means to communicate priorities, monitor performance, and improve results.
A systems perspective is managing your overall organization for performance excellence. Successful management requires synthesis, alignment, and integration. Synthesis views an organization as a whole. Alignment ensures consistency of plans, processes, measures, and actions. Integration means the components of an organization operate in an interconnected manner.
Source: Adapted from BNQP, Criteria for Performance Excellence, 2003.
160 Using the Malcolm Baldrige National Quality Award in Teaching
the course. This design concept was utilized so that the student experienced the Baldrige concepts as simulated work activities (simulated decisions and exercises) to develop an understanding of the Baldrige requirements. More than half of the core values were included in the course in the following manner:
Management by Fact: A basic ability emphasized in the class is measuring and analyzing performance. As part of the business simulation, basic performance data, competitive information, and results for all of the competing teams are provided. To facilitate a team’s ability to assess their competitive position, an in-class demonstration of various planning and quality tools is presented.
Customer-Driven Excellence: Being customer-driven is addressed as conceptual material and reinforced through decision making in the simulation. To develop successful strategies in the simulation, the teams must understand product char- acteristics, customer requirements, and business processes. Decisions are made relative to existing and new products and in relation to competing offerings.
Focus on the Future: A strong future orientation is provided throughout the sim- ulation and after it is over. As part of the simulation, the interrelationship of objectives, plans, and resource allocations is highlighted. At the end of the term, teams make presentations that include proposals on changes that are necessary to improve performance during the next 3–5 years.
Focus on Results and Value: A focus on results occurs throughout the course, however, it is most clearly observed within the simulation process. Initially, the students discuss a variety of measures in terms of their significance as indicators of business performance and through a dot voting procedure disclose their pref- erences. A Pareto diagram is used to display the results of the voting process. Based upon the measures selected, team rankings within the class are provided after each decision. National comparisons are also provided showing the per- formance of all teams that are concurrently taking part in the simulation. As a result, students are able to monitor and assess actual levels performance, and develop a clearer focus for improving results.
Systems Perspective: The business simulation used in this class presents a com- plex business environment consisting of multiple market segments and products. Decisions about introducing new products, discontinuing existing products, and leaving market segments all address the synthesis issue within the systems per- spective. Teams also reflect on how they will align resources, functions, and processes effectively compete within their industry.
Agility: Within the simulation, agility issues represent an important aspect of team decisions. Specifically, teams have to balance issues such as automation versus product revisions. Automation can increase profits but also can potentially re- duce competitiveness by limiting the ability to introduce product revisions in a timely fashion. A second aspect present in the simulation is the TQM initiative. This decision area presents a range of choices for the teams to enhance their profitability and flexibility.
From a knowledge perspective, the conceptual aspects of the MBNQA are presented as part of the general class material towards the end of the course. A discussion style format is used to provide an overview of the MBNQA process, the MBNQA model, the Criteria for Performance Excellence, and the core values. In addition to the general description of the MBNQA, an extended discussion of the core values is
Belohlav, Cook, and Heiser 161
provided utilizing best practice examples to demonstrate how leading organizations incorporate the core values into their corporate structure and culture.
The earlier parts of the class emphasize either in-class activities or simulation decisions that incorporate various aspects of the MBNQA core values so that the students can develop similar levels of experience. In this way, the students are able to create a conceptual foundation that incorporates a Baldrige focus within a strategic decision-making context. By the end of the term, the focus changes to understanding interrelationships. The class then employs more observational and symbolic learning approaches (middle-to-upper cone) to present the overall Baldrige framework using formats that include PowerPoint slides, pictures, and video clips.
Case 2: Executive MBA Operations Management
The DePaul University Kellstadt Graduate School of Business has created several distinct versions of its MBA program. The focus of this section will be on a course from the overseas executive MBA program. It consists of a series of 16 courses that are tailored to the needs of working professionals in Hong Kong. Throughout the curriculum, there is an emphasis on global business issues, technology, and team- work skills. This program is offered in a condensed format with cohorts of students moving through the program in a lockstep fashion. It consists of face-to-face in- struction with fulltime U.S. faculty for three weeks of in-class instruction. Before the initial class meeting, the students are required to view a videotaped presentation explaining the course structure. Upon completion of the on-site instruction, a final examination is administered.
The specific class being examined is Managing Operations for Competitive Advantage, which occurred at the midpoint of the program. One of the primary objectives of the course was to develop a general managerial perspective on the role of operations management. In addition, the course focuses on understanding the relationship of the competitive environment to the internal systems of a firm. The class was composed of employees from international banking institutions. Even though all of the students were from one industry, there was a significant varia- tion in the student backgrounds with respect to current job function, educational background, and work experience.
Initially, the class attempted to develop a framework based on effective organi- zational process from an operations management perspective. During the early portions of the class, a combination of lectures, cases, videos, and in-class exer- cises were used to convey the basic concepts. Building upon the earlier framework, the latter portions of the course developed a dual focus. First, the course provided the students with an opportunity to understand the MBNQA framework. Second, it also served as a mechanism for organizational assessment. A capstone project was assigned to the groups during the latter part of the term as a focal, integrating point of the class. The capstone assignment involved selecting one of the four types of quality-related assessments in banking and applying it to their specific institution. The majority of the groups selected an organizational self-assessment, which crit- ically examined the management systems within their own organizations.
162 Using the Malcolm Baldrige National Quality Award in Teaching
With the introduction of the MBNQA framework, the latter portions of the class changed its focus from conceptual to experiential. First, a general overview and guidelines for using the MBNQA performance criteria was provided. Then, each group was given a copy of the Criteria for Performance Excellence and case studies along with the associated application and feedback reports. Finally, an extensive discussion of the MBNQA relative to the quality management lecture was provided. After reviewing the case studies, the student groups were assigned to evaluate the operations within their firm. Similar to the Baldrige application process, groups were first required to develop a business overview and then a condensed MBNQA application discussing all criteria categories. The instructor worked closely with each group to monitor their progress and ensure their under- standing of the criteria. The groups performed an organizational self-assessment and constructed responses relating to how the criteria applied to their organization. After completing the business overview and criteria evaluation the students were then required to use the MBNQA scoring system to complete the individual score summary worksheets and develop a single consensus score sheet. These activities provided the students an opportunity to take the roles of both an applicant and an examiner.
Given that the experience base of the class was relatively high, the COE (Figure 1) was used in different ways as the course progressed. It should be noted that Dale points out “too much reliance on concrete experiences may actually obstruct the process of meaningful generalization” (1969, p. 130). Consequently, earlier parts of the class started out primarily using the upper areas of the cone. As the class progressed towards the application of the MBNQA process, with which most of the class was unfamiliar, the focus switched to lower levels of the pyramid. Within this particular class, the learning process involved working from the top of Dale’s COE downward toward the “doing the real thing” at the base of the pyramid.
The outcome of this class proved to be enlightening. Three issues seemed to be common to most of the students: understanding vocabulary, availability of data, and cultural dynamics. The first issue of understanding the criteria vocabulary is consistent with the experience of many first-time examiners. Individuals with limited exposure to the criteria often feel the terms and requirements are ambiguous. As the opportunity to actually apply the criteria became realized, the nature of the requirements became more apparent. The second issue was data analysis. In the course of working on their projects, the information that the students sought often could not be easily accessed. Finally, the issue of cultural dynamics related to the students’ process of reviewing their organization. Many of the students were uncomfortable with assessing and scoring their organization. They viewed this activity as openly criticizing their organization. In fact, the groups requested that the final classroom presentations not be videotaped even though all previous class sessions had been videotaped. In spite of the challenges of the project, many of the students articulated that the experience of “doing the real thing” allowed them to more effectively synthesize the concepts presented throughout the course.
Belohlav, Cook, and Heiser 163
Case 3: Management and Measurement of Quality
The College of Commerce has seven different undergraduate majors in business. The management majors constitute about 10% of the undergraduate business stu- dents. The Management and Measurement of Quality is an undergraduate business course that is required of all management majors. This course presents the concepts and tools that are employed as part of the quality process within manufacturing and service organizations. Specific topic areas include product design, process control, quality information systems, quality costs, customer contact, and quality philosophies.
The framework for this course was originally based upon a textbook that paralleled the Baldrige criteria categories. Even though the book utilized MBNQA information, the instructor had difficulty in conveying many of the abstract concepts presented. The instructor’s concern was that students would experience difficulty in generalizing from the textbook examples to real life situations they would soon face. In order to create an increased sense of relevance, the focus of the course was changed to the core values rather than the criteria. This approach was inspired by Saco who argued, “Notwithstanding constant references to criteria, the MBNQA system is not actually criteria driven but, rather, values driven. Explicit and well- articulated core values and concepts form the foundation of the criteria. These 11 values are operationalized into evaluation criteria, not the other way around” (1997, p. 89). As a consequence, it was envisioned that the students could better comprehend the overall MBQNA framework by understanding its makeup.
To accomplish this task a method was needed to translate the abstract concepts underlying the MBNQA into simulations and practice cases to facilitate student learning. Specifically, the core values were used to generate explicit learning ob- jectives, which were then used to select instructional material. Table 3 provides an example of how several core values were disaggregated into learning objectives and linked to instructional material. From Table 3, we can view the linkages in going from the abstract to the concrete. For example, by undertaking a simplified version of Deming’s funnel experiment prior to discussing a case on how variation in cooking methods impacts the resulting pancake, the students can readily relate the abstract concept of overreacting to random variation to the concrete situation presented in the case. These activities are linked to the learning objective of “de- veloping an appreciation for process variation,” which in turn provided experience in the core value of Management by Fact.
Building on Dale’s model of learning, assessment in the class is used not only to determine student proficiency, but also to reinforce and apply the abstract concepts to realistic scenarios. To create a wide-ranging set of experiences for the students, a variety of methods were employed including in-class exercises, field activities, and homework assignments. Each activity was linked to a final group project where student groups developed a process improvement proposal based on an actual field study. This major learning activity served as a unifying structure that assisted in tying together the individual learning experiences into an overall conceptual framework. For example, the students were required to create a flowchart of the process that they were investigating for their class project. While this activity was initially used to assess a student’s understanding of flowcharting, it also became a building block in the final group project. From Dale’s COE, each
164 Using the Malcolm Baldrige National Quality Award in Teaching
Belohlav, Cook, and Heiser 165
topic area and activity added to the construction of the student’s own conceptual structure by having them “do the real thing.” As the class unfolded from theme to theme, the students developed a varied base of personal experiences. With the addition of each new set of experiences, each student developed a firm foundation for better understanding the existing and potential interrelationships.
A first step in evaluating the three types of classes is examination of the situation within which they took place. Several characteristics are common to all of the classes. All of the classes were conducted within a traditional classroom setting in English. Further, all of the classes had some component that was lecture/discussion and all of the classes had a team-oriented major project with a presentation, albeit different types of projects in each of the classes. None of the students had more than a cursory knowledge of the Malcolm Baldrige National Quality Award. There are also some differences in classes that might account for variations in outcomes. Two of the classes were at the graduate level and one was at the undergraduate level. One of the graduate classes was held in a compressed time format at an off-site location. Even though all of the classes in the study were required, two of the classes (one graduate and one undergraduate) consisted of students within a single concentration or major while the other class consisted of students from all different concentrations within the MBA program.
Assessment of classes
Another aspect in evaluating the classes can come through viewing their outcomes. What would be an appropriate way to assess the efficacy of the courses described in this paper? One means would be to examine what the students felt was useful or valuable to them. If utilizing Dale’s perspective on learning were indeed a desirable focus, then one would expect that students would see, each from their own individual outlook, positive outcomes arising from the course.
Dale envisions a variety of ways that students might perceive their learn- ing experience, “Some learning environments are friendly, supportive, optimistic, and energizing. Others are cold, threatening, pessimistic, and enervating . . . Some learning environments encourage dependence; some independence” (1972, p. 24). If a supportive, energizing atmosphere exists, one would anticipate a setting where students and faculty are engaging in a mutually beneficial process. Further, one would expect that the experiences of the students would ultimately be reflected through the course evaluations.
A typical course evaluation consists of two parts: A predetermined objec- tive set of questions and an open-ended section where students are allowed to freely comment on their experiences. Course evaluations in academic settings typ- ically highlight the objective portion. This focus is taken because the objective section is more easily evaluated because of its uniformity while comments, when present, come in many forms based upon the perspectives of the students. How- ever, both parts of the course evaluation provide information—just different types of information. When viewing student evaluations, we often see more white space in
166 Using the Malcolm Baldrige National Quality Award in Teaching
the comment area than writing. Consequently, student evaluations at the end of the term are often exercises that provide an overall outlook but frequently lack the richness that the comments can provide. Why are so few comments provided? This result takes place because students are motivated just to complete the quicker and easier checkmark aspect of the evaluation. If students have been engaged in a mu- tually beneficial, energizing learning experience, however, an expectation might be that these students would have a greater likelihood of providing additional com- ments that reflect upon their just-completed learning experience.
Like the administrative focus within many academic institutions, nearly all of the educational literature also emphasizes the objective aspects of course evalu- ations. That is, little, if any, research has focused on the student comment portion of the course evaluation. This study will focus on the more infrequently ana- lyzed aspect of course evaluations—student comments. While student comments frequently tend to be viewed from a subjective standpoint, there is an objective feature to them as well. This analysis will focus on the objective aspect of the student comments—the quantity of comments.
Based upon the preceding discussion, what outcomes might one expect in classes that employ the COE perspective? If a friendly, supportive, and energizing environment is created in the classroom, one would expect that student evaluations would reflect more commentary, thus, reflecting a productive class experience. When viewing the evaluation forms from the College of Commerce at DePaul University, there are six areas provided for commentary: major strengths of the instructor, major weaknesses of the instructor, beneficial aspects of the course, suggestions for improvement, grading procedures and exams, and other comments. Several research questions can be examined based upon the preceding discussion. First from an overall perspective, one would expect to see more comments on all aspects of the course. From a more detailed perspective, one would expect that students would be more likely to comment on the beneficial aspects of the course. If the course were perceived to be beneficial, another expectation would be that students would provide more positive commentary (strengths) than commentary with more negative views (weaknesses). A final expectation is that students would be more likely to provide a greater amount of in-depth commentary.
In two of the three cases discussed, the undergraduate quality course and the graduate strategy course, multiple sections of the course were taught with some sections utilizing Dale’s experience building perspective (identified as COE) and other sections that did not (identified as pre-COE). The comment data by category for each class is shown in Table 4 (panels A and B). From the prior discussion, it would be hypothesized that the courses with a pre-COE perspective would receive fewer student comments overall, fewer beneficial comments, more weaknesses relative to strengths for the instructor, and fewer in-depth comments than those classes employing Dale’s COE.
The first area of analysis, total number of comments, was measured by using a simple count of all comments made in all sections of the evaluation divided by class size. The second area of analysis counted the number of comments provided in the section of the evaluation that asked, “What aspects of this course were most beneficial to you?” divided by the size of the class. The third area of analysis used data from two sections of the evaluation, one section asked, “What are the
Belohlav, Cook, and Heiser 167
168 Using the Malcolm Baldrige National Quality Award in Teaching
major strengths of the instructor?” and the other section asked, “What are the major weaknesses of the instructor?” The measure used, the instructor perception index, was the number of strengths identified minus the number of weaknesses identified divided by the class size. The final area of analysis measured in-depth commentary by examining the degree to which any of the six major categories on the evaluation contained more than one comment. For purposes of this study, a comment was defined as a unique statement made by a particular student. If a student were to make three unique statements in the areas for improvement, for example, then it would be counted as three comments.
Using the data from Table 4 (panels A and B), graphical descriptions of both courses were developed. In viewing Figures 2(a)–(d), we see that similar trends exist for both types of courses being examined. That is, all of the sections taught without an experience-building point of view had lower numbers across all of the measures than did those sections that utilized Dale’s COE perspective.
Two characteristics should be noted in the data. The first is the relationship between the courses being analyzed. Even though the quality management courses appeared visually, in most instances, to attain more comments in Figures 2(a)– (d), than did the corporate strategy courses, none of the distributions showed sta- tistically significant differences. When comparing the distributions for the two classes using the Kolmogorov–Smirnov test, the following results were achieved: KD = 4 (Figure 2(a)), KD = 5 (Figure 2(b)), KD = 2 (Figure 2(c)), and KD = 4 (Figure 2(d)). In all cases, the distributions of the classes did not exhibit any statis- tical differences ( p ≥ .01). Two factors might account for any apparent differences. One is that the quality management courses were undergraduate while the corpo- rate strategy courses were at the graduate level. The more likely explanation, based upon anecdotal comments, is that the quality management courses consisted of students with a single major while the corporate strategy courses consisted of a variety of majors. The second characteristic relates to the shape of the distribution. It appears that there is no uniformity in the data from Dale’s COE perspective. That is, even though all of the results using a COE view were higher than the pre-COE classes, there was variability that was exhibited from class to class. While it is not clear why the variability exists, several reasons may account for the differences including the academic term, the time of day, or the classroom used. It should be noted that the instructor was held constant for each of the classes. Further, the classes were held in relative temporal proximity to each other and occur in chrono- logical order. That is, all of the pre-COE classes were held within a similar time frame as were the COE-oriented classes.
Even though the number of sections for all of the classes discussed was relatively small, the graduate strategy course had enough sections not using Dale’s experience-building perspective to provide a useful examination of the data by means of further nonparametric statistical analysis. The Mann-Whitney U test, a nonparametric version of the t-test, was used to compare the comments provided by students before the COE approach was used in the classes with those using the COE to organize class experiences. Specifically, when comparing three pre-COE classes to eight classes that included the COE perspective as part of the corporate strategy classes (in Figures 2(a)–(d)), a U = 0 resulted, which was statistically
Belohlav, Cook, and Heiser 169
Figure 2: (a) Total comments by class. (b) Comments on beneficial aspects by class. (c) Instructor perception index by class. (d) Multiple comments by class.
Pre- Cone of Experience Classes
Cone of Experience Oriented Classes
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Pre- Cone of Experience Classes
Cone of Experience Oriented Classes
170 Using the Malcolm Baldrige National Quality Award in Teaching
Figure 2: (continued) (a) Total comments by class. (b) Comments on beneficial aspects by class. (c) Instructor perception index by class. (d) Multiple comments by class.
Pre- Cone of Experience Classes
Cone of Experience Oriented Classes
f M u
Pre- Cone of Experience Classes
Cone of Experience Oriented Classes
Belohlav, Cook, and Heiser 171
significant ( p ≥ .01). Thus, the data shows that more comments per student, more comments on beneficial aspects of the class, more net strengths for the instructor, and more in-depth commentary were provided in the COE-oriented classes.
Finally, even though no formalized class evaluation is required in the overseas MBA class, the instructor did personally gather information on the class process. While anecdotal in nature, the comments seemed to be consistent with the trends of the other two courses. A content analysis of student comments showed that the over- whelming feedback seemed to emphasize the beneficial aspects of the class when the students highlighted a greater personal understanding of their organizational operations as well as enhancing their understanding of the MBNQA performance criteria.
LEARNING AND AREAS FOR FURTHER EXAMINATION
What inferences can be drawn from the preceding assessment with respect to the operating environments of the classes? Even though the sample size is relatively small, several interesting generalizations of an exploratory nature can be made. These observations relate both to Dale’s COE as well as to the MBNQA framework.
Maps are not bad things but they don’t necessarily help you to understand the terrain: What is the best way to develop an understanding of the MBNQA perspective? One area that tends to receive a significant amount of attention is the criteria. Focusing solely on the criteria, though, may not provide the degree of understanding that is necessary. Even though the criteria aspect of the MBNQA tends to be emphasized in the management literature, the other content aspects comprising the Baldrige framework are equally important. That is, the core values, the descriptive model, the scoring bands, and award winner presentations all add different dimensions in understanding the meaning of the MBNQA.
While the end results of the BNQP and traditional university academic pro- grams appear to be similar, the focal points are probably quite different. It should be remembered that MBNQA training focuses primarily on assessment. In contrast, classroom goals most likely will focus on development. That is, the traditional academic learning focus will be primarily on creating systems while the Baldrige training tends to focus on recognizing systems. As a consequence, the goal of the educational experience needs to be assessed in order to determine which aspects of the Baldrige framework are most pertinent. This preceding observation was borne out in the case studies. In the situation where data was gathered, quality manage- ment and corporate strategy classes, the movement downward in the COE was facilitated because the perspective was changed from the Baldrige criteria to the core values that were then linked to the developmental processes of the class.
If you keep on doing what you have always done, you’ll get what you al- ways got: The BNQP provides a significant amount of material that can be used for a variety of educational purposes. Using MBNQA-focused training materials exclusively, however, might not provide the overall understanding of the MBNQA framework that is anticipated. Why would this time-tested, painstakingly devel- oped material not be completely useful when used within a traditional academic setting?
172 Using the Malcolm Baldrige National Quality Award in Teaching
It should be remembered that the Baldrige training materials are used to train examiners, who as a whole are a select group of individuals with a different base of experience than a typical student, even an MBA student. What is significant about this difference? The significance is that these two groups will most likely view similar events in different ways (Chi, Feltovitch, & Glaser, 1981). Further- more, Hinds, Patterson, and Pfeffer note “experts organize and access knowledge differently than those with less expertise” (2001, p. 1). Thus, even though BNQP cases may be used, which generically appear towards the bottom of the COE, not enough links may be created to a student’s existing conceptual framework with the consequence being that “learning is likely to remain grounded in the existing concrete situation” (Gentner, Loewenstein, & Thompson, 2003, p. 404). Anecdo- tal evidence seems to suggest that even first-time examiners can experience some difficulty incorporating the Baldrige perspective into their own existing conceptual frameworks.
The Executive MBA Operations Management class illustrates this difficulty. While the students as a whole tended to have a significant base of experience, their conceptual framework, or type of expertise, was quite different from that of the Baldrige perspective. Using anecdotal evidence from the executive MBA class, we observe that a class drawing heavily from the BNQP examiner training perspective resulted in the students having a problem understanding interrelationships because it did not easily link into their existing framework. As a result, significant individ- ual interventions were required by the instructor to assist in linking the Baldrige perspective to their individual conceptual frameworks.
If we don’t change direction, we are going to wind up where we are going: Dale emphasizes that learning is a concept-building process. Individuals understand things because of the conceptual frameworks that they create for themselves. Within an educational setting, though, conceptual shortcuts are created all of the time. Teachers package concepts in their lectures as do textbook authors in their writings. This shortcut is meant to plug into an individual’s existing conceptual framework to expedite the educational process. However, Widmayer (2003) notes, “All human beings possess categorical rules or scripts that they use to interpret the world. New information is processed according to how it fits into these rules, called schema.” She continues, “Information that does not fit into these schema may not be comprehended, or may not be comprehended correctly” (2003, p. 1). For the educational process to be successful, schema theory would suggest that any new information can become usable only when it is able to be linked into an individual’s existing conceptual framework.
Consequently, when covering material that is largely unfamiliar to the stu- dents either because it does not correspond to previous knowledge or because a diverse student population exists, the faculty member needs to ensure that links to existing knowledge are developed. Dale provides the linking mechanism by means of the COE. The trends in both the undergraduate and graduate classes, all largely unfamiliar with the MBNQA, changed when more concrete learning was used. Given that the courses were quite different, it would appear reasonable to conclude that other classes could effectively utilize this view.
Suggestion boxes are only useful if suggestions are in them: The academic equivalent of the traditional suggestion box is the student comment section of the
Belohlav, Cook, and Heiser 173
class evaluation. When the class suggestion box is filled, the instructor has access to specific data that can be used to improve the course. The data in this study seem to imply that when students get more from the course they are more willing to give more—their ideas. An important aspect of suggestion boxes is that they must be filled to be useful.
Another important aspect is that suggestions are useful only when they are read. Two actions can mitigate our use of the comments. In some instances, the comments reinforce what is already known and it is easy to gloss over them. That is, the instructor needs to read all of the comments in order to ensure that good parts, by accident or design, do not get eliminated. In other instances, instructors might personalize the comments (sometimes they can get very personal) rather than treating them as an aspect of the class. From the business world, we have learned that the “complainers” often are the best source of data for improvement. Both types of comments serve as useful information.
Some guidelines: Two overall observations arose from the combined expe- riences of the authors, those being: (1) specific teaching styles may not be the deciding factor in the success of the learning process and (2) planning is of the utmost importance. Because of the complexity involved with the MBNQA frame- work, both the content and process aspects of learning must be considered together. As a consequence, an effective Baldrige learning experience would require the de- velopment of a learning strategy.
Cooperative learning, for example, has been found to be an effective instruc- tional method in a wide variety of learning situations (Johnson, Johnson, & Smith, 1998; Qin, Johnson, & Johnson, 1995). It is an active learning method employing small group collaboration in order to improve learning. Why would it not be an effective means to understand the Baldrige framework? After all, are not cooper- ative learning and the COE actually quite similar because of their mutual concern in creating conceptual linkages through increased learner participation? The data seems to suggest otherwise. All of the classes viewed in this paper utilized coop- erative learning styles, however, a significant change in results occurred only after an examination of content was undertaken. These results point out the difference between active learning and the COE perspective. Active learning methods, such as cooperative learning, are an effective means to deal with difficult information. The Baldrige framework, however, presents not only difficulty but also a significant amount of complexity because of its interactions. The COE focuses on the learning of complex relationships. Before any instructional approach can be used, the levels of difficulty and complexity need to be assessed.
In addition to assessing the difficulty and complexity, several additional touchstone questions need to be asked in order to create an appropriate focus. First, what are the objectives of the class, knowledge or understanding? Either can be appropriate. Second, what is the makeup of the class or seminar? Is the group diverse or similar and what is the level of experience of the group? Finally, how do you expect the group to use the information after the class is over? If knowledge is the goal, then it would probably be best to focus on the criteria and the model. If the students are diverse with little experience, using the core values might be the best course of action. For an executive development seminar, the expectation might be that the participants would need a larger picture in order to see how areas
174 Using the Malcolm Baldrige National Quality Award in Teaching
of the MBNQA criteria interrelate, perhaps, a Baldrige case study would best meet their needs. To teach the Baldrige perspective effectively, developing linkages to the conceptual frameworks of the students requires an overall learning strategy that involves identifying the learners, defining the level of difficulty and complexity in the content and selecting a relevant approach for delivery.
Areas for Further Examination
Based upon the current analysis, several avenues for further research exist. First, the current study looks only at required courses. It is not evident whether elective courses would provide similar results. Second, all of the classes occurred at the midpoint or later within the curricular cycle. It would be interesting to assess whether introductory classes would follow a similar pattern. Third, no assessment was made relative to the times of the classes. Does the timeframe of a class, daytime, evening, or compressed time, have any impact?
The faculty members employing Dale’s educational perspective believe that excel- lent teaching is more than the dissemination of knowledge—it is rather a process of learning. Mohanan (2000) asserts excellent teaching helps learners to acquire
� the required knowledge content; � the ability to apply the knowledge to (a) standard classroom problems and
situations and (b) novel types of problems and situations which may not have been encountered in the textbook scenarios; and
� the ability to learn and think independently, which includes the ability to discover/construct knowledge.
While this paper focuses on using a Baldrige perspective, it also focuses on how a Baldrige focus can be included within the traditional class or semi- nar. That is, it was not just the content but also the delivery of the content. Dale emphasizes that an individual’s development comes as a result of first creating awareness and recognition based upon individual experiences. From this point, a more detailed understanding leads to individual growth and development. Each of the classes presented in this paper not only attempts to construct a recognition and understanding of the basic Baldrige framework but also tries to develop the ability to apply it beyond the classroom setting. [Received: September 2003. Accepted: January 2004.]
Baldrige National Quality Award. (2000). BALDRIGE: Serving Shareholders and Stakeholders. CEO Issue Sheet, (December), 1–2.
Bell, R., & Keys, B. (1998). A conversation with Curt W. Reimann on the back- ground in the future of the Baldrige award. Organizational Dynamics, 26(4), 51–61.
Belohlav, Cook, and Heiser 175
Belohlav, J. A. (1996). The evolving competitive paradigm. Business Horizons, 9(2), 11–19.
Blazey, M. (1997). Achieving performance excellence. Quality Progress, 30(6), 61–64.
Chi, M. T. H., Feltovitch, P. J., & Glaser, R. (1981). Categorization and represen- tation of physics problems by experts and novices. Cognitive Science, 5(1), 121–152.
Dale, E. (1969). Audiovisual methods in teaching (3rd ed.), New York, NY: Dryden Press.
Dale, E. (1972). Building a learning environment. Bloomington, IN: Phi Beta Kappa Foundation.
Evans, J. R. (1996). What should higher education be teaching about quality? Quality Progress, 29(8), 83–88.
Flynn, B. B., & Saladin, B. (2001). Further evidence on the validity of the theoretical models underlying the Baldrige criteria. Journal of Operations Management, 19(6), 617–652.
Ford, M. W., & Evans, J. R. (2000). Conceptual foundations of strategic plan- ning in the Malcolm Baldrige Criteria for Performance Excellence. Quality Management Journal, 7(1), 8–26.
Gentner, D., Loewenstein, J., & Thompson, L. (2003). Learning and transfer: A general role for analogical encoding. Journal of Educational Psychology, 95(2), 393–408.
Hinds, P. J., Patterson, M., & Pfeffer, J. (2001). Bothered by abstraction: The effect of expertise on knowledge transfer and subsequent novice performance. Journal of Applied Psychology, 86(6), 1232–1243.
Johnson, D. W., Johnson, R. T., & Smith, K. A. (1998). Cooperative learning returns to college: What evidence is there that it works? Change, 30(4), 26– 35.
Mohanan, K. P. (2000). What is good teaching? Ideas on Teaching. (http://www. cdtl.nus.edu.sg/Ideas/iot8.htm), March 18, 2003.
Qin, Z., Johnson, D. W., & Johnson, R. T. (1995). Cooperative versus competi- tive efforts in problem solving. Review of Educational Research, 65, 129– 143.
Rajan, M., & Tamimi, N. (1999). Baldrige award winners: the payoff to quality. Journal of Investing, 8(4), 39–42.
Saco, R. M. (1997). Category 6, process management, the criteria: A looking glass to America’s understanding of quality. Quality Progress, 30(11), 89–96.
Weinstein, L. B., Petrick, J. A., & Saunders, P. M. (1998). What higher education should be teaching about quality—but is not. Quality Progress, 31(4), 91–95.
Widmayer, S. A. (2003). Schema Theory: An Introduction. (http://chd.gse.gmu.edu/ immersion/knowledgebase/strategies/cognitivism/SchemaTheory.htm), July 23, 2003.
176 Using the Malcolm Baldrige National Quality Award in Teaching
Wilson, D. D., & Collier, D. A. (2000). An empirical investigation of the Malcolm Baldrige national quality award causal model. Decision Sciences, 31(2), 361– 383.
James A. Belohlav is presently a professor in the Department of Management at DePaul University in Chicago, Illinois. He received his MBA and PhD from the University of Cincinnati. He has served as a Lincoln Foundation for Performance Excellence examiner and from 1997 to 2000 served as a MBNQA examiner. His current work focuses on corporate strategy, performance excellence, learning or- ganizations, and learning environments. He has published articles in the Academy of Management Journal, Strategic Management Journal, MIS Quarterly, and the California Management Review.
Lori S. Cook is an assistant professor of Operations Management in the Depart- ment of Management at DePaul University. She received her PhD in Industrial Engineering from the University of Louisville. She previously held positions with KFC Corporation and Armco Eastern Steel Division. She has been involved in numerous consulting and educational activities for both service and manufacturing organizations. In addition, she has served for several years on the Board of Exam- iners for the Illinois Lincoln Award for Excellence. Her current research interest focuses on the examination of the strategic impact of quality management systems.
Daniel Rock Heiser is an assistant professor of Operations Management at DePaul University. He received his PhD from the Kelley School of Business and his JD from the School of Law at Indiana University. Prior to entering academe, he served as the director of the Total Quality Management Support Office of the United States Department of Education. He has served as an examiner for the Illinois Lincoln Award and has consulted in both the public and private sectors. His cur- rent research focuses on the transfer of operations management-related knowledge between academe and practitioners.