|
AGENDA SETTING AND IMPROVEMENT MONITORING IN A UNIVERSITY DEPARTMENT
Igor Dubina Altai State University, Barnaul, Russia, Email: din@econ.asu.ru and Stuart Umpleby The George Washington University, Email: umpleby@gwu.edu
AbstractA Quality Improvement Priority Matrix (QIPM) was used by the members of the Department of Management Science at The George Washington University in the years 2001 to 2005 to consider priorities and to monitor progress. Using the importance/ performance ratio (IPR), the authors clustered the features of the Department by their IPR scores into four groups – urgent, high priority, medium priority and low priority. The paper suggests a number of new ways to interpret the data obtained from a QIPM.
Research Background A Quality Improvement Priority Matrix (QIPM) is a method for achieving data-driven decision-making. A QIPM asks customers or employees to rate several features of an organization on two scales – Importance and Performance. That is, how important to them is that particular feature, and how effectively is the organization currently performing on that feature. A Quality Improvement Priority Matrix was described by the specialists from GTE Directories in their presentation in February 1995 describing how they won the Baldrige Award. (Carlson, 1995) A similar matrix, called a ¡°strategic improvement matrix,¡± was used by the people from Armstrong Building Products Operations in their presentation to the February 1996 Baldrige Award conference. (Wellendorf, 1996) A QIPM is usually used in determining priorities and for monitoring performance improvement. The features of greatest interest are those that fall in the SE quadrant defined by high importance and low performance. Those features are considered to have priority for an organization. The matrix was used in several George Washington University (GWU) student group projects in the late 1990s. And a matrix was used by members of the GWU Department of Management Science in 2001, 2002, 2003, and 2005. The faculty in the Department of Management Science at GWU evaluated various features of the Department and the School of Business, such as Funds to support research, Salaries, Coordination with other departments, Classroom facilities, Travel support, Teaching assistants and so on (a total of 52 features). Although the Department is functioning very well, improvement is always possible. We tried to define where improvement is most needed. Thus, we studied how a Quality Improvement Priority Matrix may be used for improvement monitoring in a university department. We wanted to compare the data from 2001 to 2005 to see how feature priorities had changed during this period. The first questionnaire was distributed in May 2001. The 2001 questionnaire contained 51 features related to the Department and five questions about the matrix itself. Nineteen responses were received from faculty members. The five questions asked whether the members of the Department found the exercise to be useful and whether they thought it would be helpful to other departments in the University. A large majority thought the results were useful and that similar exercises in other departments would be helpful as well. (Umpleby and Melnychenko, 2002)
The second questionnaire was distributed in May 2002 Twenty responses were received. The 2002 survey listed 52 features of the Department and included some questions seeking additional information on the features rated high on Importance and low on Performance in 2001. The third and forth questionnaires were distributed in May 2003 and April 2005 (22 and 13 responses respectively). They both also listed 52 features. We wanted to compare the data from 2001 to 2005 to see how opinions had changed during this period. In 2005 a scale from 1 to 9 was used. Both visual and algebraic analyses were made of the data obtained. We coded the four quadrants as follows: southeast quadrant as 0, northeast quadrant as 1, northwest quadrant as 2, and southwest quadrant as 3. The features of greatest interest are those that fall in the ¡°southeast¡± (0) quadrant, that is, those features rated high on Importance and low on Performance (see Figure 1). We also calculated the Importance/Performance Ratio (IPR). Controversy and Consensus in Evaluation Standard deviation was calculated as a way of identifying controversial and consensual items for each of the measures I, P, and IPR. Six features were the most controversial in terms of Importance in the years 2002-2005. They were always within the ten most controversial items: Faculty websites; Consulting opportunities in the DC area; Faculty annual reports; Opportunities to meet local businessmen and government managers; Secretarial support; and Help with writing research proposals. There were no items on which there was consensus on Importance (bottom ten in standard deviation) in the years 2002-2005.
No features of the Department were consistently controversial (top ten standard deviation) in terms of Performance in the years 2002-2005. There was consensus on the Performance of only one feature, Library book collection, in the years in 2002-2005. Similarly, controversial and consensus features relative to IPR were identified.
Analysis of Importance, Performance, and IPR
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Table 1. The most stable high Importance features (always in the first 15) from 2001 to 2005 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
¡¡
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
¡¡
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
¡¡ |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Figure 1. Quality Improvement Priority Matrix for 2005
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
We compared IPR for the years 2001-2005 and identified features which have always been in the southeast quadrant. These features have a stable high priority for the Department (see Table 5).
Table 5. The features always in the SE quadrant from 2001 to 2005 ¡¡ |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Approaches to Identifying Priorities One of the goals of this study was to develop methods to more precisely identify priorities. In particular, we tried to develop an approach for automatically clustering features with different priorities.
In earlier studies only one approach was used for this purpose: a visual analysis of the Quality Improvement Priority Matrix as is shown in Figure 1. Features in the southeast quadrant were considered to have a high priority. However, as our study demonstrates, a visual analysis of a QIPM matrix and identifying features in the SE quadrant do not discriminate priorities sufficiently, primarily because up to half of all features routinely fall into this quadrant. For example, 19 of 51 features lie in quadrant 0 in 2001, 17 of the 52 features do in 2002, 23 of 52 in 2003, and 26 of 52 in 2005.
We identified one more problem with this approach. This is a ¡®border effect¡¯. For example, a feature with a very high Importance (e.g., close to 9) and Performance slightly higher than 5 (e.g., 5.01) will fall in the NE (¡®successful¡¯) quadrant. That seems wrong because of the high priority of this feature (its IPR is close to 2). Vice versa, a feature with Importance slightly more than 5 (e.g., 5.1) and Performance slightly lower than 5 (e.g., 4.95) will fall into the SE (¡®urgent¡¯) quadrant although this factor¡¯s IPR is close to 1 (i.e., it is not urgent).
We tried another approach to identifying priorities by using average Importance and average Performance as a midpoint for the graph rather than a scale midpoint, i.e. 5. This approach to define quadrants provides a more even allocation of features within the quadrants, especially for years when average Importance and Performance significantly differs from 5. The allocation of the 2005 features in the ¡®improved¡¯ matrix is shown in the Appendix (the M2 column) and in Figure 2. For example, after applying this approach, 12 of 51 features lie in quadrant 0 in 2001, 12 of the 52 features do in 2002, 8 of 52 in 2003, and 17 of 52 in 2005. But the border effect remains and becomes apparent even more obviously. For example, for 2001, a factor with I=7.61 and P=3.18 falls into the 3rd (¡®low priority¡¯) quadrant. That seems wrong because the feature has an IPR=2.40.
Therefore, using only Importance indexes or only Performance indexes does not permit making a single list of priorities. (A feature with high Importance may have low priority and vise versa). Calculating IPR as a single index of a feature¡¯s priority (Prytula, et al, 2002) provided a more convenient priority classification. The higher the IPR, the higher priority a feature has.
Comparative analysis of the values of IPR for 4 years and the Quality Improvement Priority Matrixes for the corresponding years demonstrates that the priority of features can be identified using the value of IPR and the position of the feature in the quadrants. Visual analysis of Quality Improvement Priority Matrixes reveals a similar distribution in the data with the same IPR interval for all 4 surveys (e.g., see Figure 3 and Figure 4). This observation served as the basis for developing a new approach to identifying priorities by choosing clusters (ovals) representing different priorities. Comparing the positions of the features on the diagrams and their IPRs led to the idea that the features could be clustered by the IPR interval. ¡¡
Figure 2. ¡®Improved¡¯ Quality Improvement Priority Matrix for 2005 (Numbers show rank by IPR)
For each year the IPR values >=2, 1.5 – 1.99, 1.25 – 1.49, and < 1.25 were used to identify four clusters. These clusters identified items that were labeled as urgent, high priority, medium priority, and low priority (Figures 3 and 4). Cluster Priority IPR interval 0 urgent >=2 1 high priority (1.5 – 1.99) 2 medium priority (1.25 – 1.49) 3 low priority <1.25
¡¡ Figure 3. Priority Clusters in 2005 Figure 4. Priority Clusters in 2003
The next important question in our study was what criterion should be used to select IPR intervals to specify the clusters. To explore this question, we calculated the correlation coefficient (r) within clusters. For unclustered data, there is a low Performance – Importance correlation. For example r = 0.32 in 2001, 0.51 in 2002, 0.52 in 2003, and 0.18 in 2005. Intercorrelation within clusters (ovals) is much higher, for example for 2001 the correlations for the clusters are .96, .88, .85, and .90. Thus, one way to automatically cluster features with different priorities is to choose intervals that create clusters with the highest intercorrelation coefficient.
The second approach, we suggest, is based on calculating the coefficient of determination (r2 ) in the following regression equation: P=a0+a1I+b1C1+b2C2+b3C3 where P is Performance, I is Importance, and C1, C2 , and C3 are dummy variables corresponding to clusters. These dummy variables have values 1 or 0 depending on whether a point is or is not in the corresponding cluster (0, 1, 2, or 3). Coefficients b1, b2, and b3 represent the increased Performance for each cluster compared with the cluster 0. We suggest that the higher r2 is in this regression equation, the more precise the clustering. For example, for 2005 we have the following regression equation for the clusters indicated above: P= -1.92 + .61I + 1.59C1 + 2.77C2 + 3.72C3 with r2 = 0.90 The parallel lines on Figure 5 indicate the levels of medium Performance in the corresponding clusters. The difference between coefficients b1, b2, and b3 is close to 1. This means that the average cluster Performance changes by 1 from cluster to cluster.
Figure 5. Lines of medium performance in the priority clusters, 2005
Practically, it is more convenient to use an ¡°averaged¡± regression equation of the following kind: P=a0+a1I+a2C where C is a dummy variable corresponding to the number of the cluster. It may have values 0, 1, 2, or 3 if a point falls into the corresponding cluster. The coefficient a2 represents the average shift in performance between clusters. For clusters indicated above, we have the following equation for 2005: P=-1.56 + .62I + 1.11C with r2 =0.89 The coefficient a2 demonstrates that average shift of Performance is 1.11 for the selected clusters. The shift in average Performance may also be an additional criterion for selecting IPR interval. Selecting the shift in average Performance, it is possible to cluster features to provide the desired average shift in Performance between clusters (coefficient a2 in the regression equation). We composed a macro for Microsoft Excel for realizing this approach to identifying features with different priorities.
The number of points / features in the clusters may also be a useful additional criterion in practice. Table 6 demonstrates the influence of different IPR intervals (clusters) on the number of features in clusters, r2 , and the coefficients of the regression equation As the table demonstrates, it is possible to cluster features, depending on the specifics of the situation, according to several criteria such as the number of features in clusters, r2 , and average shift in performance. For example, choosing IPR interval thresholds as 1.9, 1.6, and 1.3 for clustering 2001 features forms a cluster with 10 urgent priority features, a cluster with 13 high priority features, a cluster with 13 medium priority features, and a cluster with 15 low priority features. The average performance shift between clusters is 1.07 and r2=0.88. Such criteria may be used for developing software for automatically clustering features with different priorities (IPRs).
Table 6. IPR intervals, coefficients of regression equation, r2, and number of features in clusters
In this way, we formulated an integrated approach to automatically clustering features with different priorities. Presently, we are creating a special software product realizing this approach. With this software, it is possible to cluster features, according to several criteria, such as number of clusters (for different levels of accuracy), number of features in clusters, IPR intervals, intercorrelation coefficient in clusters, the coefficient of determination, and average shift in performance between clusters.
Analysis of Dynamics The correlation matrix (Table 7) demonstrates that the greater the distance in time, the lower the correlation coefficients between indexes (Importance, Performance, and IPR).
|
