Metrics to Predict Contractor Performance
Authors: (RAND) Mendeloff, Burns, Liu, Schonlau
Date: July 2008
Summary
The authors of this research were tasked to study metrics for predicting contractor performance and to specifically identify metrics that could be used to predict "fatalities and serious injuries" for contractors. To achieve this objective the RAND Center for Health and Safety in the Workplace (CHSW) sought data on serious cases linked to particular workplaces. Accident investigations (AIs) carried out by OSHA were initially considered and rejected for study purposes because there are not that many of them and rare events are generally difficult to predict. At the suggestion of ORC, Rand also examined lost workday injury and illness incidence rates (LWDII rates) as an outcome. Rand ultimately focused on three outcome measures frequently used by employers to assess their predictive value: compliance with OSHA standards, OSHA lost workday injury and illness incidence rates; and fatalities. Results were identified for manufacturing and construction.
Conclusion:
The study concluded that all three measures (LWDII rates, compliance with OSHA standards, fatalities) had predictive value. However, the predictive value varied by industry sector and by the severity of the case. Lost workday injury rates had value in predicting future lost workday injury cases in manufacturing and construction, but did not have the same value in predicting fatalities and catastrophic events in either sector. A history of OSHA violations was predictive of fatal or catastrophic events in construction, but not in manufacturing. A prior fatality was predictive for both sectors; but it was a positive predictor of decreased risk of fatal events. However a history of fatalities had the opposite effect and tended to indicate elevated risk.
For the present, it seems that management system assessments have some predictive value, but further study is needed to ensure that the effort to use them as a predictor is worthwhile. Of course, the insights obtained from the information collected under these management systems may have other values that can justify their use, apart from prediction.
Study Findings:
Possible Predictive Outcomes: For both outcomes (fatal accidents and LWDII rates) Rand examined different variables:
a. Fatal and/or catastrophic events: To predict fatal or catastrophic events, the prior record of non-compliance appears to have some value in construction, but not in manufacturing. The lost workday injury rate has no value for either sector. For both sectors, the occurrence of a fatal event one or two years before does have predictive value, but the impact is the opposite of what most users of this measure assume. Fatalities are lower, not higher, if such an event had occurred. It appears that serious accidents tend to spur renewed attention to safety. However, firms with a long history of deaths pose above average risks.
b. Lost workday injury and illness incidence rates: To predict lost workday injury rates, Rand found that prior injury rates in manufacturing have considerable value, although the value declines with smaller establishment size and with the length of time that has passed. Rand also found very similar effects among construction firms. In manufacturing, non-compliance may be associated with higher injury rates; they were unable to study this issue in construction. Finally, Rand found that in the construction industry, but not in manufacturing, prior deaths also appeared to reduce injury rates in the following year or two.
The Potential Predictive Power of Safety and Health Management Systems: As part of its research, Rand reviewed work on whether assessments of firms' S&H management systems could predict future safety and health performance.
ORC had data on a limited number of firms that offered some insights (although the data were not sufficient to support statistically reliable conclusions). However, the ORC data does support findings that indicate the need for further study. For the group of participating firms, approximately 10% to 12% of the variation in the "days away from work" (DAW) injury case rate can be explained by the overall management system rating scored by ORC. The single most important of the 5 elements in that system was the risk score, measuring activities that the firm has undertaken to understand and address risks.
Deliverable:
Please do not cite or distribute this report until it has undergone peer-review. We expect a peer-reviewed copy to be available in the immediate future.
Metrics to Predict Contractor Performance (Partial)
