All of the analyses using facility data are based on North Carolina (NC) and North Dakota (ND). These datasets included a number of possible measures of shortage that could be used as dependent variables:

Effects of Nursing Shortage on Facility Operations. The surveys asked respondents an open-ended question about how nursing shortages have affected the operations of their facility. Responses were then coded into five broad categories: labor cost increase, reduced services, strain on staff, patient care problems, and organizational disturbance. More detailed codes within categories were also given (e.g., labor cost increase included breakouts for increases in agency use, recruitment costs, overtime, wages, retention expenses, development of float pools, and orientation expenses). This was an interesting variable because of in-depth discussions in the first advisory panel meeting about how the true measure of a nursing shortage should be related to patient care and facility operations. Although subjective, this variable touches on those issues. Caution was warranted, however, because the question asked about nursing shortage generally, and respondents may have answered the question thinking about LPNs as well as RNs, particularly if they were from a setting that relies heavily on LPNs (e.g., long-term care). Nonetheless, this variable was used as the dependent variable in a series of preliminary ordinary least squares (OLS) regressions.

RN Vacancy Rates. Both the NC and ND datasets included RN vacancy rates. Many facilities, however, had vacancy rates of 0, which limited the variation in the variable. Interestingly, there was very little correlation between RN vacancy rates and the number of reported effects of the nursing shortage, which was cause to question the utility of the consequences variable given its subjectivity. Vacancy rates were also used as the dependent variable in OLS regressions.

RN Turnover Rates. Turnover rates were not used in any of the in-depth analyses. In the first set of advisory panel meetings, the panelists pointed out that facilities that had a genuinely limited supply of RNs to draw from should be separated from facilities in which poor management led to large numbers of departures. Turnover can certainly reflect limited supply, but also seems likely to reflect problems of organizational culture, particularly in facilities that had low vacancy rates but high turnover (meaning that they had no trouble recruiting RNs, but had trouble retaining them.)

Time to Recruit RNs. Both datasets contained information on the average number of weeks reported to fill RN vacancies. Although theoretically a good indicator of shortage, the large amount of missing data for this variable ruled it out for practical reasons.

Difficulty Recruiting RNs. This ordinal variable was used in a series of ordered probit models conducted as part of the study. The variable used a five-point Likert scale with categories: Very Difficult, Difficult, Neutral, Easy, and Very Easy. Figure 6, which summarizes the responses for North Carolina, shows that somewhat more facilities reported difficulty than ease in recruiting RNs in 2004.

Figure 7 shows that only 4.6% of hospitals in North Carolina reported that recruiting RNs was either difficult or very difficult. The percentages were higher for home health agencies (15.2%), long-term care facilities (21.1%), and public health agencies (26.4%).

Figure 6. Facilities in North Carolina Reporting Different Levels of Difficulty Recruiting Nurses, 2004

[D]

Figure 7. Percentage of Facilities in NC Reporting That Recruiting Nurses Was Either "Difficult" or "Very Difficult", 2004

[D]

A. Preliminary Ordinary Least Squares (OLS) Regressions

OLS regression equations were estimated to predict and explain the number of adverse consequences and vacancy rates in all four types of facilities in North Carolina. First the models were estimated with both facility- and county-level explanatory variables, which was the ideal model. In recognition of the fact that facility-level variables were not available in most states, an abbreviated model using only county-level data was estimated for each facility type as well. The results for the models in which adverse consequences were the dependent variables are shown in Tables 2 through 6.

The results of these models were not particularly satisfying. Relatively few variables were strongly correlated to adverse consequences, and the explanatory power of the models (as measured by the R² statistic) was generally low. Although there were some statistically significant explanatory (independent) variables in the models for both predicted consequences and vacancy rates, the models explained only a relatively small percentage of the variation in the dependent variables. The explanatory power was even smaller when the facility-level variables (which would not be available outside of NC and ND without new data collection) were removed from the models, and only community variables were used.

The conclusion based on these models is that the variables collected by North Carolina were not adequate to accurately predict and explain either adverse consequences or vacancy rates.

Table 2. Coefficients for Full and Abbreviated OLS Regression Models to Predict Number of Adverse Effects of Nursing Shortages in Hospitals in NC

Full model R² = 0.429 Abbreviated model R² = 0.177

Table 3. Coefficients for Full OLS Regression Model to Predict RN Vacancy Rates in Nursing Homes in NC

R² = 0.35

Table 4. Coefficients for Full OLS Regression Model to Predict Number of Adverse Effects of Nursing Shortages in Home Health Agencies in NC

R² = 0.44

Table 5. Coefficients for Full OLS Regression Model to Predict Number of Adverse Effects of Nursing Shortages in Public Health Agencies in NC

R² = 0.34

Table 6. Coefficients for Abbreviated OLS Regression Model to Predict Number of Adverse Effects of Nursing Shortages in Public Health Agencies in NC

R² = 0.30

B. Ordered Probit Models

The next set of models estimated for North Carolina used the dependent variable of difficulty recruiting RNs. Although this variable was not available for RNs overall, facilities in North Carolina did rate RN recruiting difficulty on a scale of one to five for several types of RNs in several types of units (e.g., staff RNs in ICUs, nurse managers in ob/gyn floors, etc.). To translate this set of ratings into a single summary variable, a median value was calculated for all the positions that each facility had provided. Although few facilities had valid values for all of the different categories of hires because they had not recruited for particular positions in the past year, the median did provide an estimate of the overall difficulty.

A series of ordered probit models were estimated to predict and explain variations in this new median self-reported difficulty in recruiting RNs. Coefficients for the different explanatory and independent variables were estimated for the four facility types both separately and together (to predict recruiting difficulty relative to facilities of their own type and relative to all facilities). The combined model is shown in Table 7 below; the facility-specific models are available in the technical report Methods for Identifying Facilities and Communities with Shortages of Nurses.

These models showed promise in explaining difficulty recruiting RNs. Nonetheless, the models were dependent upon a number of facility-level variables, and it was not clear whether a subjective assessment of the difficult recruiting was an adequate basis for rating nursing shortages in facilities.

Table 7. Coefficient Estimates for the Ordered Probit Nursing Shortage Model Based on All Facilities in NC

McKelvey-Zavoina R² = 0.71

C. Validation of North Carolina Results

To address some of the questions regarding the adequacy of the “difficulty recruiting” variable, project staff attempted to validate the reported difficulty with a series of follow-up calls to those facilities that reported the most and least difficulty recruiting RNs. A list of the ID codes for the top- and bottom-ranked facilities was given to the North Carolina Center for Nursing, which provided contact information for those facilities without linking them to the identifiers in order to preserve the confidentiality of the data provided on the original survey. Consequently, the validation process was partially “blind,” with no one involved in the validation knowing whether the facility had original reported a very high or a very low level of recruiting difficulty. The interviewer asked for a retrospective evaluation of difficulty recruiting RNs in 2004 (the data year used in the analysis), and to control for the possibility that people would provide retrospective data based on the current situation, an assessment of the current difficulty recruiting RNs was also obtained. Results were then sent back to the North Carolina Center for Nursing, where names of facilities were stripped and original survey identifiers were reattached in order to compare original with retrospective responses.

The rank order correlation between the original data reported in 2004 and retrospective data obtained through the validation process was only 0.347, an indication that difficulty recruiting RNs was a less than ideal measure of shortage, even though the correlation was greater than would be expected by random chance (p = 0.016). Not only was the difficulty recruiting in 2004 from the interviews not highly correlated with current difficulty, but it also was not highly correlated with the original assessments made in 2004. Because this process provided only marginal support for the validity of this dependent variable, it was decided that subjective indicators of shortage were likely to be too highly influenced by personal judgments and biases of the person completing the survey (e.g., overall disposition, momentary mood) to justify using them as the basis for a shortage assessment and rating process.

D. Apply North Carolina Ordered Probit Coefficients in North Dakota

Another attempt to validate the models to use the North Carolina data on recruiting difficulty involved an attempt to apply these models to another state. The coefficients from the ordered probit models were applied to comparable data from North Dakota to compare predicted to actual reported recruiting difficulty. The coefficients from the North Carolina models proved to be poor predictors of reported difficulty in North Dakota.

This raised serious questions about the possibility of using coefficients from one state to predict or estimate the extent of shortages in another state. Although further investigation might reveal that coefficients from one state might be used in another state with similar demographic characteristics, interstate variations in health care and labor market environments seem to preclude nationwide use of a model constructed in only one state.

E. OLS Regressions for Vacancy Rates Using Combined Data from NC and ND

It was hypothesized that the relatively small sample size for models based solely on data from North Carolina might have contributed to the limited number of statistically significant coefficients, and that increasing the number of cases might yield better results. This hypothesis led to a final set of models in the study incorporating facility-level data by the study team and models based on a combined data set from both North Carolina and North Dakota. OLS regression models were estimated to predict vacancy rates at facilities in those two states combined.

The hypothesis, in fact, proved to be true. Models based on the combined dataset (shown below in Tables 8 to 11) highlighted a greater number of statistically significant explanatory variables for RN vacancy rates than models for either state alone. The overall explanatory power of these models remained only moderate, however, with much unexplained variation in vacancy rates. The long-term care model, in particular, had very limited explanatory power (R² = 0.238; Table 10). Furthermore, these models continued to rely heavily on facility-specific data that would be difficult to obtain at the national level.

Table 8. OLS Coefficient Estimates for RN Vacancy Rates in Hospital Settings for Combined NC & ND Model

R² = 0.400

Table 9. Coefficient Estimates for RN Vacancy Rates in Home Health Setting for Combined NC & ND Model

R² = 0.346

Table 10. Coefficient Estimates for Long-Term Care Setting for Combined NC & ND Model

R² = 0.238

Table 11. Coefficient Estimates for Public Health Setting for Combined NC & ND Model

R² = 0.389