Demonstrating the Success of Focused Technical Assistance by the Center for Optimizing Rural Health: Measurement Matters!

By Linnae Hutchison, MBA, MT, Director of Research, Texas A&M College of Nursing; Jane N. Bolin, RN, JD, PhD, Associate Dean for Research & Regents Professor, Texas A&M College of Nursing; and Gautaumi Shankira, MPH, Graduate Assistant, Texas A&M School of Public Health

Accountability to the public as well as federal and state funders requires that programs demonstrate that their programs and technical assistance are making a difference for the populations they serve. Assessing how well programs serve participating organizations can be challenging, because defining value and success often means different things to different people. Some appreciate strategy and expert insights, others might prioritize networking opportunities, while still others want to demonstrate concrete organizational improvement. When it comes to rural hospitals, the Health Resources and Services Administration (HRSA) and other federal funders are concerned with ensuring rural hospitals stay open to provide access for the rural populations in the states they serve. Some rural hospitals may be the only emergency care for 100-200 miles. Losing a hospital due to lack of funding or an inability to navigate payment regulations often means the loss of jobs and important care.

The Texas A&M Center for Optimizing Rural Health (CORH) works with rural hospitals, their providers, and their communities to improve the quality of care, maintain access to care, and address challenges unique to the communities they serve. CORH is a relatively young program. CORH began working with cohorts of rural hospitals in 2018.

Answering the question of how CORH should define and measure success has been the primary goal of the evaluation team. We’re making great progress and learning so many important lessons for rural stakeholders as we mature as a program and proudly serve more rural hospitals along the way. Here are some core insights:

In CORH’s earliest years, our focus was necessarily on establishing and satisfying a series of process measures because, as our hospital program participants told us, “It had only been a year.” A year is not long enough to know if CORH’s services are making a difference. These process metrics included:

When you’re new like us, determining how to define and measure success is critical. Not only is it fundamental to understanding areas of organizational strength and opportunities for improvement over time, but done right, defining and measuring success should be a strategic tool to clearly communicate how CORH is adding value. An equally important imperative: Our grant makers at HRSA need CORH to be able to demonstrate that they—and rural hospitals—are getting value from their investment in the Vulnerable Focused Rural Hospitals Assistance Program (VRHAP), which CORH administers.

1. establishing bylaws and policies and procedures for the organization,
2. ensuring that the learning sessions CORH staff and experts made available to rural institutions via webinar were offered, and,
3. facilitating ECHO challenges that featured and facilitated rural institutions sharing information about their hospital, posed a challenge. Thus, we began soliciting counsel from peer institutions on how to address various challenges.

CORH met the aforementioned process measures and “checked the boxes” in all of these areas and more, including soliciting qualitative and, where possible, quantitative feedback from each hospital participant. In so doing, we began to lay a foundation for more complicated, quantitative outcome measurement down the road.

In response to HRSA’s recent call for nationally benchmarked outcome measures—such as how rural hospitals are performing when it comes to their operations, financials and quality—the evaluation team set about to build a data-driven case for CORH to see if the numbers would also show we are making a measurable difference in these hospitals financial and operational stability.

Most of the outcome data available from CORH hospitals today is self-reported. While this data may provide interesting insights into the workings of institutions, it may not be comparable with other institutions’ data, making it difficult to make apples-to-apples performance comparisons. That is why CORH is moving to professionally developed government and private sector data sets that will provide a more objective measure of CORH’s programmatic impact on rural institutions. For example, we are analyzing The American Hospital Directory, HCAPS data, county health rankings data, and other data from Medicare on hospital admissions and readmissions, which will help us understand if there was any improvement or reduction in hospitals’ operations and financial status.

The end goal is to examine the data and look at comparable data points among rural hospital institutions so we can draw conclusions about what we’re seeing over time for CORH hospitals across different measures of operations, financial performance, and quality improvement. We will want to be able to consult data gathered before hospitals joined CORH too, so we can learn how hospitals met their margins and stayed open prior to participation in the program. Most importantly, these measurement initiatives will answer the many questions we still get from hospitals when they are applying to the VRHAP about the value of CORH.

In short, measurement matters. CORH is tapping into rich data resources as it navigates this journey.

In a preliminary review of available data to date, the below observations were made.

Tier I hospital cohorts receive the highest intensity of technical support and assistance. Preliminary review of the data sources over time suggests Tier I hospitals experienced improvements in certain key metrics including accounts receivable days, days cash on hand, and stabilization or improvement in net patient revenue.  

Tier II hospital cohorts receive less intensive technical support and assistance than Tier I hospitals; however, the hospitals had access to a broad range of resources offered through CORH. Preliminary review of the data over time suggests Tier II hospitals also experienced improvements in key metrics including accounts receivable days, days cash on hand, and readmission rates. Modest improvements in County Health Outcomes and Health Factors were noted among early cohorts.

Establishing meaningful, quantitative and comparable outcome measures over time is how rural hospitals will know if HRSA’s CORH program helped them address significant financial challenges and improve their financial and operational stability. CORH must be able to chart its own successes and identify opportunities for improvement. We look forward to providing additional updates on CORH’s measurement initiatives when we are farther along in our process.