We need healthcare transformation, but are hospital rankings of value?

Are hospital rankings of value? From an article in Beckers Hospital Review: US News shares changes to hospital rankings ahead of August release: US News & World Report has made several "refinements" to the methodology for its 2023-2024 Best Hospitals rankings and ratings, which are set for release Aug. 1. 

These kinds of rankings are useful for highlighting hospitals with grossly poor performance, but beyond that, they are still subjective. True, the measures are quantitative, but the assignment of values still reflects a certain bias, and the numbers used surely do as well.

Mr. Harder and Dr. Seo, authors of the report, wrote:

 Our mission is to serve the best interests of patients and to do so, we, like other reputable journalists, are editorially independent of our employer's business operations. To be clear, we give no consideration to whether a correspondent is affiliated with a hospital or health system that advertises in or maintains other commercial agreements with U.S. News.

US News is experiencing some resistance from law and medical schools who will no longer participate. Some hospital organizations have joined the fray. Penn Medicine, for example, announced it would no longer participate in the rankings. Of note, however, hospital rankings as based in publicly available data from CMS (Centers for Medicare & Medicaid Services). 

The US News article mentioned several refinements for forthcoming editions of Best Hospitals listed and detailed here. I’ll add my comments on some of the changes U.S. News & World Report made this year, clearly demonstrating some new sensitivity to the experience of marginalized groups.

1. Health equity

First, measures that quantify racial disparities in outcomes will shift focus from readmission to giving patients time at home (i.e., home time). 

Home time is a metric that is readily calculated from claims data and accounts for mortality, readmission to an acute care facility, and admission to a skilled nursing facility or long-term care facility after discharge. Utilization may have policy implications in assessing hospital performance on delivery of healthcare to patients.

Second, the Area Deprivation Index (ADI), a marker of neighborhood socioeconomic disadvantage, has been used in lieu of Social Vulnerability Index (SVI) in risk-adjusting this outcome measure. 

The next two paragraphs describe both methods, but you can skip to the third paragraph if you wish.

The ADI is based on a measure created by the Health Resources & Services Administration and has been refined, adapted, and validated to the Census Block Group by Dr. Amy Kind’s research team at the University of Wisconsin School of Medicine and Public Health. Datasets and maps are available at both the state and national level. An article about the website was published June 28th in the New England Journal of Medicine.

The Social Vulnerability Index uses US Census data to determine the relative social vulnerability of every census tract. The SVI ranks each tract on 14 social factors and groups them into four related themes: Socioeconomic Status, Household Composition & Disability Minority Status & Language,  and Housing & Transportation. Each tract receives a separate ranking for each of the four themes, as well as an overall ranking. The SVI can help emergency response planners and public health officials identify and map the communities that will most likely need support before, during, and after a hazardous event.

ADI can be used as an alternative to SVI, but SVI is more useful for larger geographic areas because it focuses on the county level, while ADI is better for targeting smaller areas because it includes data at the level of neighborhoods but lacks data on race/ethnicity. This shortcoming is resolved by patient data from the Virtual Research Data Center of the CMS to benchmark each hospital’s patient population more precisely against the demographic composition of its community as defined by hospital service area.

2. Greater weight on objective quality measures, less weight on expert opinion.

U.S. News increased the weight on outcome measures from 37.5 percent to 45 percent — and from 35 percent to 40 percent for some measures — in each of 11 specialties. The weight on physician opinion was reduced from up to 27.5 percent to either 12 percent or 15 percent, depending on the specialty. The outlet says it anticipates "continued diminution" of expert opinion as it adds and refines other measures, especially those that are outcome-driven.

This is interesting. It stands to reason that clinicians would tend to view the outcomes of their interventions more favorably than the data may reveal. It does expose the prevailing attitude that “truth is in the data,” while at the same time, the controversy rages about the poor quality of data in general.

3. Adjustments for COVID-19 volumes.

 U.S. News excluded certain visits from outcome measures to account for the disruption and variability of the COVID-19 pandemic. A visit was excluded if it occurred in March 2020; occurred in 2020 and the patient was diagnosed with COVID-19; or occurred between April 1, 2020, and December 31, 2020, and the hospital in which the visit occurred experienced a COVID-19 rate higher than the national mean during the month in which the visit occurred. If the patient was diagnosed with COVID-19 in 2021 and onward, the visit is not excluded but is risk adjusted.

There are endless methods for “risk-adjusting” data, but it is not disclosed which one was employed. Two different methods for years 2020 and 2021 seems a little arbitrary to me.

4. Debut of leukemia, lymphoma and myeloma rating. 

A new rating for leukemia, lymphoma and myeloma will debut in the 2023-2024 Best Hospitals rankings, joining 20 existing procedures and conditions ratings. All 21 are determined by objective quality measures. 

The key word here is “objective," meaning, data. Quality rating of these procedures should be judged by the patients. The old chestnut is, “The procedure was a success, but the patient died.”

5. Inclusion of outpatient outcome measures to two specialties.

'Prevention of outpatient procedural complications' outcome measures were added in this year's orthopedics and urology rankings. Similar outpatient outcome measures may be added in other specialties in future editions of Best Hospitals.

I would guess that outpatient orthopedics and urology procedures are no less invasive and at risk for complications. However, a significant number of these outpatient procedures are conducted in facilities either apart from the hospital physically or are even independent businesses. Since much of their “objective” data emanates from hospitals, some study is needed to compare the outcomes of the various alternatives.

6. Cardiology and heart surgery renamed, to include vascular surgery.

The specialty formerly known as cardiology and heart surgery has been renamed:cardiology, heart and vascular surgery, in recognition that vascular specialists take the lead on some cases that have consistently been included in the specialty's outcome measures.

Lumping all three together is mistake. Cardiology ranges from medication and lifestyle advice, monitoring the patient over time, discretion in preventing adverse drug reactions and even counseling family members. It is a more clinical practice. It is a completely different discipline from heart and vascular surgery, generally referred to as invasive cardiology. In fact, the risks of heart surgery and vascular surgery are also entirely different.

My take

I don’t see the value in these rankings from the perspective of a patient. In most cases, one has limited choices for hospitalization, based on proximity. If your community hospital has a middling reputation for the specialty you need, and the closest one with a better reputation is inconvenient, you mostly take your chances. There clearly are some nuggets that can help you, though.

If your local hospital has a less-than-satisfactory rating for sepsis, avoid it at all costs. If it’s obstetrics you need and find the rate of Cesarian birth is very high, or just has a higher-than-normal rate of complications, you might consider your options. But in general, these rankings are not very helpful for the individual, especially in emergencies.

And finally, something as complicated as hospital care can’t be distilled into a few metrics. I’m not entirely sure why US News even bothers. Ranking colleges? As misleading as it may be, choosing a college is a bewildering process, so knowing graduation rates, faculty-student ratios, cost and financial aid and a host of bits of information may be a useful starting point, but even then, it's best to do your homework. Gravitating to more quantitative measures for ranking hospitals at the expense of more informed evaluation is a mistake.