Access the full text.
Sign up today, get DeepDyve free for 14 days.
AG Mainous III, WJ Hueston (1995)Medicaid free distribution programs and the availability of childhood immunizations in rural practices.
RK Zimmerman, AR Medsger, EM Ricci, M Raymund, TA Mieczkowski, S Grufferman (1997)Impact of free vaccine and insurance status on physician referral of children to public vaccine clinics.
WJ Hueston, AG Mainous III, C Palmer (1994)Delays in childhood immunizations in public and private settings.
S Sudman, NM Bradburn (1973)Effects of time and memory factors on response in surveys.
JK Iglehart (1996)The National Committee for Quality Assurance.
MS Overpeck, HJ Hoffman, K Prager (1992)The lowest birth-weight infants and the US infant mortality rate: NCHS 1983 linked birth/infant death data.
CA Cronin, CM KunkleCoalition activities addressing public accountability for quality of health care delivery.
J Morrissey (1994)HEDIS needs uniform provider data.
The Report Card Pilot Project Technical Report.
RK Spoeri, R Ullman (1997)Measuring and reporting managed care performance: lessons learned and new initiatives.
J Corrigan, L RogersComparative performance measurement for health plans.
MA Miller, RW Sutter, PM Strebel, SC Hadler (1996)Cost-effectiveness of incorporating inactivated poliovirus vaccine into the routine childhood immunization schedule.
S Sudman, NM Bradburn, N SchwarzThinking About Answers: The Application of Cognitive Processes to Survey Methodology.
JK Cooper (1995)Accountability for clinical preventive services.
WJ Hueston, RL Meade, AG Mainous III (1992)Childhood immunization practices of primary care physicians.
WJ Hueston, AG Mainous III, IB Farrell (1994)Childhood immunization availability in primary care practices: effects of programs providing free immunizations to physicians.
(1991)The measles epidemic: the problems, barriers, and recommendations.
RL Braham, HS Ruchlin (1987)Physician practice profiles: a case study of the use of audit and feedback in an ambulatory care group practice.
EA Kerr, BS Mittman, RD Hays, AL Siu, B Leake, RH Brook (1995)Managed care and capitation in California: how do physicians at financial risk control their own utilization?
JE Ware Jr, CD Sherbourne (1992)The MOS 36-Item Short Form Health Survey (SF-36).
KR Smith, NE McElweeRecalling pediatric poison events: situational and temporal determinants of accuracy.
Patients, employers, and third-party payers are all calling for improved measures of health care quality. This has led to the development of "report cards," assessments that are many times applied not just to health plans but also to providers. One attempt at creating a standardized set of quality and effectiveness measures is the Health Plan Employer Data and Information Set (HEDIS). The HEDIS measures are based primarily on analyses of administrative data sets. Problems with HEDIS measures, including the probability that plans will use different data collection methods and a lack of risk adjustment, may result in incorrect conclusions about the quality of care delivered by various providers. An alternative method of standardized surveys is proposed that will overcome many of the limitations of the current HEDIS measures, provide outcome rather than process data, and provide data for developing interventions to improve quality.As managed care becomes more important as a health care financing mechanism and cost limits are set for health care, quality of care is becoming a major concern to payers, patients, and physicians providing care for covered populations. Unfortunately, this task of searching for and assessing quality performance is one of the most demanding tasks for everyone involved in the changing health care system.Leading the efforts to evaluate health plan performance has been the development of health plan "report cards."The purpose of report cards is to collect and present appropriate information to consumers and purchasers of health care. One of the most widely used standardized approaches for assessing the quality or effectiveness of a health plan's performance is the Health Plan Employer Data and Information Set (HEDIS). These measures make the health plans accountable for quality in services, such as clinical preventive services.In addition to evaluating health plans generally, measures of quality are also applied directly to physician performance. Practice profiling is an important and increasingly common way of evaluating physician behavior.A variety of measures in HEDIS (eg, screening for diabetic retinopathy, use of mammography, and Papanicolaou smears) can easily be applied to physician performance. However, the limitations of quality measures in assessing health plan performance are still present and may be accentuated if applied to physician performance.The purpose of this discussion is to examine several HEDIS quality and effectiveness measures in relation to their ease of use and informational value and propose that, for some concepts, an alternative method may be a better choice. We will first review the development of quality performance measures and discuss their use to providers, consumers, and health plans. Next, we will identify several examples in which HEDIS specifications using administrative data are particularly problematic. Finally, we will offer an alternative to HEDIS measures for these areas that build on current survey methodology, thereby supplying a standardized alternative to standard HEDIS measures with several distinct advantages.BACKGROUND ON HEDIS QUALITY AND EFFECTIVENESS MEASURESOriginally titled the HMO Employer Data and Information Set, HEDIS developed from an effort of the HMO Group to establish measures to evaluate the value of health plans.First published in 1991, HEDIS 1.0 covered data on quality, access, satisfaction, utilization, and financial information. In late 1993 the original HEDIS measures were revised with the assistance of the National Committee for Quality Assurance and published as HEDIS 2.0. These measures continued to focus on the core areas of quality, access, satisfaction, membership, utilization, and finance. The National Committee for Quality Assurance reports that in 1996 more than 330 health plans were reporting HEDIS measures and more than 50% of large corporations use HEDIS measures when making health plan purchasing decisions. Plans must implement the computation of these measures at their own cost and responsibility. The current version of HEDIS, version 3.0, was released in 1997. Version 3.0 integrates measures for Medicaid (previously a separate set of measures) with those for commercial plans, thus offering a single set of measures for both public and private plans. Version 3.0 continues to refine measures and adds an additional focus on what HEDIS terms "health outcomes." These outcome measures tend to be process measures of the delivery of appropriate preventive care and care for selected disease states (eg, rates of immunization of infants and adolescents, flu shots for at-risk adults, cervical cancer screening, treatment of otitis media, asthma treatment for children and adults, smoking cessation efforts, and the assessment of functional status for elderly patients).PROBLEMS WITH HEDISMost private health plans and public systems like Medicaid rely heavily on claims or administrative data to evaluate their systems. The core of most HEDIS measures is derived from administrative information. Therefore, the use of current administrative data specifications are one of the benefits of the HEDIS measures, because they are based on currently collected and readily available information. However, while administrative data provide a rich source of information on a variety of measures focusing on process, they have proved inadequate for addressing health outcomes. However, although HEDIS includes these new outcome-oriented measures, they offer little insight into the plan's ability to treat serious illnesses.Moreover, these measures will not allow for an examination of why performance is the way it is. In other words, these measures provide no insight to either explain why quality would be low and neither do they provide information for interventions to try and improve quality.In addition, the move to more outcome-based measures has brought new problems for the use of HEDIS specifications. Several of the stated operational definitions of quality and effectiveness measures are difficult to implement or require multiple types of data. Thus, it is unfeasible to compute the measures using standard administrative data. Consequently, health plans' desire to compute some HEDIS measures may lead them to produce measures that are similar in intent but do not follow the HEDIS definitions. If health plans use different specifications for either of these measures, comparisons will be made of measures computed according to different operational definitions, a situation that HEDIS was directly designed to avoid. Evidence indicates that health plans vary widely in how they originate, store, and collect data.In fact, the National Committee on Quality Assurance, in their audit of HEDIS 2.0, found that plans frequently departed from the specifications for computing the measures.As HEDIS moves beyond administrative performance measures to more quality performance measures, it is less dependent on administrative data and more likely to use supplemental data sources. Many of the measures suggest using chart audits as a data supplement for some measures (eg, prenatal care in the first trimester). Although claims are the suggested measure in assessing cervical cancer screening, an alternative measure specified by HEDIS is to use documentation in the medical record that includes an author-identified note that contains the date the test was performed and the result or finding. Unfortunately, doing chart audits to evaluate a health plan or even a physician's performance is an extremely labor-intensive and time-consuming task and relies on sophisticated sampling techniques to provide useful data.An alternative method to claims analysis is to survey members through standard items. A measure that acknowledges that claims may not capture the full range of member services is flu shots for older adults, which uses a survey of a random sample of health plan members. This method is used to assess patient satisfaction as well as an assessment of the functional health status of elderly patients by using the MOS 36-Item Short Form Health Status Survey.This method may hold promise as a cost-effective alternative to several HEDIS quality and effectiveness measures and may provide more reliable and consistent findings.EXAMPLES OF PROBLEMATIC HEDIS MEASURESChildhood ImmunizationsChildhood immunizations are one of the most cost-effective health interventions. Unfortunately, low immunization rates can lead to outbreaks of vaccine-preventable disease.Thus, the goal of this effectiveness measure is to determine if children are up-to-date with their immunizations. The HEDIS 3.0 system requires that this measure be based on data for individuals continuously eligible for 1 year, thus necessitating 2 years of administrative data. This data requirement allows enough time to determine if the children received the full course of immunizations, particularly because several (eg, hepatitis B vaccine) require multiple immunizations within the time frame to be up to date. Two key issues complicate the HEDIS recommended specification for assessing childhood immunizations.First, many children are referred to health departments for childhood immunizations,especially in rural areas.It may be possible for some health plans to merge data from their state's health department database to try and provide a more complete picture of the immunization history. However, immunization delivery rates within a health plan based only on paid claims will not provide a reasonable understanding of the provision or receipt of childhood immunizations.Second, problems arise due to the multiyear requirement for using administrative data. As the volume of claims increases from single to multiple years, the size of the data files and computing time required increases substantially. This requirement may have the effect of eliminating some health plans for computing this measure or computing measures that are not comparable.Survey data on childhood immunization may provide a more usable alternative for most health plans. While surveys of members for childhood immunization must rely on proxy reports (ie, parents or caretakers) of immunizations, which may suffer from recall error and self-selection bias,proxy respondents for child health care issues have been shown to be adequate sources of data on these issues.This memory recall error can be minimized by purposely limiting the length of time for recall of events in order by surveying only parents of children younger than 3 years old.Low Birth WeightThe delivery of low-birth-weight infants has important implications for utilization of resources, as well as for morbidity and mortality.Low-birth-weight infants are 40 times more likely to die than normal-birth-weight infants.Moreover, there are several modifiable risk factors of low birth weight (eg, maternal smoking) that could be affected by physician intervention.Consequently, HEDIS assesses the proportion of low-birth-weight deliveries out of the population of live births as an outcome-based indicator of quality. Because HEDIS 3.0 computations are based not on claims but on hospital discharge abstracts and/or birth certificate data, many health plans may not have that data readily available in computerized form. Although the International Classification of Diseases, Ninth Revision (ICD-9)codes used in the discharge abstract contain a fifth digit that can be used to specify the weight of an infant, the data in this field is not uniformly reliable. Noting this, HEDIS suggests that when focusing on this data using the discharge abstract, health plans need to verify the accuracy of the data.A possible alternative strategy to an analysis of discharge abstracts or birth certificates that a health plan may undertake would be to use claims data and diagnosis codes to develop a proxy measure for low-birth-weight infants. Because low birth weight and preterm birth are seen as covarying to a great degree, one could compute the number of infants who are diagnosed with low birth weight by using the diagnosis "Other Preterm Infants," ICD-9code 765.1. Some manuals suggest that this code usually implies a birth weight of 1000 to 2499 g and/or a gestation of 28 to 37 completed weeks.Similarly, the diagnosis of "Extreme Immaturity," ICD-9code 765.0, is suggested to imply a birth weight of less than 1000 g and/or a gestation of less than 28 completed weeks. There is no specific ICD-9code for either very low birth weight, normally defined as less than 1500 g, or low birth weight, normally defined as less than 2500 g.This particular strategy could be misleading, primarily because preterm delivery and low birth weight are not synonymous conditions. Although they tend to covary they are not collinear. In fact, only 54% of low-birth-weight infants are preterm.Thus, using a preterm delivery diagnosis to represent low birth weight incorporates a substantial misclassification bias into the measure.Survey data offers an effective alternative to the HEDIS specification. As with recall issues of childhood immunizations from proxies, reports of low birth weight have a certain degree of expected error. However, surveys like the National Health Interview Survey have been used successfully to gather this information.Editor's Note:Some of my experiences with health maintenance organizations' (HMO) attempts to measure quality have really been quite funny. On its first try, one HMO said we were not doing Papanicolaou smears at a high enough rate; on further inspection, several of those for whom we had not completed "indicated" smears were men. In another situation, an HMO informed our group that our pay rate would be reduced because we spent far too much money on prescription drugs—our numbers were unbelievable compared with other family physicians. "Unbelievable" was right. When we finally got them to give us a breakdown, our number one category of drugs was chemotherapeutic drugs; somehow all prescriptions in the tertiary care hospital oncology office were being attributed to us! Drs Mainous and Talbert remind us that even potentially objective measures and the use of computer databases can be fraught with problems. The HMOs have improved their abilities in quality measurement over the years, but further improvement is needed for the health of the plans and for those of us family physicians dependent on their ability to do so.—Marjorie A. Bowman, MD, MPAADVANTAGES TO SURVEYS AS AN ALTERNATIVE MEASURE FOR ASSESSING QUALITY AND EFFECTIVENESSThere are several advantages to a method of assessing quality and effectiveness through surveys rather than through an analysis of administrative data. First, an advantage of member surveys for delivery of care measures is the additional information that can be obtained for planning and decision making. Although purchasers can use the HEDIS report cards to compare health plans, these summary data can do little beyond pointing to problem areas. A survey can ask why these are problem areas. For example, a measure of breast cancer screening is useful to tell if women are receiving appropriate mammograms, but additional information needs to be gathered to determine why they are not getting services. Reasons could range from a high cost for the service to limited access to lack of physician recommendations. This knowledge is particularly important if a health plan or physician is to make changes to improve services.Second, analysis of claims data may be focusing on a soon to be outdated model of health plan data. As Medicaid moves to managed care, for instance, providers may no longer be required to submit claims like they did in a fee-for-service environment. If providers no longer submit claims then health plans will not be able to compute measures that are defined by claims analysis. However, health plans will have access to their members for surveys.Third, a disadvantage of the current HEDIS effectiveness measures is that the population being assessed is not adjusted for severity of illness or risk. Current report cards do not account for differences in plan populations. It has been shown that an inadequate strategy for risk adjustment can lead to the wrong conclusions about quality of care.By not accounting for differences in populations while comparing report cards of quality performance, a misleading picture may be presented.Surveys of the patient population could assess not just the current HEDIS measures but a dimension like functional health status, which could be used not only as an outcome but also to stratify the population.The patient's reported information on how healthy or sick he or she is can be invaluable as an outcome.Fourth, a final problem with effectiveness measures based solely on administrative data that could be attenuated with survey methodology is the time lag necessary for analyses based on claims. For example, Medicaid programs are using HEDIS to evaluate their performance but because providers have 1 year from the date of service to submit the claims, an analysis of a complete set of 1 year's worth of claims is at least 1 year old for the most recent claim.Survey data can be more timely in planning and evaluation, a situation that is particularly important if feedback is to be given to physicians on their performance and have an effect on helping consumers make knowledgeable choices.Finally, although great strides are being made in the creation of electronic medical records,an improved information system still may not overcome the limitations of analyses based on administrative data. As plans move to integrated measurement management information system, they will still not answer some of the questions that surveys can. However, the future applications of the electronic medical record suggests that neither surveys nor administrative data should be used exclusively, but rather used together in a coordinated effort.The demand for data on quality performance in health care must be balanced by evaluations that assure valid conclusions. All methods have advantages and disadvantages and inherent limitations. Surveys are not a panacea to problems in HEDIS and in fact may cost more as an evaluation tool than analysis of administrative data. Moreover, survey data is not necessarily more accurate than claims data for every measure. However, for some measures, like the examples highlighted above, they may be a viable alternative. Unless supplemented with information from additional data like medical records and enrollee surveys, administrative data sets are unlikely to provide a complete picture of the care received by a health plan's enrollees.Surveys will overcome many of the problems that HEDIS 3.0 has for some outcome measures and will allow health plans to better assess outcomes, control for health status, and focus on possible areas for intervention. Health plans and physicians whose performance is being assessed in this standardized fashion can be more confident in the results contained in their report card.CACroninCMKunkleCoalition activities addressing public accountability for quality of health care delivery.In: Markson LE, Nash DB, eds. Accountability and Quality in Health Care: The New Responsibility. Terrace, Ill: Joint Commission on Accreditation Healthcare Organizations; 1995:84-99.JKCooperAccountability for clinical preventive services.Mil Med.1995;160:297-299.RLBrahamHSRuchlinPhysician practice profiles: a case study of the use of audit and feedback in an ambulatory care group practice.Health Care Manage Rev.1987;12:11-16.EAKerrBSMittmanRDHaysALSiuBLeakeRHBrookManaged care and capitation in California: how do physicians at financial risk control their own utilization?Ann Intern Med.1995;123:500-504.RKSpoeriRUllmanMeasuring and reporting managed care performance: lessons learned and new initiatives.Ann Intern Med.1997;127:726-732.JCorriganLRogersComparative performance measurement for health plans.In: Kazandjian VA, ed. The Epidemiology of Quality. Gaithersburg, Md: Aspen; 1995:84-106.JKIglehartThe National Committee for Quality Assurance.N Engl J Med.1996;335:995-999.JMorrisseyHEDIS needs uniform provider data.Modern Healthcare.1994;24:78-79.National Committee for Quality AssuranceThe Report Card Pilot Project Technical Report.Washington, DC: National Committee for Quality Assurance; 1995.JEWare JrCDSherbourneThe MOS 36-Item Short Form Health Survey (SF-36).Med Care.1992;30:473-483.National Vaccine Advisory CommitteeThe measles epidemic: the problems, barriers, and recommendations.JAMA.1991;266:1547-1552.RKZimmermanARMedsgerEMRicciMRaymundTAMieczkowskiSGruffermanImpact of free vaccine and insurance status on physician referral of children to public vaccine clinics.JAMA.1997;278:996-1000.WJHuestonRLMeadeAGMainous IIIChildhood immunization practices of primary care physicians.Arch Fam Med.1992;1:225-228.WJHuestonAGMainous IIICPalmerDelays in childhood immunizations in public and private settings.Arch Pediatr Adolesc Med.1994;148:470-473.WJHuestonAGMainous IIIIBFarrellChildhood immunization availability in primary care practices: effects of programs providing free immunizations to physicians.Arch Fam Med.1994;3:605-609.AGMainous IIIWJHuestonMedicaid free distribution programs and the availability of childhood immunizations in rural practices.Fam Med.1995;27:166-169.SSudmanNMBradburnEffects of time and memory factors on response in surveys.J Am Stat Assoc.1973;68:805-815.MAMillerRWSutterPMStrebelSCHadlerCost-effectiveness of incorporating inactivated poliovirus vaccine into the routine childhood immunization schedule.JAMA.1996;276:967-971.KRSmithNEMcElweeRecalling pediatric poison events: situational and temporal determinants of accuracy.In: Fowler FJ Jr, ed. Health Survey Research Methods. Washington, DC: National Center for Health Services Research and Health Care Technology Assessment; 1989.SSudmanNMBradburnNSchwarzThinking About Answers: The Application of Cognitive Processes to Survey Methodology.San Francisco, Calif: Jossey-Bass; 1996:227-243.MSOverpeckHJHoffmanKPragerThe lowest birth-weight infants and the US infant mortality rate: NCHS 1983 linked birth/infant death data.Am J Public Health.1992;82:441-444.FMcLaughlinWAltemeierMChristensenRandomized trial of comprehensive care for low-income women: effect on infant birth weight.Pediatrics.1992;89:128-132.AGMainous IIIWJHuestonThe effect of smoking cessation during pregnancy on preterm delivery and low birthweight.J Fam Pract.1994;38:262-266.WJHuestonAGMainous IIIJBFarrellA cost-benefit analysis of smoking cessation programs during the first trimester of pregnancy for the prevention of low birthweight.J Fam Pract.1994;39:353-357.Not AvailableSt. Anthony's Color-Coded ICD-9-CM Book for Physician Payment.Reston, Va: St Anthony Publishing Inc; 1995.National Center for Health StatisticsVital Statistics of the United States 1987.Vol 1. Hyattsville, Md: US Dept of Health and Human Services; 1990.ARLocalioBHHamoryTJSharpSLWeaverTRTen HaveJRLandisComparing hospital mortality in adult patients with pneumonia: a case study of statistical methods in a managed care program.Ann Intern Med.1995;122:125-132.CAMcHorneyJEWare JrWRogersAERaczekJFRLuThe validity and relative precision of MOS short- and long-form health status scales and Dartmouth COOP charts.Med Care.1992;30:MS253-MS265.ECNelsonJHWassonDJJohnsonRDHaysDartmouth COOP functional health assessment charts: brief measures for clinical practice.In: Spilker B, ed. Quality of Life and Pharmacoeconomics in Clinical Trials. Philadelphia, Pa: Lippincott-Raven Publishers; 1996:161-168.MCHornbrookMJGoodmanAssessing relative health plan risk with the RAND-36 Health Survey.Inquiry.1995;32:56-74.AGMainous IIIJTalbertMKWalkerMDHagenBaseline Evaluation of the Kentucky Medicaid Program for the 1115 Waiver.Frankfort, Ky: Cabinet for Human Resources, Commonwealth of Kentucky; 1997.RHPalmerQuality health care.JAMA.1996;275:1851-1852.SMOrnsteinElectronic medical records in family practice: the time is now.J Fam Pract.1997;44:45-48.Accepted for publication December 24, 1997.Corresponding author: Arch G. Mainous III, PhD, Department of Family Medicine, Medical University of South Carolina, Charleston, SC 29425 (e-mail: firstname.lastname@example.org).
Archives of Family Medicine – American Medical Association
Published: Sep 1, 1998
Access the full text.
Sign up today, get DeepDyve free for 14 days.