Get 20M+ Full-Text Papers For Less Than $1.50/day. Subscribe now for You or Your Team.

Learn More →

Building a Research Consortium of Large Health Systems: The Cancer Research Network

Building a Research Consortium of Large Health Systems: The Cancer Research Network Abstract Critical questions about cancer prevention, care, and outcomes increasingly require research involving large patient populations and their care delivery organizations. The Cancer Research Network (CRN) includes 11 integrated health systems funded by the National Cancer Institute (NCI) to conduct collaborative cancer research. This article describes the challenges of constructing a productive consortium of large health systems, and explores the CRN's responses. The CRN was initially funded through an NCI cooperative agreement in 1999 and has since received a second 4-year grant. Leadership and policy development are provided through a steering committee, subcommittees, and an external advisory committee. The CRN includes integral and affiliated research projects supported by a Scientific and Data Resources Core. Three characteristics of the CRN intensified the general challenges of consortium research: 1) its members are large health systems with legitimate concerns about confidentiality of data about enrollees, providers, and the organization; 2) CRN research projects often generate highly sensitive data about quality of care; and therefore 3) each participating organization wants a strong voice in CRN direction. CRN experience to date confirms that a consortium of health systems with internal research capacity can address a range of important cancer research questions that would be difficult to study in other venues. The advantages and challenges of consortium research are explored, with suggestions for the development, execution, and management of multisystem population laboratories. Consortium or cooperative group research has been the foundation of American clinical cancer research. Through their large investigative teams and access to large, diverse patient populations, research networks like Cooperative Groups and the related Community Clinical Oncology Programs (CCOPs) are critical resources for evaluating therapeutic and preventive interventions. Large networks figure prominently in the new NIH Roadmap effort to reengineer the clinical research enterprise (1). More recently the cohorts and databases assembled from large clinical trials and other databases have been used to study critically important clinical and policy questions included under the general rubric of outcomes research (2). These include studies of a broader array of outcomes such as quality of life, cost-effectiveness, prognostic factors, and the impacts of therapy on important subgroups such as racial minorities or older adults. Although this is an important evolutionary step in clinical cancer research, the highly selective nature of clinical trial patient populations and databases limits their ability to answer many critical questions about care, long-term outcomes, costs, and other important issues. Such questions require more representative populations, longer follow-up periods, and a greater range of patient data. Many questions need data on the preventive and medical care experience as well as cancer characteristics of patients to describe and assess current cancer care, and to evaluate its impacts on patient outcomes and health care costs. Such data now exist through the SEER–Medicare data link for non-HMO data cancer patients who are 65 years and older (3). But population data of this sort are generally unavailable for HMO enrollees, those younger than 65 years, or those at risk of cancer. To this end, the National Cancer Institute (NCI) issued a Request for Applications (RFA) in 1997 entitled “Cancer Research Network Across Health Care Systems.” NCI's goal was to enhance research on cancer etiology, prevention, detection, and management through collaboration among large health care delivery organizations that emphasized ambulatory care; served large, stable populations of patients; and maintained databases with a rich array of patient information. Established large integrated health systems, especially those with internal research capabilities, seemed well positioned to respond to this RFA. The Cancer Research Network (CRN) was funded in 1999 (CRN 1), responded to a second competitive RFA in 2002, and has been funded for another 4 years (CRN 2). There are compelling reasons for building a consortium of research-oriented integrated delivery systems to study cancer questions. These organizations provide comprehensive services, from prevention to palliation, to a defined population over many years. Characteristics and care of all enrollees are well documented in medical charts and automated data systems. Further, a consortium of geographically dispersed health systems increases the size and diversity of potential study populations, as well as variation in medical care practices for study. Thus, these systems can address key research questions that cannot be addressed in many other settings. Finally, the major participation of large, integrated delivery systems brings a substantial and innovative component of U.S. health care delivery to the cancer research endeavor. The CRN has wrestled with the problems of any research consortium—communication, equitable sharing of opportunities for new projects and publications, and variation in resources, experience, and data. Also, research in health care delivery organizations, even those with internal research centers, includes other challenges related to concerns about privacy, litigation, and market competition. In this paper, we describe the CRN's origin and progress, elucidate challenges to building a productive consortium of large delivery organizations, and discuss steps taken to address these challenges. CRN STRUCTURE AND PROCESSES Our initial response to the RFA was a consortium of research programs based in 10 large health systems: Group Health Cooperative (GHC); Harvard Pilgrim Health Care (HPHC); Henry Ford Health System/Health Alliance Plan (HFHS/HAP); HealthPartners Research Foundation (HPRF); Meyers Primary Care Institute of Fallon/University of Massachusetts (Meyers); and Kaiser Permanente divisions in Hawaii (KPH), Oregon and Washington (KPNW), Northern California (KPNC), Colorado (KPCO), and Southern California (KPSC). In response to the 2002 RFA, we added an 11th site—Kaiser Permanente Georgia (KPGA)—to further augment geographic representation and population diversity. The CRN emanated from an established organization of health system–based research organizations—the HMO Research Network (4). The CRN's initial goals were to 1) create a productive multipotential cancer research consortium that would become a national resource for research on cancer prevention, care and outcomes; 2) develop a research theme and new projects consistent with the theme; 3) develop and use standardized approaches to data collection, management, and analysis; 4) increase enrollment of health systems' patients in NCI-supported clinical trials; 5) continuously improve the CRN's performance through ongoing evaluation. PARTICIPATING HEALTH SYSTEMS Patient Population and Clinical Care The CRN's 11 participating members have enrolled populations ranging from 207 000 to 3 129 000, for an aggregate population of more than 10 million enrollees that represents 3.5% of the U.S. population. As shown in Table 1, the populations vary in the prevalence of ethnic and racial minorities, but in aggregate include many Asian Americans, African Americans, and Latinos. For example, there are nearly 1 million African American enrollees across the CRN. Most systems have substantial Medicare enrollments, ensuring access to the full age spectrum. Table 1.  Characteristics of the health plans*   GHC  HPHC  HPRF  HFHS  Fallon  KPNC  KPNW  KPSC  KPCO  KPH  KPG  Year established  1947  1969  1957  1960  1977  1945  1942  1947  1969  1958  1985  Structure, %                            Staff/Group  80  30  64  65  64  100  100  100  100  100  90      Independent Phys. Assn.  20  70  0  30  36  0  0  0  0  0  10      Preferred provider  0  0  36  5  0  0  0  0  0  1  0  Clinic sites  30  14  347  70  22  59  27  103  17  17  10  Hospitals  2  21  84  10  1  15  1  11  2  1  3  Total enrollment, ×1000  480  770  670  600  207  3176  456  3,129  373  227  227  Retention of 1990 cohort, %                            1-y  88  86  85  89  86†  90  84  83  90  85  87      3-y  71  —  67  74  —  84  69  72  70  75  67      5-y  59  —  58  60  —  76  60  64  60  70  54  Age (y), %                            ≤24  30  28  37  39  30  34  34  36  34  34  37      25–44  24  33  39  35  30  31  29  29  27  31  36      45–64  31  28  22  20  23  25  24  25  26  23  22      65–74  7  7  1  6  9  7  7  6  9  7  3      ≥75  7  4  1  0  8  2  2  4  5  5  1  Female, %  53  53  52  54  51  51  52  52  51  51  52  Race, %                            White  89  75  85  60  87  71  82  56  75  25  59      African American  4  16  5.9  35  2  7  3  9  6  <1  33      Asian-American  5  5  5.1  <1  3  12  5  10  2  63  2      American Indian  1  <1  0.8  <1  <1  <1  1  <1  1  <1  1      Hispanic  1  4  4  <1  8  1  5  23  15  3  4      Other  0  0  0  2  0  8  5  <1  1  17  1    GHC  HPHC  HPRF  HFHS  Fallon  KPNC  KPNW  KPSC  KPCO  KPH  KPG  Year established  1947  1969  1957  1960  1977  1945  1942  1947  1969  1958  1985  Structure, %                            Staff/Group  80  30  64  65  64  100  100  100  100  100  90      Independent Phys. Assn.  20  70  0  30  36  0  0  0  0  0  10      Preferred provider  0  0  36  5  0  0  0  0  0  1  0  Clinic sites  30  14  347  70  22  59  27  103  17  17  10  Hospitals  2  21  84  10  1  15  1  11  2  1  3  Total enrollment, ×1000  480  770  670  600  207  3176  456  3,129  373  227  227  Retention of 1990 cohort, %                            1-y  88  86  85  89  86†  90  84  83  90  85  87      3-y  71  —  67  74  —  84  69  72  70  75  67      5-y  59  —  58  60  —  76  60  64  60  70  54  Age (y), %                            ≤24  30  28  37  39  30  34  34  36  34  34  37      25–44  24  33  39  35  30  31  29  29  27  31  36      45–64  31  28  22  20  23  25  24  25  26  23  22      65–74  7  7  1  6  9  7  7  6  9  7  3      ≥75  7  4  1  0  8  2  2  4  5  5  1  Female, %  53  53  52  54  51  51  52  52  51  51  52  Race, %                            White  89  75  85  60  87  71  82  56  75  25  59      African American  4  16  5.9  35  2  7  3  9  6  <1  33      Asian-American  5  5  5.1  <1  3  12  5  10  2  63  2      American Indian  1  <1  0.8  <1  <1  <1  1  <1  1  <1  1      Hispanic  1  4  4  <1  8  1  5  23  15  3  4      Other  0  0  0  2  0  8  5  <1  1  17  1  * — = information not available; GHC = Group Health Cooperative; HPHC = Harvard Pilgrim Health Care; HPRF = HealthPartners Research Foundation; HFHS = Henry Ford Health System; Fallon = Fallon Health Care System; KPNC = Kaiser Permanente Northern California; KPNW = Kaiser Permanente Northwest; KPSC = Kaiser Permanente Southern California; KPCO = Kaiser Permanente Colorado; KPH = Kaiser Permanente Hawaii; KPG = Kaiser Permanente Georgia. † Figures are for 1998 cohort. View Large These are relatively stable health systems. All have been in operation for at least 25 years. Table 1 shows the 1-, 3-, and 5-year retention rates across the 11 sites for all age groups. Disenrollment rates ranged between 10% and 17% for 1 year, 16% and 33% over 3 years, and 24% and 46% over 5 years. Evidence from CRN studies to date suggests that disenrollment rates diminish with increasing age and are still lower for enrollees diagnosed with cancer. In a recent CRN study, the likelihood of disenrollment after a cancer diagnosis was less than 1% per year. Each of these health systems has a group/staff model component, and several have network model components that cover various proportions of enrollees (Table 1). All provide comprehensive cancer services from primary prevention to end-of-life care such as hospice. All have medical and surgical oncologists and radiation therapists on staff or in their affiliated medical groups. HMO Research Programs All participating health systems have well-established public domain research programs. The year of research program initiation, research budget, and number of doctoral-level investigators are shown in Table 2. Some research programs are decades old, have budgets exceeding $10 million, and have 20 or more doctoral-level investigators, whereas a few are small and of recent origin. Nine programs are exclusively HMO sponsored, whereas two (Harvard Pilgrim and Meyers) represent joint programs with local academic institutions (Harvard University and the University of Massachusetts Medical Schools, respectively). A third, Henry Ford Health System, owns the HMO (Health Alliance Plan), but the researchers are not HMO employees; they are housed in the Medical Group. Across the centers, research tends to fall into four related areas: epidemiologic, clinical, health services, and behavioral research. Laboratory-based research is unusual among these institutions, except at Henry Ford and HealthPartners, although collaboration with academically based bench scientists is increasing. Table 2.  Characteristics of the research centers*   GHC  HPHC  HPRF  HFHS  Meyers  KPNC  KPNW  KPSC  KPCO  KPH  KPG  Year established  1983  1969  1989  1977  1996  1961  1964  1975  1987  1991  1988  2001 funding, mil $  9  13.1  3  45  1.2  13  14  7  3.5  2.1  1.7      Extramural, %  92  100  80  50  70  80  87  90  75  75  80  No. of full-time MD/PhD-level staff  24  43  12  48  9  20  21  9  11  2  1.9  No. of programmers  25  9  10  8  2  20  30  7  5  1  2  Total staff  165  76  78  125  28  300  200  67  17  30  10  Research facilities                            Survey research  X  X  X  X  X    X    X          Clinical center  X  X    X  X  X  X      X        Chart abstraction  X  X  X  X  X  X  X  X  X  X  X    GHC  HPHC  HPRF  HFHS  Meyers  KPNC  KPNW  KPSC  KPCO  KPH  KPG  Year established  1983  1969  1989  1977  1996  1961  1964  1975  1987  1991  1988  2001 funding, mil $  9  13.1  3  45  1.2  13  14  7  3.5  2.1  1.7      Extramural, %  92  100  80  50  70  80  87  90  75  75  80  No. of full-time MD/PhD-level staff  24  43  12  48  9  20  21  9  11  2  1.9  No. of programmers  25  9  10  8  2  20  30  7  5  1  2  Total staff  165  76  78  125  28  300  200  67  17  30  10  Research facilities                            Survey research  X  X  X  X  X    X    X          Clinical center  X  X    X  X  X  X      X        Chart abstraction  X  X  X  X  X  X  X  X  X  X  X  * See Table 1 for abbreviations. View Large Table 3 summarizes the automated data availability and longevity. Essentially all HMOs have computerized data on enrollment, usage, health care costs, inpatient and outpatient diagnoses, laboratory tests, and pharmacy (5). For most CRN sites, these data systems have been in place for many years, facilitating studies with lengthy periods of follow-up. Seven systems contribute to local SEER registries, and their research units have established linkages that give them access to SEER data on their enrollees. Two other organizations use local registries, and the remaining two use computerized diagnoses and related data to identify cancer patients. A few sites have established rapid ascertainment systems to facilitate study of incident cases. Table 3.  Year of availability for key data resources*   GHC  HPHC  HPRF  HFHS  Fallon  KPNC  KPNW  KPSC  KPCO  KPH  KPG  Automated medical record  2003  1969  1997  1988  2005  2004  1997  2004  1997  2001  2004  Administrative data                            Membership  1980  1969  1990  1980  1987  1970  1982  1971  1992  1998  1995      Outside claims  1979  1990  1990  1991  1987  1991  1986  1991  1993  1995  1995      Patient scheduling  1984  —  1990  1988  1987  1987  1985  1993  1992  1992  1995      Deaths  1972  1979  —  1990  —  —  1979  1988  1988  1992  1995      Cost  1998  1989  1990  1990  1988  —  1998  1996  1997  1997  1995  Automated clinical data                            Outpatient visits  1992  1969  1990  1988  1987  1994  1987  1993  1992  1995  1995      Hospitalizations  1979  1990  1990  1989  1987  1979  1965  1981  1991  1988  1995      Emergency room  1992  —  1990  1988  1987  1988  1985  1993  1994  —  1995      Pharmacy  1977  1988  1990  1992  1987  1993  1986  1992  1993  1992  1995      Laboratory  1988  1969  1994  1995  1990  1994  1993  1991  1994  1988  1995      Long-term care  1992  1990  1990  1995  —  —  1986  1995  1994  —  1995      Home health care  1992  1990  1990  1995  —  —  1987  1995  1994  —  1995      Radiology  1986  1969 text  1990 text  1988 partial  1996  1992  1989  2001  1992  1991  1995      Pathology  —  1969 text  1990 text  1988 partial  1996  1972 partial  1970  —  1994  1995  1995  Cancer registry  1974  1982  —  1972  —  1973  1960  1988  1987  1973  2004    GHC  HPHC  HPRF  HFHS  Fallon  KPNC  KPNW  KPSC  KPCO  KPH  KPG  Automated medical record  2003  1969  1997  1988  2005  2004  1997  2004  1997  2001  2004  Administrative data                            Membership  1980  1969  1990  1980  1987  1970  1982  1971  1992  1998  1995      Outside claims  1979  1990  1990  1991  1987  1991  1986  1991  1993  1995  1995      Patient scheduling  1984  —  1990  1988  1987  1987  1985  1993  1992  1992  1995      Deaths  1972  1979  —  1990  —  —  1979  1988  1988  1992  1995      Cost  1998  1989  1990  1990  1988  —  1998  1996  1997  1997  1995  Automated clinical data                            Outpatient visits  1992  1969  1990  1988  1987  1994  1987  1993  1992  1995  1995      Hospitalizations  1979  1990  1990  1989  1987  1979  1965  1981  1991  1988  1995      Emergency room  1992  —  1990  1988  1987  1988  1985  1993  1994  —  1995      Pharmacy  1977  1988  1990  1992  1987  1993  1986  1992  1993  1992  1995      Laboratory  1988  1969  1994  1995  1990  1994  1993  1991  1994  1988  1995      Long-term care  1992  1990  1990  1995  —  —  1986  1995  1994  —  1995      Home health care  1992  1990  1990  1995  —  —  1987  1995  1994  —  1995      Radiology  1986  1969 text  1990 text  1988 partial  1996  1992  1989  2001  1992  1991  1995      Pathology  —  1969 text  1990 text  1988 partial  1996  1972 partial  1970  —  1994  1995  1995  Cancer registry  1974  1982  —  1972  —  1973  1960  1988  1987  1973  2004  * Text = with search capability; — = information not available in a single comprehensive data file. See Table 1 for other abbreviations. View Large The CRN's Virtual Data Warehouse provides a quantitative snapshot of cancer cases available for study. Based on automated data from 2003, the number of incident cancer cases at eight CRN member organizations was as follows: 3505 colorectal, 3868 lung, 6520 breast, and 5926 prostate cancers. The three CRN sites not included have a combined enrollment of approximately 1.3 million. The CRN also provides rich opportunities to study less common cancers. For example, in 2003 the eight organizations recorded 308 esophageal, 589 stomach, 667 pancreatic, and 575 ovarian cancers. Thus, for studies of most tumors, the CRN has access to sizable numbers of patients. CRN STRUCTURE The CRN is an “Alliance Structure,” more specifically a lateral alliance, as described in the book Managing a Health Care Alliance, Improving Community Cancer Care by Kaluzny and colleagues (6). They define “lateral” alliances as “those in which similar types of organizations pool information and resources to pursue mutually beneficial goals and interests.” Evaluation of these alliances, especially CCOPs, has provided important insights into elements of a successful research alliance (7): interdependence for meeting key research needs, effective communication, mutual adaptation and collective problem solving. To foster interdependence and collective problem solving, the structure and policies of the CRN encourage the participation of all members of the consortium in decision making and all aspects of the research process. Successful alliances pay close attention to the processes of interaction and communication among alliance partners, and they opt for increasing the number of opportunities for interaction. We formed a communications committee to assess routinely whether CRN members feel that their concerns and ideas are being heard, are informed of decisions and developments, and to make recommendations to enhance communication. CRN Leadership Structure From its inception, the CRN has been governed by a steering committee that includes the CRN Principal Investigator (PI), a “site” PI at each health system, the PIs of the core research projects (if not a site PI), and the NCI project officer. Despite its large size, this group operates by consensus and makes all major decisions that way. The steering committee meets monthly by conference call and in person twice a year. The Steering Committee receives advice from an advisory group of distinguished cancer researchers, the Academic Liaison Committee (ALC), and three committees (Fig. 1). In addition to providing general scientific guidance, ALC members review new proposals and help identify research opportunities. The committees play central roles in the conduct of CRN business. The Publications Committee formulates policy concerning publications and conducts prepublication review of all manuscripts. The New Proposals Committee reviews brief descriptions of proposed new projects for scientific merit, duplication or interference with other CRN activities, and feasibility. The Communications Committee assesses the quality of communication among CRN participants and makes recommendations to the Steering Committee for improvements such as the development of a newsletter, “CRN Connection,” and revisions to the CRN's Web site. Fig. 1. View largeDownload slide CRN organizational structure. Fig. 1. View largeDownload slide CRN organizational structure. Scientific Core Resources The original CRN structure included a Data Resources Coordinating Center (DRCC), which bore the primary responsibility for meeting the data standardization goal, and seven Expert Teams. The evolving role of the DRCC has been a primary topic of discussion throughout the CRN's lifecycle. Its functions and progress toward data standardization are discussed more fully below. Seven Expert Teams were established in CRN1. By design, the teams functioned more as interest groups than as typical cores of program project grants. Therefore, they received very limited funding and were expected to determine their own course. Teams included biostatistics, health economics, survey measurement, clinical trials, survivorship, pharmacoepidemiology, and genetics. Several served as incubators for new research ideas and proposals. For example, the Pharmacoepidemiology Expert Team has been funded by the Agency for Healthcare Research and Quality as a Center for Education and Research in Therapeutics (8). Interest groups now play a crucial role in the scientific life of the CRN, generating new collaborative relationships and proposals. The topics have expanded to include obesity, end-of-life care, quality of cancer care, the development of biorepositories, and racial disparities. Interest groups meet regularly by conference call or in-person during CRN annual meetings. The CRN1 Survey Measurement and Economics Expert Teams provided advice and products useful to several projects—for example, a literature review on the assessment of race and ethnicity (9) and a review of practices around the use of incentives for surveying providers (10). In CRN2, these teams were combined with the DRCC in a new core resource, the Scientific and Data Resources Core (SDRC), whose more recent activities are described below. CRN PROGRESS Create a Productive Multi-institutional Cancer Research Consortium The CRN's initial focus was to build confidence and trust in the consortium. The involved research institutions had legitimate concerns about the new entity. Delivery system–based research programs have expended considerable time and resources to build population laboratories, and they feared loss of control over access to and use of their laboratories. The Steering Committee concluded that they as a group would make all major policy decisions and that they would operate by consensus. It was also decided that involvement in any of the scientific activities of the Network is voluntary. As trust built over time, attention shifted to the development of structures and processes that could efficiently conduct collaborative research, generate new proposals, and attract new investigators. The initial RFA made explicit NCI's intent that this network or consortium ultimately serve as a national resource for researchers at NCI and other cancer research organizations, not just researchers based in the member institutions. CRN members view this expectation with cautious optimism, as CRN research benefits from collaboration with biomedical and clinical cancer researchers who are in short supply in most health system–based research centers. But valid analyses of organizational, administrative, and clinical data require local expertise and experience, and injudicious use could expose the contributing organizations to possible litigation or competitive disadvantage if measures of care quality or other sensitive indicators could be linked to particular sites. Thus, the Steering Committee felt the need to have investigators who are familiar with the data play a major role in analysis and interpretation, and decided to provide access to CRN populations and data to outside researchers only when there are CRN collaborators on the project. The CRN has also taken steps to increase opportunities for collaboration. First, the Academic Liaison Committee, in addition to its advisory function, provides linkages to researchers and resources outside the CRN. Second, the CRN has begun to establish formal relationships with comprehensive cancer centers and other research organizations with relevant scientific interests and expertise. Third, the CRN has established a public Web site (http://crn.cancer.gov) to augment its visibility to cancer researchers. Develop a Research Theme and New Projects Consistent With the Theme The Institute of Medicine's report, Ensuring Quality Cancer Care, highlighted deficiencies in current services for cancer patients and the need for cancer health services research “to assess patterns of cancer care and factors associated with the receipt of good care” (11). The President's Cancer Panel described managed care as “both a problem and an opportunity for cancer care” (12). The problem is the still unproven fear that cost-related restrictions on care will limit access to optimal care for managed-care patients. One opportunity is that large, integrated delivery and financing systems can develop policies and programs that promote high-quality cancer care. A second opportunity is that managed-care organizations can become population research laboratories for examining the impact of policies, new technologies and interventions, provider practices and perspectives, and features of the delivery of cancer services on patient outcomes. Thus, our research theme—“to conduct research on the effectiveness of cancer prevention and treatment modalities and programs to identify those patient, treatment, and delivery system factors associated with differences in outcomes”—was selected because of its consistency with the goals of the RFA, the NIH Roadmap, and the interests and expertise of CRN investigators. The original and competitive renewal RFAs required proposals for specific research projects consistent with the research theme. Our initial integral projects focused on: tobacco control (13–15), the effectiveness of breast and cervical cancer screening (16–19), and efficacy of preventing breast cancer mortality in high-risk women through either early screening or prophylactic mastectomy (20–24). All projects employed rigorous, observational research designs and capitalized on the CRN's rich resources by involving multiple sites, their populations, and data resources. All studies were population based, i.e., able to identify either the entire population of interest (e.g., all women with invasive cervical cancer; all women dying of breast cancer), or a random sample of the larger population (e.g., current cigarette smokers). The major findings from the two breast cancer projects (18,23) have important implications for care. Even in the presence of organized, active programs to increase participation in screening, the major predictor of invasive disease is the failure to be screened (18). And contralateral prophylactic mastectomy reduces mortality significantly among women with unilateral breast cancer (23). These findings have already influenced screening and treatment policies among CRN HMOs. The Steering Committee elected to have a rigorous open competition to select the required research projects for the renewal proposal. We sought proposals consistent with the objectives of the NCI RFA, which included a much broader array of research areas than our initial research theme. We intensively reviewed 22 concept proposals ranging in scope from etiology to end-of-life care and selected three projects. One project explores the possible prognostic significance of various biologic characteristics among women with ductal carcinoma in situ of the breast. This study includes approximately 3700 women, one of the largest cohorts with this important condition yet assembled. It is the first CRN study to combine a population-based epidemiological cohort study with sophisticated analyses of pathological material for biological prognostic factors. A second was a direct outgrowth of the tobacco project in CRN1—a randomized trial evaluating the effects of electronic medical record system enhancements on adherence to tobacco control guidelines and cessation rates. The third study is a randomized trial examining the feasibility and efficacy of providing tailored Web-based nutritional information to improve the dietary behaviors of HMO enrollees. The last two studies provide CRN investigators with opportunities to explore the research and intervention potential of two important technological advances in medical practice—electronic medical records and interactive patient-oriented Web sites. The generation of fundable new projects has been a primary aim of the CRN from the beginning. To date, 21 projects have been funded in addition to the six core projects, and several others are under review. Table 4 lists all funded projects. They collectively study several different cancer sites and cover the natural history of cancer from prevention to end of life care. Of the 21 noncore projects, six are investigator-initiated grants from NCI; six are NCI administrative or minority supplements; two are contracts from the Centers for Disease Control and Prevention; one is an NCI contract; and six were supported through pilot funds awarded to the CRN in its renewal. Most projects involve multiple sites, and several involve collaborators from outside research institutions. Table 4.  CRN-funded projects by year Project Title  Funding Source  Year funded  HMOs Investigating Tobacco (HIT)  NCI–CRN1 Core Project  1999  Detecting Early Tumors Enables Cancer Therapy (DETECT)  NCI–CRN1 Core Project  1999  Program Testing Early Cancer Treatment & Screening (PROTECTS)  NCI–CRN1 Core Project  1999  Clinical & Pathologic Predictors of Ductal Carcinoma in Situ  NCI–CRN2 Core Project  2003  Using EMRs to Improve Adherence to Tobacco Control Guidelines (HIT 2)  NCI–CRN2 Core Project  2003  Making Effective Nutritional Choices (MENU)  NCI–CRN2 Core Project  2003  Design, Implementation & Analysis of a Clinician Survey (DETECT add-on)  NCI Supplement  2000  Pilot Study to Identify Organizational Barriers to HMO Participation in Clinical Trials  NCI Supplement  2000  Evaluation of End-of-Life Care for Prostate Cancer in the Managed-Care Environment  CDC Task Order  2000  Colon Cancer Survivors—Medications and Risk of Recurrence  NCI R01  2000  Enrolling Vietnamese and Chinese Women in Breast Cancer Treatment and Prevention Trials  NCI Supplement  2001  Patient-Oriented Outcomes of Prophylactic Mastectomy  NCI R01  2001  Cancer Surveillance in HMO Administrative Data  NCI R01  2001  The Impact of Endocrine Therapy on Survival in Men with Local or Regional Prostate Cancer-Feasibility Study  NCI Supplement  2001  A Pilot Study of Disenrollment among HMO Patients with Cancer  NCI Supplement  2001  Lung/Colon Cancer Outcomes: The Cancer Research Network  NCI Cooperative Agreement  2001  Breast Cancer Treatment Effectiveness in Older Women  NCI R01  2002  Evaluation of Hospice Referral and Palliative Care for Ovarian Cancer in the Managed Care Environment  CDC Task Order  2002  HRT Initiation and Cessation Following Results From WHI  NCI Supplement  2002  Medication Use and Risk of Esophageal Adenocarcinoma & Barrett's Esophagus  NCI Contract  2002  Michigan Center for Health Communications Research  NCI P50  2003  Accuracy of Automated Data on Colorectal Cancer Screening  NCI–CRN Pilot  2004  African American Disparities in Lung Cancer Outcomes  NCI–CRN Pilot  2004  Investigation of Age Specific Differences & Cancer of the Cecal Colon  NCI–CRN Pilot  2004  Do Acute and Chronic Illnesses Trump Preventive Care  NCI–CRN Pilot  2004  Minority Supplement—Evaluation of patterns of cancer screening within a retrospective cohort of cancer survivors  NCI Supplement  2004  Multicenter Study of Pancreatic Cancer Etiology  NCI R01  2004  Use of an interactive voice response system, with physician feedback, to reduce cancer symptoms: a pilot study  NCI–CRN Pilot  2005  Informing an R01 Application: Interviewing Colorectal Cancer Survivors  NCI–CRN Pilot  2005  Project Title  Funding Source  Year funded  HMOs Investigating Tobacco (HIT)  NCI–CRN1 Core Project  1999  Detecting Early Tumors Enables Cancer Therapy (DETECT)  NCI–CRN1 Core Project  1999  Program Testing Early Cancer Treatment & Screening (PROTECTS)  NCI–CRN1 Core Project  1999  Clinical & Pathologic Predictors of Ductal Carcinoma in Situ  NCI–CRN2 Core Project  2003  Using EMRs to Improve Adherence to Tobacco Control Guidelines (HIT 2)  NCI–CRN2 Core Project  2003  Making Effective Nutritional Choices (MENU)  NCI–CRN2 Core Project  2003  Design, Implementation & Analysis of a Clinician Survey (DETECT add-on)  NCI Supplement  2000  Pilot Study to Identify Organizational Barriers to HMO Participation in Clinical Trials  NCI Supplement  2000  Evaluation of End-of-Life Care for Prostate Cancer in the Managed-Care Environment  CDC Task Order  2000  Colon Cancer Survivors—Medications and Risk of Recurrence  NCI R01  2000  Enrolling Vietnamese and Chinese Women in Breast Cancer Treatment and Prevention Trials  NCI Supplement  2001  Patient-Oriented Outcomes of Prophylactic Mastectomy  NCI R01  2001  Cancer Surveillance in HMO Administrative Data  NCI R01  2001  The Impact of Endocrine Therapy on Survival in Men with Local or Regional Prostate Cancer-Feasibility Study  NCI Supplement  2001  A Pilot Study of Disenrollment among HMO Patients with Cancer  NCI Supplement  2001  Lung/Colon Cancer Outcomes: The Cancer Research Network  NCI Cooperative Agreement  2001  Breast Cancer Treatment Effectiveness in Older Women  NCI R01  2002  Evaluation of Hospice Referral and Palliative Care for Ovarian Cancer in the Managed Care Environment  CDC Task Order  2002  HRT Initiation and Cessation Following Results From WHI  NCI Supplement  2002  Medication Use and Risk of Esophageal Adenocarcinoma & Barrett's Esophagus  NCI Contract  2002  Michigan Center for Health Communications Research  NCI P50  2003  Accuracy of Automated Data on Colorectal Cancer Screening  NCI–CRN Pilot  2004  African American Disparities in Lung Cancer Outcomes  NCI–CRN Pilot  2004  Investigation of Age Specific Differences & Cancer of the Cecal Colon  NCI–CRN Pilot  2004  Do Acute and Chronic Illnesses Trump Preventive Care  NCI–CRN Pilot  2004  Minority Supplement—Evaluation of patterns of cancer screening within a retrospective cohort of cancer survivors  NCI Supplement  2004  Multicenter Study of Pancreatic Cancer Etiology  NCI R01  2004  Use of an interactive voice response system, with physician feedback, to reduce cancer symptoms: a pilot study  NCI–CRN Pilot  2005  Informing an R01 Application: Interviewing Colorectal Cancer Survivors  NCI–CRN Pilot  2005  View Large Develop and Use Standardized Approaches to Data Collection, Management and Analysis From the outset, the sensitivity of health system data, especially data related to quality or delivery of care, has been a major concern. Therefore, the Steering Committee rejected the notion of the DRCC serving as a repository of generic data on the enrollees of each HMO for use in current and future studies. HMO leaders were reluctant to send generic data outside their organizations because of concerns regarding patient and institutional confidentiality. The challenge then was to develop a strategy for efficiently assembling analytic datasets using standardized data without a centralized database. Next, the Steering Committee considered whether the DRCC should function as a typical statistical coordinating center: collecting project-specific data from each site, creating analytic files, and conducting the analyses. This notion was also rejected because the participating research centers sought to enhance their data and biostatistical resources and competency through involvement in the data management and analysis functions of CRN projects and activities. Lack of consensus and the rapid startup of the core research projects slowed progress toward data standardization in CRN1. But as new proposals developed and were implemented, demands for more efficient aggregation and use of automated data across HMOs became more urgent. The Steering Committee combined the functions of the DRCC, Survey Measurement, and Economic Expert Teams in a single Scientific and Data Resources Core (SDRC). The overall goal of the SDRC is to increase the use of common approaches to measurement of key constructs, as well as the collection, transfer, and management of data. The SDRC developed the strategy of producing a Virtual Data Warehouse (VDW)—a uniform data dictionary and related site-specific code that guides the assemblage of standardized site-specific databases in each organization. The administrative and clinical data systems across CRN sites vary widely, making it a considerable task to generate comparable data across sites. Prior to the development of the VDW, each project had to resolve these issues for itself. If a given multisite project wanted to identify enrollees with certain demographic and disease characteristics prior to the VDW, project investigators would have to agree on variable definitions and programmers at each site would have to create code to run against their organization's administrative and clinical data systems. The VDW now contains the relevant conceptual and operational definition(s) of variables and the computer code used at the 11 sites to collect those variables. A projectwide programmer at any one site can now write a single program for the other collaborating sites to use that identifies the relevant subjects in each participating site from the VDW database maintained at each site. To increase the visibility of this core and enhance communication with projects, we invite all project PIs to join the SDRC and the Steering Committee in a bimonthly conference call, the Project Leaders Forum. Agendas for these calls are scientific, including discussion of project-specific research issues, or broader conversations about expansion of the VDW or other methods issues. Increase Enrollment of HMO Patients in NCI-Supported Clinical Trials Enrollment of adults in cancer clinical trials continues to be a substantial barrier to progress (25) in cancer control. Two CRN projects have addressed NCI's priorities to increase the enrollment of adult cancer patients, especially racial and ethnic minorities, in clinical trials. One project evaluated barriers to clinical trial enrollment from the perspective of an organization's leaders and oncologists (26). The other is assessing strategies for increasing enrollment of Vietnamese and Chinese women in breast cancer prevention and treatment trials. These two projects explore barriers and facilitators to clinical trial recruitment and enrollment that can inform the design and testing of interventions to increase trial participation. Also, one CRN site is testing an automated system for proactively and routinely notifying clinicians of cancer trials. Continuously Improve the CRN's Performance Through Ongoing Evaluation In the RFA, NCI stipulated that the awardee periodically evaluate its performance. We chose to make this evaluation a central element of CRN management. The evaluation regularly assesses the following aspects of CRN structure and function: performance of research projects, effectiveness of core resources, effectiveness of infrastructure (PI's office, committees, web site), extent of collaboration and quality of communication, impact on CRN staff and organizations, scientific productivity. This formative evaluation is undertaken annually, with broad discussion of results and development of specific plans for improvement. CHALLENGES Several important challenges faced by the CRN have relevance for other multisite research structures. Representativeness of the CRN Systems and Populations CRN proposals and manuscripts have been criticized by reviewers who feel that our health systems' populations, practices, and data are unrepresentative. We feel that these concerns are often unjustified. The enrollees in our systems cover the demographic spectrum of the American population, as several organizations serve Medicaid and Medicare recipients in addition to commercial clients, and most have minority representation similar to or even larger than that of their surrounding communities. The growing diversification of these health systems means that the professional caregivers and practice systems are more typical than in the traditional HMO setting. That these systems are innovative may render them less representative, but, in our view, more interesting to study. Privacy and Confidentiality New privacy regulations have added to already serious concerns about patient and institutional confidentiality for the CRN. CRN organizations have their own institutional review boards (IRBs), each of which has developed local interpretations of new federal privacy regulations. These interpretations are not uniform, and individual projects have had to vary consent and survey procedures across sites. For each project, participating health systems must obtain IRB approval for all study components, materials and modifications—an iterative process that has occasionally compromised efficiency (27). Protecting the confidentiality of quality data on individual patients has proven to be especially challenging, and a significant threat to projects (28). For example, the project exploring the effectiveness of breast and cervical cancer screening programs and practices specifically assessed the prevalence of false-negative mammograms and Pap smears as well as screen-positive patients who did not receive follow-up. Several participating organizations and their clinical leaders were understandably concerned that research rereadings of specimens, especially from patients presenting with advanced cancer, could be subpoenaed for malpractice litigation. Because of this, some sites would not provide access to specimens until we could assure the sites of protection of the research data from legal discovery. To this end, we requested a certificate of confidentiality, an assurance document from the Department of Health and Human Services, which prevents individually identifiable research data from being subject to legal discovery. Upon receipt of the certificate, all participating sites agreed to submit their patient material for research. Concurrently with the reissuance of the CRN RFA, NCI and the Agency for Healthcare Research and Quality (AHRQ) had been discussing the need for closer collaboration in the area of quality-of-cancer-care research. As a result of these discussions, it was determined that AHRQ cosponsorship of NCI research networks that have a substantial health services research element would be appropriate. As a result, AHRQ is a cosponsor of CRN2. One benefit of cosponsorship is that AHRQ-sponsored projects receive broad protection from legal discovery beyond that conferred by the certificate of confidentiality. Financial Barriers to Collaboration The CRN has encountered financial disincentives inherent in multisite research. In addition to the more evident costs of collaboration associated with contracts, administration, and communication across sites, there are less obvious costs that exert an impact on project budgets. A major issue in the CRN has been differences in IRB-related consent requirements. These multiple recruitment and consent protocols add substantial costs to data collection. For example, a telephone survey center may have to program multiple consent approaches into their telephone interviewing system and train interviewers to use different procedures. Differences in data collection logistics may also necessitate development of multiple versions of protocols to address the unique features of some sites. For example, essentially all CRN sites have multiple clinical locations each with their own medical record facility. In some organizations, researchers can access medical records in a central location, reducing travel requirements. In other HMOs, research staff must travel long distances to review records. Grant reviewers often do not understand the nuances in conducting research in different settings and insist on the application of standardized data collection costs, often pegged at the costs of lower cost sites. As a result, data collection may be substantially underfunded. Project Leadership CRN projects involve multiple research teams and their associated delivery systems. Many CRN investigators had limited experience with multisite research, and the associated administrative and scientific complexities of participating in, much less managing, multisite projects. This produced opportunities for misunderstandings, miscommunications, failures to appreciate important differences among sites, and failures to acknowledge and respond to the research aspirations of participating investigators. While not unique to health system research centers, the risk of these problems may be higher in the CRN because leadership of the individual projects is spread across the 11 sites by design. This structure places major responsibility for project performance and morale on the individual project PIs. Increasing Complexity of Integrated Health Systems Most CRN member organizations began as prepaid group practices—the original HMOs. But market pressures have induced many of these organizations to diversify, most often by developing networks of physicians paid by fee-for-service reimbursement, or by increasing the fee-for-service activities of the group practice. Prepaid group practice facilitated population-based research because of its defined population of enrollees, committed clinicians, and relatively complete data on, e.g., usage, drugs, and laboratory results. Although we can still define network enrollees, their providers have less organizational commitment to the health system and are harder to engage in data collection or intervention studies. Also, network data are far less complete, as they rely on claims. Further, patients outside of the prepaid group practice are less likely to use laboratories or pharmacies that make relevant data available for research. The growing diversity of financing and delivery arrangements within health systems increases not only the representativeness but also the complexity of research in these settings. This trend underscores the importance of locally knowledgeable researchers able to navigate the political and informational complexities of each organization. LESSONS LEARNED The experience of the CRN confirms the rich research opportunities provided by a coordinated research program involving large health systems. The CRN has already generated many new projects reflecting questions posed by investigators from the 11 sites, affiliated academic institutions, and NCI. The rich array of automated data available in the CRN has been, as expected, integral to most projects and proposals. However, finding acceptable and efficient ways to assemble and analyze these data in this era of heightened privacy concerns and regulations remains a constant challenge. The Participating Institutions The CRN's progress confirms the potential of large health plans with internal research programs to address critical cancer research questions. Population size and diversity across CRN sites facilitate examination of questions related to important disease or population subgroups—e.g., women with DCIS, Vietnamese Americans, or recent smoking quitters. The presence of local researchers familiar with the system and the data has proven essential. These health systems are innovative, especially in the use of information technology. Nine of the 11 health systems will have sophisticated electronic medical record systems in full operation shortly, and all but one will use the same software. Most have or are developing interactive patient Web sites. These system changes create extraordinary opportunities to collect clinical and patient data efficiently, as well as intervene with clinicians and patients. CRN investigators will soon be able to conduct detailed studies of clinical cancer care relying exclusively on existing electronic data. Protecting Privacy and Confidentiality To retain anonymity of sites in presenting potentially sensitive organizational data, we have developed methods for protecting site identification. For example, since organization size and ethnic and racial composition vary widely and could point to the organization, we have agreed to present only site-specific data when it is crucial to the research question being addressed. If site-specific data are presented, it is done in a manner that does not reveal sample size or demographic characteristics that would identify the health system. Leadership Development The evaluation helped us identify problems in project management and opportunities to resolve them. Ongoing discussion of the project leader role and the establishment of clear expectations for project communications has improved morale and efficiency. There is a steep learning curve for project and site principal investigators. The Project Leaders Forum, an important communication vehicle in CRN 2, provides support and mentoring for less experienced project PIs. CONCLUSIONS Consortia of large health systems providing comprehensive care to many patients at risk of or suffering from cancer can serve as rich laboratories for studying a variety of important questions about prevention and management of cancer or other health conditions. Such laboratories should play a central role in accelerating the translation of research findings into practice (1). The CRN represents one strategy for building such a laboratory by constructing a consortium of integrated care systems with established research programs. Its development was facilitated by the prior collaborative experience of CRN sites in the HMO Research Network. The CRN has made substantial progress toward its goal of producing high-quality collaborative research projects and proposals. Its democratic approach to decision making, emphasis on communications, supportive infrastructure, focused core resources, and ongoing evaluation have created a culture and processes supportive of collaboration. However, in our years of operation, we have become much more aware of the added costs, financial and interpersonal, of multisite collaboration and the pressures on developers of new projects to identify the optimum number of sites needed to achieve aims of a given project. We hope that the experience of the CRN will contribute to building a body of evidence that will facilitate generation of more and better large health care delivery laboratories and provide a model for the emerging NIH Roadmap. The CRN has benefited enormously from the leadership and support of its NCI project officer, Dr. Martin Brown. Funded by the National Cancer Institute (U19 CA 79689, Increasing Effectiveness of Cancer Control Interventions, Edward H. Wagner, MD, MPH, Principal Investigator). References (1) Zerhouni E. Medicine. The NIH Roadmap. Science  2003; 302: 63–72. Google Scholar (2) Lipscomb J, Donaldson MS, Arora NK, Brown ML, Clauser SB, et al. Cancer outcomes research. J Natl Cancer Inst Monogr  2004; 33: 178–97. Google Scholar (3) Warren JL, Klabunde CN, Schrag D, Bach PB, Riley GF. Overview of the SEER-Medicare data: content, research applications, and generalizability to the United States elderly population. Med Care  2002; 40(8 Suppl): IV-3–18. Google Scholar (4) Vogt TM, Lafata JE, Tolsma D, Greene SM. The role of research in integrated health systems: the HMO Research Network. Am J Manag Care  2004; 10: 643–8. Google Scholar (5) Selby JV. Linking automated databases for research in managed care settings. Ann Intern Med  1997; 127(8 Pt 2): 719–24. Google Scholar (6) Kaluzny AD, Warnecke RB and Associates, editors. Managing a health care alliance: improving community cancer care. San Francisco (CA): Jossey-Bass; 1996. Google Scholar (7) Klabunde CN, Kaluzny AD. Accrual to the breast cancer prevention trial by participating Community Clinical Oncology Programs: a panel data analysis. Breast Cancer Res Treat  1995; 35: 43–50. Google Scholar (8) Platt R, Davis R, Finkelstein J, Go AS, Gurwitz JH, Roblin D, et al. Multicenter epidemiologic and health services research on therapeutics in the HMO Research Network Center for Education and Research on Therapeutics. Pharmacoepidemiol Drug Saf  2001; 10: 373–7. Google Scholar (9) Ford ME, Hill DD, Nerenz D, Hornbrook M, Zapka JG, Meenan R, et al. Categorizing race and ethnicity in the HMO Cancer Research Network. Ethn Dis  2002; 12: 135–40. Google Scholar (10) Field TS, Cadoret CA, Brown ML, Ford ME, Greene SM, Hill D, et al. Surveying physicians: do components of the “Total Design Approach” to optimizing survey response rates apply to physicians? Med Care  2002; 40: 596–606. Google Scholar (11) Hewitt M and Simone JV, editors. National Cancer Policy Board, Institute of Medicine and National Research Council. Ensuring Quality Cancer Care. Washington (DC): National Academy Press; 1999. p. 223. Google Scholar (12) President's Cancer Panel. Voices of a Broken System: Real People, Real Problems. Annual Report, 2000–2001. Available at: http://deainfo.nci.nih.gov/advisory/pcp. Google Scholar (13) Rigotti NA, Quinn VP, Stevens VJ, Solberg LI, Hollis JF, Rosenthal AC, et al. Tobacco-control policies in 11 leading managed care organizations: progress and challenges. Eff Clin Pract  2002; 5: 130–6. Google Scholar (14) Solberg LI, Hollis JA, Stevens VJ, Rigotti NA, Quinn VP, Aickin M. Does methodology affect the ability to monitor tobacco control activities? Implications for HEDIS and other performance measures. Prev Med  2003; 37: 33–40. Google Scholar (15) Solberg LI, Quinn VP, Stevens VJ, Vogt TM, Rigotti NA, Zapka JG, et al. Tobacco control efforts in managed care: what do the doctors think? Am J Manag Care  2004; 10: 193–8. Google Scholar (16) Puleo E, Zapka JG, White MJ, Mouchawar J, Somkin C, Taplin SH. Caffeine, cajoling and other strategies to maximize clinician survey response rates. Eval Health Prof  2002; 25: 169–84. Google Scholar (17) Zapka JG, Taplin SH, Solberg LI, Manos MM. A framework for improving the quality of cancer care: the case of breast and cervical cancer screening. Cancer Epidemiol Biomarkers Prev  2003; 12: 4–13. Google Scholar (18) Taplin SH, Ichikawa L, Yood MU, Manos MM, Geiger AM, Weinmann S, et al. Reason for late-stage breast cancer: absence of screening or detection, or breakdown in follow-up? J Natl Cancer Inst  2004; 96: 1518–27. Google Scholar (19) Zapka JG, Puleo E, Taplin SH, Goins KV, Yood MU, Mouchawar J, et al. Processes of care in cervical and breast cancer screening and follow-up-the importance of communication. Prev Med  2004; 39: 81–90. Google Scholar (20) Reisch LM, Fosse JS, Beverly K, Yu O, Barlow WE, Harris EL, et al. Training, quality assurance, and assessment of medical record abstraction in a multisite study. Am J Epidemiol  2003; 157: 546–51. Google Scholar (21) Geiger AM, Greene SM, Pardee R, Hart G, Herrinton LJ, Macedo AM, et al. A computerized system to facilitate medical record abstraction in cancer research. Cancer Causes Control  2003; 14: 469–76. Google Scholar (22) Herrinton LJ, Barlow WE, Yu O, Geiger AM, Elmore JG, et al. Efficacy of prophylactic mastectomy in women with unilateral breast cancer. J Clin Oncol. Epub 2005 Mar 28. Google Scholar (23) Geiger AM, Yu O, Herrinton LJ, Barlow WE, Harris EL, et al. A Population-Based Study of Bilateral Prophylactic Mastectomy Efficacy in Women at Elevated Risk for Breast Cancer in Community Practices. Arch Int Med  2005; 23: 4275–86. Google Scholar (24) Elmore JG, Reisch LM, Barton MB, Barlow WE, Rolnick S, et al. Efficacy of breast cancer screening in the community according to risk level. J Natl Cancer Inst  2005; 97: 1035–43. Google Scholar (25) Sateren WB, Trimble EL, Abrams J, Brawley O, Breen N, Ford L, et al. How sociodemographics, presence of oncology specialists, and hospital cancer programs affect accrual to cancer treatment trials. J Clin Oncol  2002; 20: 2109–17. Google Scholar (26) Somkin CP, Altschuler A, Ackerson A, Geiger AM, Greene SM, et al. Organizational barriers to physician participation in cancer clinical trials. Am J Manag Care  2005; 11: 413–21. Google Scholar (27) Greene SM, Geiger AM, Harris El, Altschuler A, Nekhlyudov L, et al. Impact of IRB Requirements on a Multicenter Survey of Prophylactic Mastectomy Outcomes. Ann Epidemiol 2005 Jul 6 [Epub]. Google Scholar (28) Carney PA, Geller BM, Moffett H, Ganger M, Sewell M, Barlow WE, et al. Current medicolegal and confidentiality issues in large, multicenter research programs. Am J Epidemiol  2000; 152: 371–8. Google Scholar © The Author 2005. Published by Oxford University Press. All rights reserved. For Permissions, please e-mail: journals.permissions@oxfordjournals.org. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png JNCI Monographs Oxford University Press

Loading next page...
 
/lp/oxford-university-press/building-a-research-consortium-of-large-health-systems-the-cancer-iV2o0ElPEz

References (28)

Publisher
Oxford University Press
Copyright
© The Author 2005. Published by Oxford University Press. All rights reserved. For Permissions, please e-mail: journals.permissions@oxfordjournals.org.
ISSN
1052-6773
eISSN
1745-6614
DOI
10.1093/jncimonographs/lgi032
pmid
16287880
Publisher site
See Article on Publisher Site

Abstract

Abstract Critical questions about cancer prevention, care, and outcomes increasingly require research involving large patient populations and their care delivery organizations. The Cancer Research Network (CRN) includes 11 integrated health systems funded by the National Cancer Institute (NCI) to conduct collaborative cancer research. This article describes the challenges of constructing a productive consortium of large health systems, and explores the CRN's responses. The CRN was initially funded through an NCI cooperative agreement in 1999 and has since received a second 4-year grant. Leadership and policy development are provided through a steering committee, subcommittees, and an external advisory committee. The CRN includes integral and affiliated research projects supported by a Scientific and Data Resources Core. Three characteristics of the CRN intensified the general challenges of consortium research: 1) its members are large health systems with legitimate concerns about confidentiality of data about enrollees, providers, and the organization; 2) CRN research projects often generate highly sensitive data about quality of care; and therefore 3) each participating organization wants a strong voice in CRN direction. CRN experience to date confirms that a consortium of health systems with internal research capacity can address a range of important cancer research questions that would be difficult to study in other venues. The advantages and challenges of consortium research are explored, with suggestions for the development, execution, and management of multisystem population laboratories. Consortium or cooperative group research has been the foundation of American clinical cancer research. Through their large investigative teams and access to large, diverse patient populations, research networks like Cooperative Groups and the related Community Clinical Oncology Programs (CCOPs) are critical resources for evaluating therapeutic and preventive interventions. Large networks figure prominently in the new NIH Roadmap effort to reengineer the clinical research enterprise (1). More recently the cohorts and databases assembled from large clinical trials and other databases have been used to study critically important clinical and policy questions included under the general rubric of outcomes research (2). These include studies of a broader array of outcomes such as quality of life, cost-effectiveness, prognostic factors, and the impacts of therapy on important subgroups such as racial minorities or older adults. Although this is an important evolutionary step in clinical cancer research, the highly selective nature of clinical trial patient populations and databases limits their ability to answer many critical questions about care, long-term outcomes, costs, and other important issues. Such questions require more representative populations, longer follow-up periods, and a greater range of patient data. Many questions need data on the preventive and medical care experience as well as cancer characteristics of patients to describe and assess current cancer care, and to evaluate its impacts on patient outcomes and health care costs. Such data now exist through the SEER–Medicare data link for non-HMO data cancer patients who are 65 years and older (3). But population data of this sort are generally unavailable for HMO enrollees, those younger than 65 years, or those at risk of cancer. To this end, the National Cancer Institute (NCI) issued a Request for Applications (RFA) in 1997 entitled “Cancer Research Network Across Health Care Systems.” NCI's goal was to enhance research on cancer etiology, prevention, detection, and management through collaboration among large health care delivery organizations that emphasized ambulatory care; served large, stable populations of patients; and maintained databases with a rich array of patient information. Established large integrated health systems, especially those with internal research capabilities, seemed well positioned to respond to this RFA. The Cancer Research Network (CRN) was funded in 1999 (CRN 1), responded to a second competitive RFA in 2002, and has been funded for another 4 years (CRN 2). There are compelling reasons for building a consortium of research-oriented integrated delivery systems to study cancer questions. These organizations provide comprehensive services, from prevention to palliation, to a defined population over many years. Characteristics and care of all enrollees are well documented in medical charts and automated data systems. Further, a consortium of geographically dispersed health systems increases the size and diversity of potential study populations, as well as variation in medical care practices for study. Thus, these systems can address key research questions that cannot be addressed in many other settings. Finally, the major participation of large, integrated delivery systems brings a substantial and innovative component of U.S. health care delivery to the cancer research endeavor. The CRN has wrestled with the problems of any research consortium—communication, equitable sharing of opportunities for new projects and publications, and variation in resources, experience, and data. Also, research in health care delivery organizations, even those with internal research centers, includes other challenges related to concerns about privacy, litigation, and market competition. In this paper, we describe the CRN's origin and progress, elucidate challenges to building a productive consortium of large delivery organizations, and discuss steps taken to address these challenges. CRN STRUCTURE AND PROCESSES Our initial response to the RFA was a consortium of research programs based in 10 large health systems: Group Health Cooperative (GHC); Harvard Pilgrim Health Care (HPHC); Henry Ford Health System/Health Alliance Plan (HFHS/HAP); HealthPartners Research Foundation (HPRF); Meyers Primary Care Institute of Fallon/University of Massachusetts (Meyers); and Kaiser Permanente divisions in Hawaii (KPH), Oregon and Washington (KPNW), Northern California (KPNC), Colorado (KPCO), and Southern California (KPSC). In response to the 2002 RFA, we added an 11th site—Kaiser Permanente Georgia (KPGA)—to further augment geographic representation and population diversity. The CRN emanated from an established organization of health system–based research organizations—the HMO Research Network (4). The CRN's initial goals were to 1) create a productive multipotential cancer research consortium that would become a national resource for research on cancer prevention, care and outcomes; 2) develop a research theme and new projects consistent with the theme; 3) develop and use standardized approaches to data collection, management, and analysis; 4) increase enrollment of health systems' patients in NCI-supported clinical trials; 5) continuously improve the CRN's performance through ongoing evaluation. PARTICIPATING HEALTH SYSTEMS Patient Population and Clinical Care The CRN's 11 participating members have enrolled populations ranging from 207 000 to 3 129 000, for an aggregate population of more than 10 million enrollees that represents 3.5% of the U.S. population. As shown in Table 1, the populations vary in the prevalence of ethnic and racial minorities, but in aggregate include many Asian Americans, African Americans, and Latinos. For example, there are nearly 1 million African American enrollees across the CRN. Most systems have substantial Medicare enrollments, ensuring access to the full age spectrum. Table 1.  Characteristics of the health plans*   GHC  HPHC  HPRF  HFHS  Fallon  KPNC  KPNW  KPSC  KPCO  KPH  KPG  Year established  1947  1969  1957  1960  1977  1945  1942  1947  1969  1958  1985  Structure, %                            Staff/Group  80  30  64  65  64  100  100  100  100  100  90      Independent Phys. Assn.  20  70  0  30  36  0  0  0  0  0  10      Preferred provider  0  0  36  5  0  0  0  0  0  1  0  Clinic sites  30  14  347  70  22  59  27  103  17  17  10  Hospitals  2  21  84  10  1  15  1  11  2  1  3  Total enrollment, ×1000  480  770  670  600  207  3176  456  3,129  373  227  227  Retention of 1990 cohort, %                            1-y  88  86  85  89  86†  90  84  83  90  85  87      3-y  71  —  67  74  —  84  69  72  70  75  67      5-y  59  —  58  60  —  76  60  64  60  70  54  Age (y), %                            ≤24  30  28  37  39  30  34  34  36  34  34  37      25–44  24  33  39  35  30  31  29  29  27  31  36      45–64  31  28  22  20  23  25  24  25  26  23  22      65–74  7  7  1  6  9  7  7  6  9  7  3      ≥75  7  4  1  0  8  2  2  4  5  5  1  Female, %  53  53  52  54  51  51  52  52  51  51  52  Race, %                            White  89  75  85  60  87  71  82  56  75  25  59      African American  4  16  5.9  35  2  7  3  9  6  <1  33      Asian-American  5  5  5.1  <1  3  12  5  10  2  63  2      American Indian  1  <1  0.8  <1  <1  <1  1  <1  1  <1  1      Hispanic  1  4  4  <1  8  1  5  23  15  3  4      Other  0  0  0  2  0  8  5  <1  1  17  1    GHC  HPHC  HPRF  HFHS  Fallon  KPNC  KPNW  KPSC  KPCO  KPH  KPG  Year established  1947  1969  1957  1960  1977  1945  1942  1947  1969  1958  1985  Structure, %                            Staff/Group  80  30  64  65  64  100  100  100  100  100  90      Independent Phys. Assn.  20  70  0  30  36  0  0  0  0  0  10      Preferred provider  0  0  36  5  0  0  0  0  0  1  0  Clinic sites  30  14  347  70  22  59  27  103  17  17  10  Hospitals  2  21  84  10  1  15  1  11  2  1  3  Total enrollment, ×1000  480  770  670  600  207  3176  456  3,129  373  227  227  Retention of 1990 cohort, %                            1-y  88  86  85  89  86†  90  84  83  90  85  87      3-y  71  —  67  74  —  84  69  72  70  75  67      5-y  59  —  58  60  —  76  60  64  60  70  54  Age (y), %                            ≤24  30  28  37  39  30  34  34  36  34  34  37      25–44  24  33  39  35  30  31  29  29  27  31  36      45–64  31  28  22  20  23  25  24  25  26  23  22      65–74  7  7  1  6  9  7  7  6  9  7  3      ≥75  7  4  1  0  8  2  2  4  5  5  1  Female, %  53  53  52  54  51  51  52  52  51  51  52  Race, %                            White  89  75  85  60  87  71  82  56  75  25  59      African American  4  16  5.9  35  2  7  3  9  6  <1  33      Asian-American  5  5  5.1  <1  3  12  5  10  2  63  2      American Indian  1  <1  0.8  <1  <1  <1  1  <1  1  <1  1      Hispanic  1  4  4  <1  8  1  5  23  15  3  4      Other  0  0  0  2  0  8  5  <1  1  17  1  * — = information not available; GHC = Group Health Cooperative; HPHC = Harvard Pilgrim Health Care; HPRF = HealthPartners Research Foundation; HFHS = Henry Ford Health System; Fallon = Fallon Health Care System; KPNC = Kaiser Permanente Northern California; KPNW = Kaiser Permanente Northwest; KPSC = Kaiser Permanente Southern California; KPCO = Kaiser Permanente Colorado; KPH = Kaiser Permanente Hawaii; KPG = Kaiser Permanente Georgia. † Figures are for 1998 cohort. View Large These are relatively stable health systems. All have been in operation for at least 25 years. Table 1 shows the 1-, 3-, and 5-year retention rates across the 11 sites for all age groups. Disenrollment rates ranged between 10% and 17% for 1 year, 16% and 33% over 3 years, and 24% and 46% over 5 years. Evidence from CRN studies to date suggests that disenrollment rates diminish with increasing age and are still lower for enrollees diagnosed with cancer. In a recent CRN study, the likelihood of disenrollment after a cancer diagnosis was less than 1% per year. Each of these health systems has a group/staff model component, and several have network model components that cover various proportions of enrollees (Table 1). All provide comprehensive cancer services from primary prevention to end-of-life care such as hospice. All have medical and surgical oncologists and radiation therapists on staff or in their affiliated medical groups. HMO Research Programs All participating health systems have well-established public domain research programs. The year of research program initiation, research budget, and number of doctoral-level investigators are shown in Table 2. Some research programs are decades old, have budgets exceeding $10 million, and have 20 or more doctoral-level investigators, whereas a few are small and of recent origin. Nine programs are exclusively HMO sponsored, whereas two (Harvard Pilgrim and Meyers) represent joint programs with local academic institutions (Harvard University and the University of Massachusetts Medical Schools, respectively). A third, Henry Ford Health System, owns the HMO (Health Alliance Plan), but the researchers are not HMO employees; they are housed in the Medical Group. Across the centers, research tends to fall into four related areas: epidemiologic, clinical, health services, and behavioral research. Laboratory-based research is unusual among these institutions, except at Henry Ford and HealthPartners, although collaboration with academically based bench scientists is increasing. Table 2.  Characteristics of the research centers*   GHC  HPHC  HPRF  HFHS  Meyers  KPNC  KPNW  KPSC  KPCO  KPH  KPG  Year established  1983  1969  1989  1977  1996  1961  1964  1975  1987  1991  1988  2001 funding, mil $  9  13.1  3  45  1.2  13  14  7  3.5  2.1  1.7      Extramural, %  92  100  80  50  70  80  87  90  75  75  80  No. of full-time MD/PhD-level staff  24  43  12  48  9  20  21  9  11  2  1.9  No. of programmers  25  9  10  8  2  20  30  7  5  1  2  Total staff  165  76  78  125  28  300  200  67  17  30  10  Research facilities                            Survey research  X  X  X  X  X    X    X          Clinical center  X  X    X  X  X  X      X        Chart abstraction  X  X  X  X  X  X  X  X  X  X  X    GHC  HPHC  HPRF  HFHS  Meyers  KPNC  KPNW  KPSC  KPCO  KPH  KPG  Year established  1983  1969  1989  1977  1996  1961  1964  1975  1987  1991  1988  2001 funding, mil $  9  13.1  3  45  1.2  13  14  7  3.5  2.1  1.7      Extramural, %  92  100  80  50  70  80  87  90  75  75  80  No. of full-time MD/PhD-level staff  24  43  12  48  9  20  21  9  11  2  1.9  No. of programmers  25  9  10  8  2  20  30  7  5  1  2  Total staff  165  76  78  125  28  300  200  67  17  30  10  Research facilities                            Survey research  X  X  X  X  X    X    X          Clinical center  X  X    X  X  X  X      X        Chart abstraction  X  X  X  X  X  X  X  X  X  X  X  * See Table 1 for abbreviations. View Large Table 3 summarizes the automated data availability and longevity. Essentially all HMOs have computerized data on enrollment, usage, health care costs, inpatient and outpatient diagnoses, laboratory tests, and pharmacy (5). For most CRN sites, these data systems have been in place for many years, facilitating studies with lengthy periods of follow-up. Seven systems contribute to local SEER registries, and their research units have established linkages that give them access to SEER data on their enrollees. Two other organizations use local registries, and the remaining two use computerized diagnoses and related data to identify cancer patients. A few sites have established rapid ascertainment systems to facilitate study of incident cases. Table 3.  Year of availability for key data resources*   GHC  HPHC  HPRF  HFHS  Fallon  KPNC  KPNW  KPSC  KPCO  KPH  KPG  Automated medical record  2003  1969  1997  1988  2005  2004  1997  2004  1997  2001  2004  Administrative data                            Membership  1980  1969  1990  1980  1987  1970  1982  1971  1992  1998  1995      Outside claims  1979  1990  1990  1991  1987  1991  1986  1991  1993  1995  1995      Patient scheduling  1984  —  1990  1988  1987  1987  1985  1993  1992  1992  1995      Deaths  1972  1979  —  1990  —  —  1979  1988  1988  1992  1995      Cost  1998  1989  1990  1990  1988  —  1998  1996  1997  1997  1995  Automated clinical data                            Outpatient visits  1992  1969  1990  1988  1987  1994  1987  1993  1992  1995  1995      Hospitalizations  1979  1990  1990  1989  1987  1979  1965  1981  1991  1988  1995      Emergency room  1992  —  1990  1988  1987  1988  1985  1993  1994  —  1995      Pharmacy  1977  1988  1990  1992  1987  1993  1986  1992  1993  1992  1995      Laboratory  1988  1969  1994  1995  1990  1994  1993  1991  1994  1988  1995      Long-term care  1992  1990  1990  1995  —  —  1986  1995  1994  —  1995      Home health care  1992  1990  1990  1995  —  —  1987  1995  1994  —  1995      Radiology  1986  1969 text  1990 text  1988 partial  1996  1992  1989  2001  1992  1991  1995      Pathology  —  1969 text  1990 text  1988 partial  1996  1972 partial  1970  —  1994  1995  1995  Cancer registry  1974  1982  —  1972  —  1973  1960  1988  1987  1973  2004    GHC  HPHC  HPRF  HFHS  Fallon  KPNC  KPNW  KPSC  KPCO  KPH  KPG  Automated medical record  2003  1969  1997  1988  2005  2004  1997  2004  1997  2001  2004  Administrative data                            Membership  1980  1969  1990  1980  1987  1970  1982  1971  1992  1998  1995      Outside claims  1979  1990  1990  1991  1987  1991  1986  1991  1993  1995  1995      Patient scheduling  1984  —  1990  1988  1987  1987  1985  1993  1992  1992  1995      Deaths  1972  1979  —  1990  —  —  1979  1988  1988  1992  1995      Cost  1998  1989  1990  1990  1988  —  1998  1996  1997  1997  1995  Automated clinical data                            Outpatient visits  1992  1969  1990  1988  1987  1994  1987  1993  1992  1995  1995      Hospitalizations  1979  1990  1990  1989  1987  1979  1965  1981  1991  1988  1995      Emergency room  1992  —  1990  1988  1987  1988  1985  1993  1994  —  1995      Pharmacy  1977  1988  1990  1992  1987  1993  1986  1992  1993  1992  1995      Laboratory  1988  1969  1994  1995  1990  1994  1993  1991  1994  1988  1995      Long-term care  1992  1990  1990  1995  —  —  1986  1995  1994  —  1995      Home health care  1992  1990  1990  1995  —  —  1987  1995  1994  —  1995      Radiology  1986  1969 text  1990 text  1988 partial  1996  1992  1989  2001  1992  1991  1995      Pathology  —  1969 text  1990 text  1988 partial  1996  1972 partial  1970  —  1994  1995  1995  Cancer registry  1974  1982  —  1972  —  1973  1960  1988  1987  1973  2004  * Text = with search capability; — = information not available in a single comprehensive data file. See Table 1 for other abbreviations. View Large The CRN's Virtual Data Warehouse provides a quantitative snapshot of cancer cases available for study. Based on automated data from 2003, the number of incident cancer cases at eight CRN member organizations was as follows: 3505 colorectal, 3868 lung, 6520 breast, and 5926 prostate cancers. The three CRN sites not included have a combined enrollment of approximately 1.3 million. The CRN also provides rich opportunities to study less common cancers. For example, in 2003 the eight organizations recorded 308 esophageal, 589 stomach, 667 pancreatic, and 575 ovarian cancers. Thus, for studies of most tumors, the CRN has access to sizable numbers of patients. CRN STRUCTURE The CRN is an “Alliance Structure,” more specifically a lateral alliance, as described in the book Managing a Health Care Alliance, Improving Community Cancer Care by Kaluzny and colleagues (6). They define “lateral” alliances as “those in which similar types of organizations pool information and resources to pursue mutually beneficial goals and interests.” Evaluation of these alliances, especially CCOPs, has provided important insights into elements of a successful research alliance (7): interdependence for meeting key research needs, effective communication, mutual adaptation and collective problem solving. To foster interdependence and collective problem solving, the structure and policies of the CRN encourage the participation of all members of the consortium in decision making and all aspects of the research process. Successful alliances pay close attention to the processes of interaction and communication among alliance partners, and they opt for increasing the number of opportunities for interaction. We formed a communications committee to assess routinely whether CRN members feel that their concerns and ideas are being heard, are informed of decisions and developments, and to make recommendations to enhance communication. CRN Leadership Structure From its inception, the CRN has been governed by a steering committee that includes the CRN Principal Investigator (PI), a “site” PI at each health system, the PIs of the core research projects (if not a site PI), and the NCI project officer. Despite its large size, this group operates by consensus and makes all major decisions that way. The steering committee meets monthly by conference call and in person twice a year. The Steering Committee receives advice from an advisory group of distinguished cancer researchers, the Academic Liaison Committee (ALC), and three committees (Fig. 1). In addition to providing general scientific guidance, ALC members review new proposals and help identify research opportunities. The committees play central roles in the conduct of CRN business. The Publications Committee formulates policy concerning publications and conducts prepublication review of all manuscripts. The New Proposals Committee reviews brief descriptions of proposed new projects for scientific merit, duplication or interference with other CRN activities, and feasibility. The Communications Committee assesses the quality of communication among CRN participants and makes recommendations to the Steering Committee for improvements such as the development of a newsletter, “CRN Connection,” and revisions to the CRN's Web site. Fig. 1. View largeDownload slide CRN organizational structure. Fig. 1. View largeDownload slide CRN organizational structure. Scientific Core Resources The original CRN structure included a Data Resources Coordinating Center (DRCC), which bore the primary responsibility for meeting the data standardization goal, and seven Expert Teams. The evolving role of the DRCC has been a primary topic of discussion throughout the CRN's lifecycle. Its functions and progress toward data standardization are discussed more fully below. Seven Expert Teams were established in CRN1. By design, the teams functioned more as interest groups than as typical cores of program project grants. Therefore, they received very limited funding and were expected to determine their own course. Teams included biostatistics, health economics, survey measurement, clinical trials, survivorship, pharmacoepidemiology, and genetics. Several served as incubators for new research ideas and proposals. For example, the Pharmacoepidemiology Expert Team has been funded by the Agency for Healthcare Research and Quality as a Center for Education and Research in Therapeutics (8). Interest groups now play a crucial role in the scientific life of the CRN, generating new collaborative relationships and proposals. The topics have expanded to include obesity, end-of-life care, quality of cancer care, the development of biorepositories, and racial disparities. Interest groups meet regularly by conference call or in-person during CRN annual meetings. The CRN1 Survey Measurement and Economics Expert Teams provided advice and products useful to several projects—for example, a literature review on the assessment of race and ethnicity (9) and a review of practices around the use of incentives for surveying providers (10). In CRN2, these teams were combined with the DRCC in a new core resource, the Scientific and Data Resources Core (SDRC), whose more recent activities are described below. CRN PROGRESS Create a Productive Multi-institutional Cancer Research Consortium The CRN's initial focus was to build confidence and trust in the consortium. The involved research institutions had legitimate concerns about the new entity. Delivery system–based research programs have expended considerable time and resources to build population laboratories, and they feared loss of control over access to and use of their laboratories. The Steering Committee concluded that they as a group would make all major policy decisions and that they would operate by consensus. It was also decided that involvement in any of the scientific activities of the Network is voluntary. As trust built over time, attention shifted to the development of structures and processes that could efficiently conduct collaborative research, generate new proposals, and attract new investigators. The initial RFA made explicit NCI's intent that this network or consortium ultimately serve as a national resource for researchers at NCI and other cancer research organizations, not just researchers based in the member institutions. CRN members view this expectation with cautious optimism, as CRN research benefits from collaboration with biomedical and clinical cancer researchers who are in short supply in most health system–based research centers. But valid analyses of organizational, administrative, and clinical data require local expertise and experience, and injudicious use could expose the contributing organizations to possible litigation or competitive disadvantage if measures of care quality or other sensitive indicators could be linked to particular sites. Thus, the Steering Committee felt the need to have investigators who are familiar with the data play a major role in analysis and interpretation, and decided to provide access to CRN populations and data to outside researchers only when there are CRN collaborators on the project. The CRN has also taken steps to increase opportunities for collaboration. First, the Academic Liaison Committee, in addition to its advisory function, provides linkages to researchers and resources outside the CRN. Second, the CRN has begun to establish formal relationships with comprehensive cancer centers and other research organizations with relevant scientific interests and expertise. Third, the CRN has established a public Web site (http://crn.cancer.gov) to augment its visibility to cancer researchers. Develop a Research Theme and New Projects Consistent With the Theme The Institute of Medicine's report, Ensuring Quality Cancer Care, highlighted deficiencies in current services for cancer patients and the need for cancer health services research “to assess patterns of cancer care and factors associated with the receipt of good care” (11). The President's Cancer Panel described managed care as “both a problem and an opportunity for cancer care” (12). The problem is the still unproven fear that cost-related restrictions on care will limit access to optimal care for managed-care patients. One opportunity is that large, integrated delivery and financing systems can develop policies and programs that promote high-quality cancer care. A second opportunity is that managed-care organizations can become population research laboratories for examining the impact of policies, new technologies and interventions, provider practices and perspectives, and features of the delivery of cancer services on patient outcomes. Thus, our research theme—“to conduct research on the effectiveness of cancer prevention and treatment modalities and programs to identify those patient, treatment, and delivery system factors associated with differences in outcomes”—was selected because of its consistency with the goals of the RFA, the NIH Roadmap, and the interests and expertise of CRN investigators. The original and competitive renewal RFAs required proposals for specific research projects consistent with the research theme. Our initial integral projects focused on: tobacco control (13–15), the effectiveness of breast and cervical cancer screening (16–19), and efficacy of preventing breast cancer mortality in high-risk women through either early screening or prophylactic mastectomy (20–24). All projects employed rigorous, observational research designs and capitalized on the CRN's rich resources by involving multiple sites, their populations, and data resources. All studies were population based, i.e., able to identify either the entire population of interest (e.g., all women with invasive cervical cancer; all women dying of breast cancer), or a random sample of the larger population (e.g., current cigarette smokers). The major findings from the two breast cancer projects (18,23) have important implications for care. Even in the presence of organized, active programs to increase participation in screening, the major predictor of invasive disease is the failure to be screened (18). And contralateral prophylactic mastectomy reduces mortality significantly among women with unilateral breast cancer (23). These findings have already influenced screening and treatment policies among CRN HMOs. The Steering Committee elected to have a rigorous open competition to select the required research projects for the renewal proposal. We sought proposals consistent with the objectives of the NCI RFA, which included a much broader array of research areas than our initial research theme. We intensively reviewed 22 concept proposals ranging in scope from etiology to end-of-life care and selected three projects. One project explores the possible prognostic significance of various biologic characteristics among women with ductal carcinoma in situ of the breast. This study includes approximately 3700 women, one of the largest cohorts with this important condition yet assembled. It is the first CRN study to combine a population-based epidemiological cohort study with sophisticated analyses of pathological material for biological prognostic factors. A second was a direct outgrowth of the tobacco project in CRN1—a randomized trial evaluating the effects of electronic medical record system enhancements on adherence to tobacco control guidelines and cessation rates. The third study is a randomized trial examining the feasibility and efficacy of providing tailored Web-based nutritional information to improve the dietary behaviors of HMO enrollees. The last two studies provide CRN investigators with opportunities to explore the research and intervention potential of two important technological advances in medical practice—electronic medical records and interactive patient-oriented Web sites. The generation of fundable new projects has been a primary aim of the CRN from the beginning. To date, 21 projects have been funded in addition to the six core projects, and several others are under review. Table 4 lists all funded projects. They collectively study several different cancer sites and cover the natural history of cancer from prevention to end of life care. Of the 21 noncore projects, six are investigator-initiated grants from NCI; six are NCI administrative or minority supplements; two are contracts from the Centers for Disease Control and Prevention; one is an NCI contract; and six were supported through pilot funds awarded to the CRN in its renewal. Most projects involve multiple sites, and several involve collaborators from outside research institutions. Table 4.  CRN-funded projects by year Project Title  Funding Source  Year funded  HMOs Investigating Tobacco (HIT)  NCI–CRN1 Core Project  1999  Detecting Early Tumors Enables Cancer Therapy (DETECT)  NCI–CRN1 Core Project  1999  Program Testing Early Cancer Treatment & Screening (PROTECTS)  NCI–CRN1 Core Project  1999  Clinical & Pathologic Predictors of Ductal Carcinoma in Situ  NCI–CRN2 Core Project  2003  Using EMRs to Improve Adherence to Tobacco Control Guidelines (HIT 2)  NCI–CRN2 Core Project  2003  Making Effective Nutritional Choices (MENU)  NCI–CRN2 Core Project  2003  Design, Implementation & Analysis of a Clinician Survey (DETECT add-on)  NCI Supplement  2000  Pilot Study to Identify Organizational Barriers to HMO Participation in Clinical Trials  NCI Supplement  2000  Evaluation of End-of-Life Care for Prostate Cancer in the Managed-Care Environment  CDC Task Order  2000  Colon Cancer Survivors—Medications and Risk of Recurrence  NCI R01  2000  Enrolling Vietnamese and Chinese Women in Breast Cancer Treatment and Prevention Trials  NCI Supplement  2001  Patient-Oriented Outcomes of Prophylactic Mastectomy  NCI R01  2001  Cancer Surveillance in HMO Administrative Data  NCI R01  2001  The Impact of Endocrine Therapy on Survival in Men with Local or Regional Prostate Cancer-Feasibility Study  NCI Supplement  2001  A Pilot Study of Disenrollment among HMO Patients with Cancer  NCI Supplement  2001  Lung/Colon Cancer Outcomes: The Cancer Research Network  NCI Cooperative Agreement  2001  Breast Cancer Treatment Effectiveness in Older Women  NCI R01  2002  Evaluation of Hospice Referral and Palliative Care for Ovarian Cancer in the Managed Care Environment  CDC Task Order  2002  HRT Initiation and Cessation Following Results From WHI  NCI Supplement  2002  Medication Use and Risk of Esophageal Adenocarcinoma & Barrett's Esophagus  NCI Contract  2002  Michigan Center for Health Communications Research  NCI P50  2003  Accuracy of Automated Data on Colorectal Cancer Screening  NCI–CRN Pilot  2004  African American Disparities in Lung Cancer Outcomes  NCI–CRN Pilot  2004  Investigation of Age Specific Differences & Cancer of the Cecal Colon  NCI–CRN Pilot  2004  Do Acute and Chronic Illnesses Trump Preventive Care  NCI–CRN Pilot  2004  Minority Supplement—Evaluation of patterns of cancer screening within a retrospective cohort of cancer survivors  NCI Supplement  2004  Multicenter Study of Pancreatic Cancer Etiology  NCI R01  2004  Use of an interactive voice response system, with physician feedback, to reduce cancer symptoms: a pilot study  NCI–CRN Pilot  2005  Informing an R01 Application: Interviewing Colorectal Cancer Survivors  NCI–CRN Pilot  2005  Project Title  Funding Source  Year funded  HMOs Investigating Tobacco (HIT)  NCI–CRN1 Core Project  1999  Detecting Early Tumors Enables Cancer Therapy (DETECT)  NCI–CRN1 Core Project  1999  Program Testing Early Cancer Treatment & Screening (PROTECTS)  NCI–CRN1 Core Project  1999  Clinical & Pathologic Predictors of Ductal Carcinoma in Situ  NCI–CRN2 Core Project  2003  Using EMRs to Improve Adherence to Tobacco Control Guidelines (HIT 2)  NCI–CRN2 Core Project  2003  Making Effective Nutritional Choices (MENU)  NCI–CRN2 Core Project  2003  Design, Implementation & Analysis of a Clinician Survey (DETECT add-on)  NCI Supplement  2000  Pilot Study to Identify Organizational Barriers to HMO Participation in Clinical Trials  NCI Supplement  2000  Evaluation of End-of-Life Care for Prostate Cancer in the Managed-Care Environment  CDC Task Order  2000  Colon Cancer Survivors—Medications and Risk of Recurrence  NCI R01  2000  Enrolling Vietnamese and Chinese Women in Breast Cancer Treatment and Prevention Trials  NCI Supplement  2001  Patient-Oriented Outcomes of Prophylactic Mastectomy  NCI R01  2001  Cancer Surveillance in HMO Administrative Data  NCI R01  2001  The Impact of Endocrine Therapy on Survival in Men with Local or Regional Prostate Cancer-Feasibility Study  NCI Supplement  2001  A Pilot Study of Disenrollment among HMO Patients with Cancer  NCI Supplement  2001  Lung/Colon Cancer Outcomes: The Cancer Research Network  NCI Cooperative Agreement  2001  Breast Cancer Treatment Effectiveness in Older Women  NCI R01  2002  Evaluation of Hospice Referral and Palliative Care for Ovarian Cancer in the Managed Care Environment  CDC Task Order  2002  HRT Initiation and Cessation Following Results From WHI  NCI Supplement  2002  Medication Use and Risk of Esophageal Adenocarcinoma & Barrett's Esophagus  NCI Contract  2002  Michigan Center for Health Communications Research  NCI P50  2003  Accuracy of Automated Data on Colorectal Cancer Screening  NCI–CRN Pilot  2004  African American Disparities in Lung Cancer Outcomes  NCI–CRN Pilot  2004  Investigation of Age Specific Differences & Cancer of the Cecal Colon  NCI–CRN Pilot  2004  Do Acute and Chronic Illnesses Trump Preventive Care  NCI–CRN Pilot  2004  Minority Supplement—Evaluation of patterns of cancer screening within a retrospective cohort of cancer survivors  NCI Supplement  2004  Multicenter Study of Pancreatic Cancer Etiology  NCI R01  2004  Use of an interactive voice response system, with physician feedback, to reduce cancer symptoms: a pilot study  NCI–CRN Pilot  2005  Informing an R01 Application: Interviewing Colorectal Cancer Survivors  NCI–CRN Pilot  2005  View Large Develop and Use Standardized Approaches to Data Collection, Management and Analysis From the outset, the sensitivity of health system data, especially data related to quality or delivery of care, has been a major concern. Therefore, the Steering Committee rejected the notion of the DRCC serving as a repository of generic data on the enrollees of each HMO for use in current and future studies. HMO leaders were reluctant to send generic data outside their organizations because of concerns regarding patient and institutional confidentiality. The challenge then was to develop a strategy for efficiently assembling analytic datasets using standardized data without a centralized database. Next, the Steering Committee considered whether the DRCC should function as a typical statistical coordinating center: collecting project-specific data from each site, creating analytic files, and conducting the analyses. This notion was also rejected because the participating research centers sought to enhance their data and biostatistical resources and competency through involvement in the data management and analysis functions of CRN projects and activities. Lack of consensus and the rapid startup of the core research projects slowed progress toward data standardization in CRN1. But as new proposals developed and were implemented, demands for more efficient aggregation and use of automated data across HMOs became more urgent. The Steering Committee combined the functions of the DRCC, Survey Measurement, and Economic Expert Teams in a single Scientific and Data Resources Core (SDRC). The overall goal of the SDRC is to increase the use of common approaches to measurement of key constructs, as well as the collection, transfer, and management of data. The SDRC developed the strategy of producing a Virtual Data Warehouse (VDW)—a uniform data dictionary and related site-specific code that guides the assemblage of standardized site-specific databases in each organization. The administrative and clinical data systems across CRN sites vary widely, making it a considerable task to generate comparable data across sites. Prior to the development of the VDW, each project had to resolve these issues for itself. If a given multisite project wanted to identify enrollees with certain demographic and disease characteristics prior to the VDW, project investigators would have to agree on variable definitions and programmers at each site would have to create code to run against their organization's administrative and clinical data systems. The VDW now contains the relevant conceptual and operational definition(s) of variables and the computer code used at the 11 sites to collect those variables. A projectwide programmer at any one site can now write a single program for the other collaborating sites to use that identifies the relevant subjects in each participating site from the VDW database maintained at each site. To increase the visibility of this core and enhance communication with projects, we invite all project PIs to join the SDRC and the Steering Committee in a bimonthly conference call, the Project Leaders Forum. Agendas for these calls are scientific, including discussion of project-specific research issues, or broader conversations about expansion of the VDW or other methods issues. Increase Enrollment of HMO Patients in NCI-Supported Clinical Trials Enrollment of adults in cancer clinical trials continues to be a substantial barrier to progress (25) in cancer control. Two CRN projects have addressed NCI's priorities to increase the enrollment of adult cancer patients, especially racial and ethnic minorities, in clinical trials. One project evaluated barriers to clinical trial enrollment from the perspective of an organization's leaders and oncologists (26). The other is assessing strategies for increasing enrollment of Vietnamese and Chinese women in breast cancer prevention and treatment trials. These two projects explore barriers and facilitators to clinical trial recruitment and enrollment that can inform the design and testing of interventions to increase trial participation. Also, one CRN site is testing an automated system for proactively and routinely notifying clinicians of cancer trials. Continuously Improve the CRN's Performance Through Ongoing Evaluation In the RFA, NCI stipulated that the awardee periodically evaluate its performance. We chose to make this evaluation a central element of CRN management. The evaluation regularly assesses the following aspects of CRN structure and function: performance of research projects, effectiveness of core resources, effectiveness of infrastructure (PI's office, committees, web site), extent of collaboration and quality of communication, impact on CRN staff and organizations, scientific productivity. This formative evaluation is undertaken annually, with broad discussion of results and development of specific plans for improvement. CHALLENGES Several important challenges faced by the CRN have relevance for other multisite research structures. Representativeness of the CRN Systems and Populations CRN proposals and manuscripts have been criticized by reviewers who feel that our health systems' populations, practices, and data are unrepresentative. We feel that these concerns are often unjustified. The enrollees in our systems cover the demographic spectrum of the American population, as several organizations serve Medicaid and Medicare recipients in addition to commercial clients, and most have minority representation similar to or even larger than that of their surrounding communities. The growing diversification of these health systems means that the professional caregivers and practice systems are more typical than in the traditional HMO setting. That these systems are innovative may render them less representative, but, in our view, more interesting to study. Privacy and Confidentiality New privacy regulations have added to already serious concerns about patient and institutional confidentiality for the CRN. CRN organizations have their own institutional review boards (IRBs), each of which has developed local interpretations of new federal privacy regulations. These interpretations are not uniform, and individual projects have had to vary consent and survey procedures across sites. For each project, participating health systems must obtain IRB approval for all study components, materials and modifications—an iterative process that has occasionally compromised efficiency (27). Protecting the confidentiality of quality data on individual patients has proven to be especially challenging, and a significant threat to projects (28). For example, the project exploring the effectiveness of breast and cervical cancer screening programs and practices specifically assessed the prevalence of false-negative mammograms and Pap smears as well as screen-positive patients who did not receive follow-up. Several participating organizations and their clinical leaders were understandably concerned that research rereadings of specimens, especially from patients presenting with advanced cancer, could be subpoenaed for malpractice litigation. Because of this, some sites would not provide access to specimens until we could assure the sites of protection of the research data from legal discovery. To this end, we requested a certificate of confidentiality, an assurance document from the Department of Health and Human Services, which prevents individually identifiable research data from being subject to legal discovery. Upon receipt of the certificate, all participating sites agreed to submit their patient material for research. Concurrently with the reissuance of the CRN RFA, NCI and the Agency for Healthcare Research and Quality (AHRQ) had been discussing the need for closer collaboration in the area of quality-of-cancer-care research. As a result of these discussions, it was determined that AHRQ cosponsorship of NCI research networks that have a substantial health services research element would be appropriate. As a result, AHRQ is a cosponsor of CRN2. One benefit of cosponsorship is that AHRQ-sponsored projects receive broad protection from legal discovery beyond that conferred by the certificate of confidentiality. Financial Barriers to Collaboration The CRN has encountered financial disincentives inherent in multisite research. In addition to the more evident costs of collaboration associated with contracts, administration, and communication across sites, there are less obvious costs that exert an impact on project budgets. A major issue in the CRN has been differences in IRB-related consent requirements. These multiple recruitment and consent protocols add substantial costs to data collection. For example, a telephone survey center may have to program multiple consent approaches into their telephone interviewing system and train interviewers to use different procedures. Differences in data collection logistics may also necessitate development of multiple versions of protocols to address the unique features of some sites. For example, essentially all CRN sites have multiple clinical locations each with their own medical record facility. In some organizations, researchers can access medical records in a central location, reducing travel requirements. In other HMOs, research staff must travel long distances to review records. Grant reviewers often do not understand the nuances in conducting research in different settings and insist on the application of standardized data collection costs, often pegged at the costs of lower cost sites. As a result, data collection may be substantially underfunded. Project Leadership CRN projects involve multiple research teams and their associated delivery systems. Many CRN investigators had limited experience with multisite research, and the associated administrative and scientific complexities of participating in, much less managing, multisite projects. This produced opportunities for misunderstandings, miscommunications, failures to appreciate important differences among sites, and failures to acknowledge and respond to the research aspirations of participating investigators. While not unique to health system research centers, the risk of these problems may be higher in the CRN because leadership of the individual projects is spread across the 11 sites by design. This structure places major responsibility for project performance and morale on the individual project PIs. Increasing Complexity of Integrated Health Systems Most CRN member organizations began as prepaid group practices—the original HMOs. But market pressures have induced many of these organizations to diversify, most often by developing networks of physicians paid by fee-for-service reimbursement, or by increasing the fee-for-service activities of the group practice. Prepaid group practice facilitated population-based research because of its defined population of enrollees, committed clinicians, and relatively complete data on, e.g., usage, drugs, and laboratory results. Although we can still define network enrollees, their providers have less organizational commitment to the health system and are harder to engage in data collection or intervention studies. Also, network data are far less complete, as they rely on claims. Further, patients outside of the prepaid group practice are less likely to use laboratories or pharmacies that make relevant data available for research. The growing diversity of financing and delivery arrangements within health systems increases not only the representativeness but also the complexity of research in these settings. This trend underscores the importance of locally knowledgeable researchers able to navigate the political and informational complexities of each organization. LESSONS LEARNED The experience of the CRN confirms the rich research opportunities provided by a coordinated research program involving large health systems. The CRN has already generated many new projects reflecting questions posed by investigators from the 11 sites, affiliated academic institutions, and NCI. The rich array of automated data available in the CRN has been, as expected, integral to most projects and proposals. However, finding acceptable and efficient ways to assemble and analyze these data in this era of heightened privacy concerns and regulations remains a constant challenge. The Participating Institutions The CRN's progress confirms the potential of large health plans with internal research programs to address critical cancer research questions. Population size and diversity across CRN sites facilitate examination of questions related to important disease or population subgroups—e.g., women with DCIS, Vietnamese Americans, or recent smoking quitters. The presence of local researchers familiar with the system and the data has proven essential. These health systems are innovative, especially in the use of information technology. Nine of the 11 health systems will have sophisticated electronic medical record systems in full operation shortly, and all but one will use the same software. Most have or are developing interactive patient Web sites. These system changes create extraordinary opportunities to collect clinical and patient data efficiently, as well as intervene with clinicians and patients. CRN investigators will soon be able to conduct detailed studies of clinical cancer care relying exclusively on existing electronic data. Protecting Privacy and Confidentiality To retain anonymity of sites in presenting potentially sensitive organizational data, we have developed methods for protecting site identification. For example, since organization size and ethnic and racial composition vary widely and could point to the organization, we have agreed to present only site-specific data when it is crucial to the research question being addressed. If site-specific data are presented, it is done in a manner that does not reveal sample size or demographic characteristics that would identify the health system. Leadership Development The evaluation helped us identify problems in project management and opportunities to resolve them. Ongoing discussion of the project leader role and the establishment of clear expectations for project communications has improved morale and efficiency. There is a steep learning curve for project and site principal investigators. The Project Leaders Forum, an important communication vehicle in CRN 2, provides support and mentoring for less experienced project PIs. CONCLUSIONS Consortia of large health systems providing comprehensive care to many patients at risk of or suffering from cancer can serve as rich laboratories for studying a variety of important questions about prevention and management of cancer or other health conditions. Such laboratories should play a central role in accelerating the translation of research findings into practice (1). The CRN represents one strategy for building such a laboratory by constructing a consortium of integrated care systems with established research programs. Its development was facilitated by the prior collaborative experience of CRN sites in the HMO Research Network. The CRN has made substantial progress toward its goal of producing high-quality collaborative research projects and proposals. Its democratic approach to decision making, emphasis on communications, supportive infrastructure, focused core resources, and ongoing evaluation have created a culture and processes supportive of collaboration. However, in our years of operation, we have become much more aware of the added costs, financial and interpersonal, of multisite collaboration and the pressures on developers of new projects to identify the optimum number of sites needed to achieve aims of a given project. We hope that the experience of the CRN will contribute to building a body of evidence that will facilitate generation of more and better large health care delivery laboratories and provide a model for the emerging NIH Roadmap. The CRN has benefited enormously from the leadership and support of its NCI project officer, Dr. Martin Brown. Funded by the National Cancer Institute (U19 CA 79689, Increasing Effectiveness of Cancer Control Interventions, Edward H. Wagner, MD, MPH, Principal Investigator). References (1) Zerhouni E. Medicine. The NIH Roadmap. Science  2003; 302: 63–72. Google Scholar (2) Lipscomb J, Donaldson MS, Arora NK, Brown ML, Clauser SB, et al. Cancer outcomes research. J Natl Cancer Inst Monogr  2004; 33: 178–97. Google Scholar (3) Warren JL, Klabunde CN, Schrag D, Bach PB, Riley GF. Overview of the SEER-Medicare data: content, research applications, and generalizability to the United States elderly population. Med Care  2002; 40(8 Suppl): IV-3–18. Google Scholar (4) Vogt TM, Lafata JE, Tolsma D, Greene SM. The role of research in integrated health systems: the HMO Research Network. Am J Manag Care  2004; 10: 643–8. Google Scholar (5) Selby JV. Linking automated databases for research in managed care settings. Ann Intern Med  1997; 127(8 Pt 2): 719–24. Google Scholar (6) Kaluzny AD, Warnecke RB and Associates, editors. Managing a health care alliance: improving community cancer care. San Francisco (CA): Jossey-Bass; 1996. Google Scholar (7) Klabunde CN, Kaluzny AD. Accrual to the breast cancer prevention trial by participating Community Clinical Oncology Programs: a panel data analysis. Breast Cancer Res Treat  1995; 35: 43–50. Google Scholar (8) Platt R, Davis R, Finkelstein J, Go AS, Gurwitz JH, Roblin D, et al. Multicenter epidemiologic and health services research on therapeutics in the HMO Research Network Center for Education and Research on Therapeutics. Pharmacoepidemiol Drug Saf  2001; 10: 373–7. Google Scholar (9) Ford ME, Hill DD, Nerenz D, Hornbrook M, Zapka JG, Meenan R, et al. Categorizing race and ethnicity in the HMO Cancer Research Network. Ethn Dis  2002; 12: 135–40. Google Scholar (10) Field TS, Cadoret CA, Brown ML, Ford ME, Greene SM, Hill D, et al. Surveying physicians: do components of the “Total Design Approach” to optimizing survey response rates apply to physicians? Med Care  2002; 40: 596–606. Google Scholar (11) Hewitt M and Simone JV, editors. National Cancer Policy Board, Institute of Medicine and National Research Council. Ensuring Quality Cancer Care. Washington (DC): National Academy Press; 1999. p. 223. Google Scholar (12) President's Cancer Panel. Voices of a Broken System: Real People, Real Problems. Annual Report, 2000–2001. Available at: http://deainfo.nci.nih.gov/advisory/pcp. Google Scholar (13) Rigotti NA, Quinn VP, Stevens VJ, Solberg LI, Hollis JF, Rosenthal AC, et al. Tobacco-control policies in 11 leading managed care organizations: progress and challenges. Eff Clin Pract  2002; 5: 130–6. Google Scholar (14) Solberg LI, Hollis JA, Stevens VJ, Rigotti NA, Quinn VP, Aickin M. Does methodology affect the ability to monitor tobacco control activities? Implications for HEDIS and other performance measures. Prev Med  2003; 37: 33–40. Google Scholar (15) Solberg LI, Quinn VP, Stevens VJ, Vogt TM, Rigotti NA, Zapka JG, et al. Tobacco control efforts in managed care: what do the doctors think? Am J Manag Care  2004; 10: 193–8. Google Scholar (16) Puleo E, Zapka JG, White MJ, Mouchawar J, Somkin C, Taplin SH. Caffeine, cajoling and other strategies to maximize clinician survey response rates. Eval Health Prof  2002; 25: 169–84. Google Scholar (17) Zapka JG, Taplin SH, Solberg LI, Manos MM. A framework for improving the quality of cancer care: the case of breast and cervical cancer screening. Cancer Epidemiol Biomarkers Prev  2003; 12: 4–13. Google Scholar (18) Taplin SH, Ichikawa L, Yood MU, Manos MM, Geiger AM, Weinmann S, et al. Reason for late-stage breast cancer: absence of screening or detection, or breakdown in follow-up? J Natl Cancer Inst  2004; 96: 1518–27. Google Scholar (19) Zapka JG, Puleo E, Taplin SH, Goins KV, Yood MU, Mouchawar J, et al. Processes of care in cervical and breast cancer screening and follow-up-the importance of communication. Prev Med  2004; 39: 81–90. Google Scholar (20) Reisch LM, Fosse JS, Beverly K, Yu O, Barlow WE, Harris EL, et al. Training, quality assurance, and assessment of medical record abstraction in a multisite study. Am J Epidemiol  2003; 157: 546–51. Google Scholar (21) Geiger AM, Greene SM, Pardee R, Hart G, Herrinton LJ, Macedo AM, et al. A computerized system to facilitate medical record abstraction in cancer research. Cancer Causes Control  2003; 14: 469–76. Google Scholar (22) Herrinton LJ, Barlow WE, Yu O, Geiger AM, Elmore JG, et al. Efficacy of prophylactic mastectomy in women with unilateral breast cancer. J Clin Oncol. Epub 2005 Mar 28. Google Scholar (23) Geiger AM, Yu O, Herrinton LJ, Barlow WE, Harris EL, et al. A Population-Based Study of Bilateral Prophylactic Mastectomy Efficacy in Women at Elevated Risk for Breast Cancer in Community Practices. Arch Int Med  2005; 23: 4275–86. Google Scholar (24) Elmore JG, Reisch LM, Barton MB, Barlow WE, Rolnick S, et al. Efficacy of breast cancer screening in the community according to risk level. J Natl Cancer Inst  2005; 97: 1035–43. Google Scholar (25) Sateren WB, Trimble EL, Abrams J, Brawley O, Breen N, Ford L, et al. How sociodemographics, presence of oncology specialists, and hospital cancer programs affect accrual to cancer treatment trials. J Clin Oncol  2002; 20: 2109–17. Google Scholar (26) Somkin CP, Altschuler A, Ackerson A, Geiger AM, Greene SM, et al. Organizational barriers to physician participation in cancer clinical trials. Am J Manag Care  2005; 11: 413–21. Google Scholar (27) Greene SM, Geiger AM, Harris El, Altschuler A, Nekhlyudov L, et al. Impact of IRB Requirements on a Multicenter Survey of Prophylactic Mastectomy Outcomes. Ann Epidemiol 2005 Jul 6 [Epub]. Google Scholar (28) Carney PA, Geller BM, Moffett H, Ganger M, Sewell M, Barlow WE, et al. Current medicolegal and confidentiality issues in large, multicenter research programs. Am J Epidemiol  2000; 152: 371–8. Google Scholar © The Author 2005. Published by Oxford University Press. All rights reserved. For Permissions, please e-mail: journals.permissions@oxfordjournals.org.

Journal

JNCI MonographsOxford University Press

Published: Nov 1, 2005

There are no references for this article.