Get 20M+ Full-Text Papers For Less Than $1.50/day. Subscribe now for You or Your Team.

Learn More →

Webinar Training: an acceptable, feasible and effective approach for multi-site medical record abstraction: the BOWII experience

Webinar Training: an acceptable, feasible and effective approach for multi-site medical record... Background: Abstractor training is a key element in creating valid and reliable data collection procedures. The choice between in-person vs. remote or simultaneous vs. sequential abstractor training has considerable consequences for time and resource utilization. We conducted a web-based (webinar) abstractor training session to standardize training across six individual Cancer Research Network (CRN) sites for a study of breast cancer treatment effects in older women (BOWII). The goals of this manuscript are to describe the training session, its participants and participants’ evaluation of webinar technology for abstraction training. Findings: A webinar was held for all six sites with the primary purpose of simultaneously training staff and ensuring consistent abstraction across sites. The training session involved sequential review of over 600 data elements outlined in the coding manual in conjunction with the display of data entry fields in the study’s electronic data collection system. Post-training evaluation was conducted via Survey Monkey . Inter-rater reliability measures for abstractors within each site were conducted three months after the commencement of data collection. Ten of the 16 people who participated in the training completed the online survey. Almost all (90%) of the 10 trainees had previous medical record abstraction experience and nearly two-thirds reported over 10 years of experience. Half of the respondents had previously participated in a webinar, among which three had participated in a webinar for training purposes. All rated the knowledge and information delivered through the webinar as useful and reported it adequately prepared them for data collection. Moreover, all participants would recommend this platform for multi-site abstraction training. Consistent with participant-reported training effectiveness, results of data collection inter-rater agreement within sites ranged from 89 to 98%, with a weighted average of 95% agreement across sites. Conclusions: Conducting training via web-based technology was an acceptable and effective approach to standardizing medical record review across multiple sites for this group of experienced abstractors. Given the substantial time and cost savings achieved with the webinar, coupled with participants’ positive evaluation of the training session, researchers should consider this instructional method as part of training efforts to ensure high quality data collection in multi-site studies. Introduction site data and differences in quality of data abstraction Medical record review is a common data collection between sites can introduce variability. Abstractor training method for conducting epidemiologic research. Although is a key element in minimizing interobserver variability to investigators in sites from various geographical locations create reliable data collection procedures. The choice often collaborate to include diverse populations to between in-person vs. remote or simultaneous vs. sequen- enhance generalizability of study results, idiosyncrasies of tial abstractor training has considerable consequences for time and resource utilization. Advances in information * Correspondence: Chantal.C.Avila@kp.org technology have produced readily available, low cost, effi- Kaiser Permanente Southern California, Research & Evaluation, Pasadena, CA, cient alternatives to traditional training approaches. Faced USA with the challenges of collecting complex medical record Full list of author information is available at the end of the article © 2011 Avila et al; licensee BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Avila et al. BMC Research Notes 2011, 4:430 Page 2 of 7 http://www.biomedcentral.com/1756-0500/4/430 data for a follow-up study of breast cancer treatment California; Group Health Cooperative, Seattle, Washing- effects in older women (BOWII), we conducted a web- ton; Henry Ford Health System, Detroit, Michigan; based (webinar) abstractor training session to standardize HealthPartners, Minneapolis, Minnesota; Fallon Commu- training across six individual Cancer Research Network nity Health Plan, Worcester, Massachusetts; Lovelace (CRN) sites. Evaluation of this web-based platform for Health System, Albuquerque, New Mexico. The study pro- multi-site medical record review instruction would be tocol was approved by the institutional review board at valuable to researchers considering approaches to or plan- each of the participating CRN sites. The CRN is a consor- ning abstractor training. tium of 14 integrated health care delivery systems with over 11 million enrollees. The overall goal of the CRN is Web-based training is increasingly being used in educa- tional and business settings as an effective, low-cost to improve the effectiveness of preventive, curative, and method to teach students and train employees. Although supportive interventions for both major cancers and rare the literature suggests that online training is just as effec- tumors. tive, or slightly more effective than in-person/classroom- based instruction for cognitive and procedural learning Medical Record Abstraction Instrument [1-3] participant-reported satisfaction levels of online BOWII data collection was conducted via medical record instruction have been mixed [2,3]. For example, some stu- review and focused on capture of information related to dies have found participants reporting higher or compar- follow-up care and late treatment effects. The study’selec- able levels of satisfaction with online training courses tronic data collection system (DCS2) consisted of an when compared to in-person instruction [2,4] while other ACCESS database requiring direct data entry of over 600 studies have found participants reporting less satisfaction data elements covering five content areas: 1) demo- with online instruction [1,5]. Differences in satisfaction graphics, 2) surveillance visits, 3) surveillance mammogra- with online instruction may be explained by participants’ phy, 4) recurrences and/or subsequent breast cancer familiarity with the subject content and prior experience diagnoses and 5) comorbidities. The DCS2 captured with this training modality [6,7]. Although some research detailed information on breast cancer cases’ follow-up vis- has been conducted to assess the effectiveness of online its and mammography screenings such as visit dates, rea- medical and nurse training programs [8-11], very little lit- son for visits, type of practitioner seen during visits and erature has been published on the topic of medical record whether a clinical breast exam was performed during vis- abstractor training [12,13] and to the best of the authors’ its. Also captured was whether women had a recurrence knowledge no published studies have examined partici- and/or second primary breast cancer as well as comorbid- pant-reported effectiveness and satisfaction with web- ities existing before or developed after an initial breast based training for medical record abstraction. To address cancer diagnosis. Comorbidities and invasive malignancies, this lack in the literature, we report our experience with a including breast, were captured for matched comparison simultaneous web-based medical record review training subjects. Instructions on how to identify and code each session for epidemiological research purposes across six data element contained in the DCS2 was thoroughly docu- study sites. The goals of this manuscript are to describe mented in the DCS2 coding manual including the data the web-based training session, its participants and partici- element number, definition and synonymous terms, cod- pants’ evaluation of webinar technology for abstraction ing ranges and directives. training. Medical Record Abstractor Training Study Methods A single three-hour webinar session was held for all six The BOWII multi-site cohort study is a follow-up to the data collection sites with the primary purpose of simulta- existing study cohort (BOWI) which extended data col- neously training all study personnel and ensuring consis- lection through five additional years of follow-up and tent approaches to abstraction across all sites. Webinar added a comparison cohort. A detailed description of the participants (hereafter referred to as participants) con- BOWI sampling and data collection procedures has been sisted of chart abstractors as well as other team members published elsewhere [14]. BOWII included women 65+ critical to the success of the study including study investi- years of age diagnosed between 1990 and 1994 with stage gators and project coordinators. Participants either con- I or II breast cancer (N = 1405, breast cancer cases) and nected to the online training session from their own matched comparisons (N = 1405, women without breast computers or from a computer set up in a conference cancer) followed for a maximum of 15 years. The com- room which projected the session onto a screen allowing parison cohort was matched on breast cancer cases’ age, simultaneous viewing by study participants. study site, and breast cancer diagnosis year. Abstractor training was led by a single instructor who is The study was conducted at six CRN sites in the United a Registered Health Information Technician and Certified States of America (USA): Kaiser Permanente, Southern Tumor Registrar with over 10 years experience conducting Avila et al. BMC Research Notes 2011, 4:430 Page 3 of 7 http://www.biomedcentral.com/1756-0500/4/430 research related medical record reviews including abstrac- well as resolutions to these issues, and distributed the tion for the BOWI study. Abstractor training focused on updated Q&A form for discussion during the call. The medical record review and capture of data elements as Q&A form provided documentation of issues raised and defined in the coding manual and data-entered directly decisions made, and was used as a source of reference into the electronic DCS2. material for the abstractors. Conference calls were held for Prior to the training session, participants from the sites the first six months of data collection, with the majority of were asked to pilot test several medical record abstractions questions being addressed during the first three months of using the DCS2. Both content and system issues identified data collection. These monthly calls were discontinued after six months as new issues became infrequent and during pilot testing were sent to the instructor prior to the webinar. These questions and resolutions were compiled abstractors became more experienced. into a standardized question and answer (Q&A) form and disseminated to participants for review in preparation for Post-Webinar Technical Support discussion and demonstration during the webinar. Each abstractor was issued his or her own copy of the The web-based training session involved sequential DCS2 for data collection. Minimal technical support was review of each data element outlined in the coding manual required as almost all programming issues were identified in conjunction with the display of the data entry field andresolvedduringpilottesting.Inthe rare instance, (including drop down menus and labels) in the DCS2. The when an abstractor did experience a technical difficulty, instructor demonstrated how to navigate the data collec- the problem was quickly resolved by the DCS2 developer tion instrument and capture each of the data elements at the lead site. Consequently, delays in data collection contained in the tool. The instructor then proceeded to were minimal. address each question on the Q&A form, while simulta- neously navigating through the DCS2 forms displayed on Inter-Rater Reliability the screen. The instructor engaged trainees in discussion Because the BOWII study included multiple abstractors at by providing helpful hints based on personal chart review multiple sites, inter-rater reliability measures were con- experience and facilitated communication between partici- ducted for each abstractor within each participating site. pants by asking if they had any questions, comments and/ Inter-rater reliability was done approximately three or suggestions that they would like to share with the months after the commencement of data collection and group. The goal of the discussion was to create a common after completion of a minimum of 40 medical record understanding of the data elements to ensure consistency reviews per abstractor (20 breast cancer cases and 20 com- of data collection across abstractors and sites. parisons). The inter-rater reliability electronic data capture system (IRR2) was developed in ACCESS with similar Post-Webinar Training Evaluation front end views as the DCS2 and contained a subset of Post-training evaluation was conducted to assess the effec- 48 key data elements capturing reasons for end-of-study tiveness of the webinar modality for data collection train- follow up, breast cancer recurrence, surveillance mammo- ing and participant satisfaction with the training webinar. graphy, and comorbidity data. Following the conclusion of the webinar training, partici- A sample of ten (5 breast cancer cases and 5 compari- pants were contacted via email requesting their feedback sons) medical records completed by each abstractor were on the training session and provided with the link to parti- randomly selected and a designated independent rater cipate in the survey. Participant evaluations were com- from each site re-abstracted the 48 data elements into the pleted anonymously online using Survey Monkey and IRR2 system to evaluate data quality. The database was took approximately 10 to 15 minutes per person to com- uploaded to a secure website for download and analysis by plete. (See Additional File 1 for participant survey.) the study statistician. The re-abstracted value for each data element was compared to the originally abstracted Post-Webinar Training Support value and percent agreement was computed for each Approximately one week after the webinar training, the abstracted medical record and for the entire study. instructor conducted a follow-up conference call with each of the six sites to resolve any remaining site-specific Results issues or new questions that arose with the commence- Of the 16 people who participated in the webinar train- ment of data collection. Subsequently, the instructor was ing, 10 (62.5%) completed the online survey. As reflected available via email to answer any questions. To share reso- in Table 1, almost all (90.0%) of the 10 participants had lution of issues and ensure consistency of data capture previous medical record abstraction experience and across sites, the instructor held monthly multi-site confer- nearly two-thirds reported over 10 years of medical ence calls. Prior to each conference call, the instructor record review experience. In addition, more than half compiled any new questions received from the sites, as (62.5%) of respondents with prior abstraction experience Avila et al. BMC Research Notes 2011, 4:430 Page 4 of 7 http://www.biomedcentral.com/1756-0500/4/430 Table 1 Participant Abstraction and Online Training Experience Prior to Webinar Survey Question Webinar Participants N= 10 N% Abstracted medical records for research projects before Yes 9 90.0 No 1 10.0 Years of experience abstracting medical records for research < 1 year 1 12.5 1 to 3 years 2 25.0 10 or more years 5 62.5 Abstracted data from paper or electronic medical record directly into an electronic data collection system before Yes 5 62.5 No 2 25.0 Not sure 1 12.5 Previously participated in a webinar Yes, for medical record abstracting for research 1 12.5 Yes, for another type of training 2 25.0 Yes, for non-training purposes 1 12.5 No 4 50.0 NOTE: Counts may not sum to total due to missing data. reported having experience abstracting directly into an more effective or just as effective for medical record review electronic data collection system. Half of the respondents training relative to other types of training modalities. also reported having previously participated in a webinar, Nearly two-thirds of participants (62.5%) rated their ability among which three had previously participated in a webi- to do a chart abstraction as “excellent” or “good” before nar for training purposes. Similarly, participants reported the webinar versus 100% after the training webinar. More- frequent use of the internet for common personal activ- over, all participants reported they would recommend this ities such as shopping/banking/downloading music/social platform for multi-site medical record review training. media (Table 2). Consistent with participants’ reports of the effectiveness of the webinar training, results of inter-rater agreement for Table 3 reflects participants’ evaluation of the webinar platform for abstraction training. All of the respondents data collection within sites ranged from 89.0 to 98.1%, rated the webinar as a good format for chart abstraction with a weighted average of 95.0% agreement across sites training. The majority of participants reported that the (data not shown). Nevertheless, respondents reported a webinar training helped them to better understand the preference for in-person training over a web-based medical record abstraction content (87.5%) and use of the instructional approach, and site-specific webinar training data collection system (75.0%). In addition, the majority of over multi-site web-based training (Figure 1). participants (87.5%) rated the webinar highly on its ability to facilitate discussion of questions and issues. All rated Discussion the knowledge and information delivered through the Collecting complex medical record data presents consid- webinar as useful and reported that the webinar ade- erable challenges in multi-site studies including standar- quately prepared them for data collection. Almost three- dization of data collection procedures, and time and quarters of participants reported the webinar training was resource utilization associated with in-person abstractor Table 2 Webinar Participants Use of Internet (N = 10) Internet Usage Skype Social Media Download Music/ Online Banking/ Video/Docs Shopping N% N % N % N % At least once a week 1 12.5 5 62.5 3 42.9 6 75.0 At least once a month 0 00.0 1 12.5 2 28.6 1 12.5 Couple times a year 2 25.0 0 00.0 0 00.0 1 12.5 Never 5 62.5 2 25.0 2 28.6 0 00.0 NOTE: Counts may not sum to total due to missing data. Avila et al. BMC Research Notes 2011, 4:430 Page 5 of 7 http://www.biomedcentral.com/1756-0500/4/430 Table 3 Participant Rating of Webinar for Abstraction Training Survey Question Webinar Participants N=10 N% Webinar is a good format for chart abstraction training Yes, very much 2 25.0 Yes, to some extent 6 75.0 Webinar helped me better understand medical record abstraction content Strongly agree 1 12.5 Agree 6 75.0 Neither agree nor disagree 1 12.5 Webinar helped me better understand use of DCS2 data entry system Strongly agree 2 25.00 Agree 4 50.00 Neither agree nor disagree 2 25.00 Webinar format facilitated discussion of questions and issues Strongly agree 3 37.50 Agree 4 50.00 Neither agree nor disagree 1 12.50 Felt comfortable asking questions/participating in discussions during webinar Strongly agree 4 50.00 Agree 3 37.50 Neither agree nor disagree 1 12.50 Knowledge and information delivered through webinar Very useful 4 50.0 Useful 4 50.0 How well has the webinar prepared you for abstraction using the DCS2 More than adequate 6 75.00 Adequate 2 25.00 Effectiveness of webinar format for abstraction training relative to other types of training More effective 1 14.29 About the same 4 57.14 Less effective 2 28.57 Ability to do a medical record abstraction using the DCS2 before webinar Excellent 1 12.50 Good 4 50.00 Fair 3 37.50 Ability to do a medical record abstraction using the DCS2 after webinar Excellent 3 37.50 Good 5 62.50 Would recommend webinar training for medical record abstractors working on future research projects that involve multiple sites Yes 5 71.43 Probably 2 28.57 NOTE: Counts may not sum to total due to missing data. training. To address these issues, the BOW II study Reports from the post-training survey suggest web- adopted webinar technology to orient and train abstrac- based instruction is a viable cost- saving alternative to tors from six diverse health plans across the USA. The in-person training with abstractors traveling to a central webinar session proved to be an acceptable, feasible and location or an instructor traveling to individual sites. effective component of a comprehensive effort to ensure For example, one-day in-person visits to the six partici- high quality data collection. pating sites could cost as much as six-thousand dollars Avila et al. BMC Research Notes 2011, 4:430 Page 6 of 7 http://www.biomedcentral.com/1756-0500/4/430 Figure 1 Webinar Participants’ Training Preferences. Legend text: In-Person, Just My Site. In-Person, Multiple Sites. Webinar, Just My Site. Webinar, Multiple sites. more personal experience which may explain why partici- compared to a cost of one-hundred and seventy-three dollars for a three-hour training webinar (assuming an pants stated a preference for in-person training. Never- average cost of $1,000 for airfare and one night’shotel theless, given the high quality of data collection based on accommodations for the instructor vs six cents a minute IRR results, coupled with the ability of the webinar to per caller for a three hour webinar). Webinar training facilitate rapport between sites, the substantial time and also affords considerable time savings (e.g. minimum of cost savings achieved, and participants’ positive evalua- 1 day per site for instructor travel vs three hours for a tion of the webinar session, researchers should consider webinar). web-based training for use in multi-site studies. Importantly, the results of the post-training inter-rater assessment support the effectiveness of using webinar Conclusions technology as an integral part of training abstractors to Conducting medical record abstraction training via web- produce reliable results. The webinar not only provided based technology was an acceptable and effective abstractors with consistent instruction and the ability to approach to assist in standardizing a complex medical learn from others’ questions, but it also fostered commu- record review across six health plans. Researchers nication between participants at the various sites which should consider this cost-effective instructional method set the stage for ongoing interactions between the as part of training efforts to ensure high quality data abstractors. The rapport developed during the webinar collection in multi-site studies. facilitated open discussion during the subsequent multi- site Q&A calls, contributing to consistency of abstraction Additional material across sites and, in turn, resulting in high quality data collection. Of note, the supplemental support trainings Additional file 1: Post Webinar Training Evaluation Survey. Description: Screen shots of post webinar training evaluation survey. would have been conducted as part of our comprehensive training approach regardless of the modality chosen to conduct the multi-site medical record review training. In addition to the webinar being an effective training Acknowledgements The BOWII study is supported by Public Health Service grant R01CA093772- modality, it led to improvements in the coding manual 05A2 (Silliman, PI) from the National Cancer Institute, National Institutes of and electronic data system (e.g. resolution of errors iden- Heath, Department of Health and Human Services. We thank the chart tified in the coding manual and inconsistencies between abstractors, programmers, site project managers and the principle investigators for their contributions to the data collection and data the coding manual and DCS2), as well as the identifica- management of the study including: Rebecca A. Silliman, Jaclyn L.F. Bosco tion of site-specific issues and the standardization of data and Soe Soe Thwin at Boston University Medical Center, Section of collection procedures. Geriatrics; Terry Field and Hassan Fouayzi at Fallon Clinic/Meyers Primary Care Institute, University of Massachusetts Medical School; Marianne Ulcickas Our results are limited by the small sample size and Yood at Henry Ford Healthcare System/Yale University School of Medicine; may not generalize to abstractors with less experience in Rita Montague at Henry Ford Healthcare System; Diana S.M. Buist at Group medical record review or lack of familiarity with the Health Cooperative; Feifei Wei at HealthPartners. internet. There are distinct advantages to in-person train- Author details ing such as the opportunity to observe participants’ per- Kaiser Permanente Southern California, Research & Evaluation, Pasadena, CA, formance and provide one-on-one instruction. It is also a USA. Division of Public Health Sciences, Wake Forest University School of Avila et al. BMC Research Notes 2011, 4:430 Page 7 of 7 http://www.biomedcentral.com/1756-0500/4/430 Medicine, Winston-Salem, NC, USA. HealthPartners Research Foundation, Minneapolis, MN, USA. Clinical and Outcomes Research, Lovelace Respiratory Research Institute, Albuquerque, NM, USA. Section of Geriatrics, Boston University Medical Center, Boston, MA, USA. Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, Switzerland. National Institute of Cancer Epidemiology and Registration (NICER) Institute of Social and Preventive Medicine (ISPM), University of Zürich, Zürich, Switzerland. Authors’ contributions All authors have contributed to the development of the manuscript. AG and KCG co-designed the Post Webinar Training Evaluation Survey. TK and MSC conducted the literature review. CA conducted the data analysis, interpretation of results and the drafting of the manuscript. VQ, KCG and TK assisted in the interpretation of results. VQ and KCG partipated in substantive revisions to the draft manuscript. All authors have read and approved the manuscript. Competing interests The authors declare that they have no competing interests. Received: 19 February 2011 Accepted: 20 October 2011 Published: 20 October 2011 References 1. Rivera J, Rice M: A comparison of student outcomes & satisfaction betweenyu traditional & Web based course offerings. Online Journal of Distance Learning Administration 2002. 2. Schimming LM: Measuring medical student preference: a comparison of classroom versus online instruction for teaching PubMed. J Med Libr Assoc 2008, 96(3):217-222. 3. Goldberg HR, Haase E, Shoukas A: Redefining classroom instruction. Adv Physiol Educ 2006, 30(3):124-127. 4. Campbell M, Floyd J, Sheridan JB: Assessment of student performance and attitudes for courses taught online versus onsite. The Journal of Applied Business Research 2002, 18(2):45-51. 5. Carr: Online Psychology Instruction is effective but not satisfying, study finds. Chronicle of Higher Education 2002, 46(27)-A48, 2/5p. 6. Chen YC: Evaluating the learning effectiveness of using web-based instruction: An individual differences approach. International Journal of Information and Communication Technology Education 2005, 1(1):69-82. 7. Carswell L, et al: Distant education via the Internet: the student experience. British Journal of Education Technology 2000, 31(1):29-46. 8. Tuttle BD, Von Isenburg M, Schardt C: PubMed instruction for medical students: searching for a better way. Med Ref Serv Q 2009, 28(3):199-210. 9. Premkumar K, Ross AG, J L: Technology-enhanced learning of community health in undergraduate medical education. Can J Public Health 2010, 101(2):165-170. 10. Casebeer L, Brown J, Roepke N: Evidence-based choices of physicians: a comparative analysis of physicians participating in Internet CME and non-participants. BMC Med Educ 2010, 10:10-42. 11. Sisson SD, Hill-Briggs F, D L: How to improve medical education website design. BMC Med Educ 2010, 21:10-30. 12. Reisch LM, et al: Training, quality assurance, and assessment of medical record abstraction in a multisite study. Am J Epidemiol 2003, , 157: 546-551. 13. Liddy C, Wiens M, W H: Methods to achieve high interrater reliability in data collection from primary care records. Ann Fam Med 2011, 19(1):57-62. Submit your next manuscript to BioMed Central 14. Enger SM, et al: Breast cancer treatment of older women in integrated health care settings. J Clin Oncol 2006, 24:4377-4383. and take full advantage of: doi:10.1186/1756-0500-4-430 • Convenient online submission Cite this article as: Avila et al.: Webinar Training: an acceptable, feasible • Thorough peer review and effective approach for multi-site medical record abstraction: the BOWII experience. BMC Research Notes 2011 4:430. • No space constraints or color figure charges • Immediate publication on acceptance • Inclusion in PubMed, CAS, Scopus and Google Scholar • Research which is freely available for redistribution Submit your manuscript at www.biomedcentral.com/submit http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png BMC Research Notes Springer Journals

Webinar Training: an acceptable, feasible and effective approach for multi-site medical record abstraction: the BOWII experience

Loading next page...
 
/lp/springer-journals/webinar-training-an-acceptable-feasible-and-effective-approach-for-xiH5YodxLY

References (20)

Publisher
Springer Journals
Copyright
Copyright © 2011 by Avila et al; licensee BioMed Central Ltd.
Subject
Biomedicine; Biomedicine general; Medicine/Public Health, general; Life Sciences, general
eISSN
1756-0500
DOI
10.1186/1756-0500-4-430
pmid
22013969
Publisher site
See Article on Publisher Site

Abstract

Background: Abstractor training is a key element in creating valid and reliable data collection procedures. The choice between in-person vs. remote or simultaneous vs. sequential abstractor training has considerable consequences for time and resource utilization. We conducted a web-based (webinar) abstractor training session to standardize training across six individual Cancer Research Network (CRN) sites for a study of breast cancer treatment effects in older women (BOWII). The goals of this manuscript are to describe the training session, its participants and participants’ evaluation of webinar technology for abstraction training. Findings: A webinar was held for all six sites with the primary purpose of simultaneously training staff and ensuring consistent abstraction across sites. The training session involved sequential review of over 600 data elements outlined in the coding manual in conjunction with the display of data entry fields in the study’s electronic data collection system. Post-training evaluation was conducted via Survey Monkey . Inter-rater reliability measures for abstractors within each site were conducted three months after the commencement of data collection. Ten of the 16 people who participated in the training completed the online survey. Almost all (90%) of the 10 trainees had previous medical record abstraction experience and nearly two-thirds reported over 10 years of experience. Half of the respondents had previously participated in a webinar, among which three had participated in a webinar for training purposes. All rated the knowledge and information delivered through the webinar as useful and reported it adequately prepared them for data collection. Moreover, all participants would recommend this platform for multi-site abstraction training. Consistent with participant-reported training effectiveness, results of data collection inter-rater agreement within sites ranged from 89 to 98%, with a weighted average of 95% agreement across sites. Conclusions: Conducting training via web-based technology was an acceptable and effective approach to standardizing medical record review across multiple sites for this group of experienced abstractors. Given the substantial time and cost savings achieved with the webinar, coupled with participants’ positive evaluation of the training session, researchers should consider this instructional method as part of training efforts to ensure high quality data collection in multi-site studies. Introduction site data and differences in quality of data abstraction Medical record review is a common data collection between sites can introduce variability. Abstractor training method for conducting epidemiologic research. Although is a key element in minimizing interobserver variability to investigators in sites from various geographical locations create reliable data collection procedures. The choice often collaborate to include diverse populations to between in-person vs. remote or simultaneous vs. sequen- enhance generalizability of study results, idiosyncrasies of tial abstractor training has considerable consequences for time and resource utilization. Advances in information * Correspondence: Chantal.C.Avila@kp.org technology have produced readily available, low cost, effi- Kaiser Permanente Southern California, Research & Evaluation, Pasadena, CA, cient alternatives to traditional training approaches. Faced USA with the challenges of collecting complex medical record Full list of author information is available at the end of the article © 2011 Avila et al; licensee BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Avila et al. BMC Research Notes 2011, 4:430 Page 2 of 7 http://www.biomedcentral.com/1756-0500/4/430 data for a follow-up study of breast cancer treatment California; Group Health Cooperative, Seattle, Washing- effects in older women (BOWII), we conducted a web- ton; Henry Ford Health System, Detroit, Michigan; based (webinar) abstractor training session to standardize HealthPartners, Minneapolis, Minnesota; Fallon Commu- training across six individual Cancer Research Network nity Health Plan, Worcester, Massachusetts; Lovelace (CRN) sites. Evaluation of this web-based platform for Health System, Albuquerque, New Mexico. The study pro- multi-site medical record review instruction would be tocol was approved by the institutional review board at valuable to researchers considering approaches to or plan- each of the participating CRN sites. The CRN is a consor- ning abstractor training. tium of 14 integrated health care delivery systems with over 11 million enrollees. The overall goal of the CRN is Web-based training is increasingly being used in educa- tional and business settings as an effective, low-cost to improve the effectiveness of preventive, curative, and method to teach students and train employees. Although supportive interventions for both major cancers and rare the literature suggests that online training is just as effec- tumors. tive, or slightly more effective than in-person/classroom- based instruction for cognitive and procedural learning Medical Record Abstraction Instrument [1-3] participant-reported satisfaction levels of online BOWII data collection was conducted via medical record instruction have been mixed [2,3]. For example, some stu- review and focused on capture of information related to dies have found participants reporting higher or compar- follow-up care and late treatment effects. The study’selec- able levels of satisfaction with online training courses tronic data collection system (DCS2) consisted of an when compared to in-person instruction [2,4] while other ACCESS database requiring direct data entry of over 600 studies have found participants reporting less satisfaction data elements covering five content areas: 1) demo- with online instruction [1,5]. Differences in satisfaction graphics, 2) surveillance visits, 3) surveillance mammogra- with online instruction may be explained by participants’ phy, 4) recurrences and/or subsequent breast cancer familiarity with the subject content and prior experience diagnoses and 5) comorbidities. The DCS2 captured with this training modality [6,7]. Although some research detailed information on breast cancer cases’ follow-up vis- has been conducted to assess the effectiveness of online its and mammography screenings such as visit dates, rea- medical and nurse training programs [8-11], very little lit- son for visits, type of practitioner seen during visits and erature has been published on the topic of medical record whether a clinical breast exam was performed during vis- abstractor training [12,13] and to the best of the authors’ its. Also captured was whether women had a recurrence knowledge no published studies have examined partici- and/or second primary breast cancer as well as comorbid- pant-reported effectiveness and satisfaction with web- ities existing before or developed after an initial breast based training for medical record abstraction. To address cancer diagnosis. Comorbidities and invasive malignancies, this lack in the literature, we report our experience with a including breast, were captured for matched comparison simultaneous web-based medical record review training subjects. Instructions on how to identify and code each session for epidemiological research purposes across six data element contained in the DCS2 was thoroughly docu- study sites. The goals of this manuscript are to describe mented in the DCS2 coding manual including the data the web-based training session, its participants and partici- element number, definition and synonymous terms, cod- pants’ evaluation of webinar technology for abstraction ing ranges and directives. training. Medical Record Abstractor Training Study Methods A single three-hour webinar session was held for all six The BOWII multi-site cohort study is a follow-up to the data collection sites with the primary purpose of simulta- existing study cohort (BOWI) which extended data col- neously training all study personnel and ensuring consis- lection through five additional years of follow-up and tent approaches to abstraction across all sites. Webinar added a comparison cohort. A detailed description of the participants (hereafter referred to as participants) con- BOWI sampling and data collection procedures has been sisted of chart abstractors as well as other team members published elsewhere [14]. BOWII included women 65+ critical to the success of the study including study investi- years of age diagnosed between 1990 and 1994 with stage gators and project coordinators. Participants either con- I or II breast cancer (N = 1405, breast cancer cases) and nected to the online training session from their own matched comparisons (N = 1405, women without breast computers or from a computer set up in a conference cancer) followed for a maximum of 15 years. The com- room which projected the session onto a screen allowing parison cohort was matched on breast cancer cases’ age, simultaneous viewing by study participants. study site, and breast cancer diagnosis year. Abstractor training was led by a single instructor who is The study was conducted at six CRN sites in the United a Registered Health Information Technician and Certified States of America (USA): Kaiser Permanente, Southern Tumor Registrar with over 10 years experience conducting Avila et al. BMC Research Notes 2011, 4:430 Page 3 of 7 http://www.biomedcentral.com/1756-0500/4/430 research related medical record reviews including abstrac- well as resolutions to these issues, and distributed the tion for the BOWI study. Abstractor training focused on updated Q&A form for discussion during the call. The medical record review and capture of data elements as Q&A form provided documentation of issues raised and defined in the coding manual and data-entered directly decisions made, and was used as a source of reference into the electronic DCS2. material for the abstractors. Conference calls were held for Prior to the training session, participants from the sites the first six months of data collection, with the majority of were asked to pilot test several medical record abstractions questions being addressed during the first three months of using the DCS2. Both content and system issues identified data collection. These monthly calls were discontinued after six months as new issues became infrequent and during pilot testing were sent to the instructor prior to the webinar. These questions and resolutions were compiled abstractors became more experienced. into a standardized question and answer (Q&A) form and disseminated to participants for review in preparation for Post-Webinar Technical Support discussion and demonstration during the webinar. Each abstractor was issued his or her own copy of the The web-based training session involved sequential DCS2 for data collection. Minimal technical support was review of each data element outlined in the coding manual required as almost all programming issues were identified in conjunction with the display of the data entry field andresolvedduringpilottesting.Inthe rare instance, (including drop down menus and labels) in the DCS2. The when an abstractor did experience a technical difficulty, instructor demonstrated how to navigate the data collec- the problem was quickly resolved by the DCS2 developer tion instrument and capture each of the data elements at the lead site. Consequently, delays in data collection contained in the tool. The instructor then proceeded to were minimal. address each question on the Q&A form, while simulta- neously navigating through the DCS2 forms displayed on Inter-Rater Reliability the screen. The instructor engaged trainees in discussion Because the BOWII study included multiple abstractors at by providing helpful hints based on personal chart review multiple sites, inter-rater reliability measures were con- experience and facilitated communication between partici- ducted for each abstractor within each participating site. pants by asking if they had any questions, comments and/ Inter-rater reliability was done approximately three or suggestions that they would like to share with the months after the commencement of data collection and group. The goal of the discussion was to create a common after completion of a minimum of 40 medical record understanding of the data elements to ensure consistency reviews per abstractor (20 breast cancer cases and 20 com- of data collection across abstractors and sites. parisons). The inter-rater reliability electronic data capture system (IRR2) was developed in ACCESS with similar Post-Webinar Training Evaluation front end views as the DCS2 and contained a subset of Post-training evaluation was conducted to assess the effec- 48 key data elements capturing reasons for end-of-study tiveness of the webinar modality for data collection train- follow up, breast cancer recurrence, surveillance mammo- ing and participant satisfaction with the training webinar. graphy, and comorbidity data. Following the conclusion of the webinar training, partici- A sample of ten (5 breast cancer cases and 5 compari- pants were contacted via email requesting their feedback sons) medical records completed by each abstractor were on the training session and provided with the link to parti- randomly selected and a designated independent rater cipate in the survey. Participant evaluations were com- from each site re-abstracted the 48 data elements into the pleted anonymously online using Survey Monkey and IRR2 system to evaluate data quality. The database was took approximately 10 to 15 minutes per person to com- uploaded to a secure website for download and analysis by plete. (See Additional File 1 for participant survey.) the study statistician. The re-abstracted value for each data element was compared to the originally abstracted Post-Webinar Training Support value and percent agreement was computed for each Approximately one week after the webinar training, the abstracted medical record and for the entire study. instructor conducted a follow-up conference call with each of the six sites to resolve any remaining site-specific Results issues or new questions that arose with the commence- Of the 16 people who participated in the webinar train- ment of data collection. Subsequently, the instructor was ing, 10 (62.5%) completed the online survey. As reflected available via email to answer any questions. To share reso- in Table 1, almost all (90.0%) of the 10 participants had lution of issues and ensure consistency of data capture previous medical record abstraction experience and across sites, the instructor held monthly multi-site confer- nearly two-thirds reported over 10 years of medical ence calls. Prior to each conference call, the instructor record review experience. In addition, more than half compiled any new questions received from the sites, as (62.5%) of respondents with prior abstraction experience Avila et al. BMC Research Notes 2011, 4:430 Page 4 of 7 http://www.biomedcentral.com/1756-0500/4/430 Table 1 Participant Abstraction and Online Training Experience Prior to Webinar Survey Question Webinar Participants N= 10 N% Abstracted medical records for research projects before Yes 9 90.0 No 1 10.0 Years of experience abstracting medical records for research < 1 year 1 12.5 1 to 3 years 2 25.0 10 or more years 5 62.5 Abstracted data from paper or electronic medical record directly into an electronic data collection system before Yes 5 62.5 No 2 25.0 Not sure 1 12.5 Previously participated in a webinar Yes, for medical record abstracting for research 1 12.5 Yes, for another type of training 2 25.0 Yes, for non-training purposes 1 12.5 No 4 50.0 NOTE: Counts may not sum to total due to missing data. reported having experience abstracting directly into an more effective or just as effective for medical record review electronic data collection system. Half of the respondents training relative to other types of training modalities. also reported having previously participated in a webinar, Nearly two-thirds of participants (62.5%) rated their ability among which three had previously participated in a webi- to do a chart abstraction as “excellent” or “good” before nar for training purposes. Similarly, participants reported the webinar versus 100% after the training webinar. More- frequent use of the internet for common personal activ- over, all participants reported they would recommend this ities such as shopping/banking/downloading music/social platform for multi-site medical record review training. media (Table 2). Consistent with participants’ reports of the effectiveness of the webinar training, results of inter-rater agreement for Table 3 reflects participants’ evaluation of the webinar platform for abstraction training. All of the respondents data collection within sites ranged from 89.0 to 98.1%, rated the webinar as a good format for chart abstraction with a weighted average of 95.0% agreement across sites training. The majority of participants reported that the (data not shown). Nevertheless, respondents reported a webinar training helped them to better understand the preference for in-person training over a web-based medical record abstraction content (87.5%) and use of the instructional approach, and site-specific webinar training data collection system (75.0%). In addition, the majority of over multi-site web-based training (Figure 1). participants (87.5%) rated the webinar highly on its ability to facilitate discussion of questions and issues. All rated Discussion the knowledge and information delivered through the Collecting complex medical record data presents consid- webinar as useful and reported that the webinar ade- erable challenges in multi-site studies including standar- quately prepared them for data collection. Almost three- dization of data collection procedures, and time and quarters of participants reported the webinar training was resource utilization associated with in-person abstractor Table 2 Webinar Participants Use of Internet (N = 10) Internet Usage Skype Social Media Download Music/ Online Banking/ Video/Docs Shopping N% N % N % N % At least once a week 1 12.5 5 62.5 3 42.9 6 75.0 At least once a month 0 00.0 1 12.5 2 28.6 1 12.5 Couple times a year 2 25.0 0 00.0 0 00.0 1 12.5 Never 5 62.5 2 25.0 2 28.6 0 00.0 NOTE: Counts may not sum to total due to missing data. Avila et al. BMC Research Notes 2011, 4:430 Page 5 of 7 http://www.biomedcentral.com/1756-0500/4/430 Table 3 Participant Rating of Webinar for Abstraction Training Survey Question Webinar Participants N=10 N% Webinar is a good format for chart abstraction training Yes, very much 2 25.0 Yes, to some extent 6 75.0 Webinar helped me better understand medical record abstraction content Strongly agree 1 12.5 Agree 6 75.0 Neither agree nor disagree 1 12.5 Webinar helped me better understand use of DCS2 data entry system Strongly agree 2 25.00 Agree 4 50.00 Neither agree nor disagree 2 25.00 Webinar format facilitated discussion of questions and issues Strongly agree 3 37.50 Agree 4 50.00 Neither agree nor disagree 1 12.50 Felt comfortable asking questions/participating in discussions during webinar Strongly agree 4 50.00 Agree 3 37.50 Neither agree nor disagree 1 12.50 Knowledge and information delivered through webinar Very useful 4 50.0 Useful 4 50.0 How well has the webinar prepared you for abstraction using the DCS2 More than adequate 6 75.00 Adequate 2 25.00 Effectiveness of webinar format for abstraction training relative to other types of training More effective 1 14.29 About the same 4 57.14 Less effective 2 28.57 Ability to do a medical record abstraction using the DCS2 before webinar Excellent 1 12.50 Good 4 50.00 Fair 3 37.50 Ability to do a medical record abstraction using the DCS2 after webinar Excellent 3 37.50 Good 5 62.50 Would recommend webinar training for medical record abstractors working on future research projects that involve multiple sites Yes 5 71.43 Probably 2 28.57 NOTE: Counts may not sum to total due to missing data. training. To address these issues, the BOW II study Reports from the post-training survey suggest web- adopted webinar technology to orient and train abstrac- based instruction is a viable cost- saving alternative to tors from six diverse health plans across the USA. The in-person training with abstractors traveling to a central webinar session proved to be an acceptable, feasible and location or an instructor traveling to individual sites. effective component of a comprehensive effort to ensure For example, one-day in-person visits to the six partici- high quality data collection. pating sites could cost as much as six-thousand dollars Avila et al. BMC Research Notes 2011, 4:430 Page 6 of 7 http://www.biomedcentral.com/1756-0500/4/430 Figure 1 Webinar Participants’ Training Preferences. Legend text: In-Person, Just My Site. In-Person, Multiple Sites. Webinar, Just My Site. Webinar, Multiple sites. more personal experience which may explain why partici- compared to a cost of one-hundred and seventy-three dollars for a three-hour training webinar (assuming an pants stated a preference for in-person training. Never- average cost of $1,000 for airfare and one night’shotel theless, given the high quality of data collection based on accommodations for the instructor vs six cents a minute IRR results, coupled with the ability of the webinar to per caller for a three hour webinar). Webinar training facilitate rapport between sites, the substantial time and also affords considerable time savings (e.g. minimum of cost savings achieved, and participants’ positive evalua- 1 day per site for instructor travel vs three hours for a tion of the webinar session, researchers should consider webinar). web-based training for use in multi-site studies. Importantly, the results of the post-training inter-rater assessment support the effectiveness of using webinar Conclusions technology as an integral part of training abstractors to Conducting medical record abstraction training via web- produce reliable results. The webinar not only provided based technology was an acceptable and effective abstractors with consistent instruction and the ability to approach to assist in standardizing a complex medical learn from others’ questions, but it also fostered commu- record review across six health plans. Researchers nication between participants at the various sites which should consider this cost-effective instructional method set the stage for ongoing interactions between the as part of training efforts to ensure high quality data abstractors. The rapport developed during the webinar collection in multi-site studies. facilitated open discussion during the subsequent multi- site Q&A calls, contributing to consistency of abstraction Additional material across sites and, in turn, resulting in high quality data collection. Of note, the supplemental support trainings Additional file 1: Post Webinar Training Evaluation Survey. Description: Screen shots of post webinar training evaluation survey. would have been conducted as part of our comprehensive training approach regardless of the modality chosen to conduct the multi-site medical record review training. In addition to the webinar being an effective training Acknowledgements The BOWII study is supported by Public Health Service grant R01CA093772- modality, it led to improvements in the coding manual 05A2 (Silliman, PI) from the National Cancer Institute, National Institutes of and electronic data system (e.g. resolution of errors iden- Heath, Department of Health and Human Services. We thank the chart tified in the coding manual and inconsistencies between abstractors, programmers, site project managers and the principle investigators for their contributions to the data collection and data the coding manual and DCS2), as well as the identifica- management of the study including: Rebecca A. Silliman, Jaclyn L.F. Bosco tion of site-specific issues and the standardization of data and Soe Soe Thwin at Boston University Medical Center, Section of collection procedures. Geriatrics; Terry Field and Hassan Fouayzi at Fallon Clinic/Meyers Primary Care Institute, University of Massachusetts Medical School; Marianne Ulcickas Our results are limited by the small sample size and Yood at Henry Ford Healthcare System/Yale University School of Medicine; may not generalize to abstractors with less experience in Rita Montague at Henry Ford Healthcare System; Diana S.M. Buist at Group medical record review or lack of familiarity with the Health Cooperative; Feifei Wei at HealthPartners. internet. There are distinct advantages to in-person train- Author details ing such as the opportunity to observe participants’ per- Kaiser Permanente Southern California, Research & Evaluation, Pasadena, CA, formance and provide one-on-one instruction. It is also a USA. Division of Public Health Sciences, Wake Forest University School of Avila et al. BMC Research Notes 2011, 4:430 Page 7 of 7 http://www.biomedcentral.com/1756-0500/4/430 Medicine, Winston-Salem, NC, USA. HealthPartners Research Foundation, Minneapolis, MN, USA. Clinical and Outcomes Research, Lovelace Respiratory Research Institute, Albuquerque, NM, USA. Section of Geriatrics, Boston University Medical Center, Boston, MA, USA. Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, Switzerland. National Institute of Cancer Epidemiology and Registration (NICER) Institute of Social and Preventive Medicine (ISPM), University of Zürich, Zürich, Switzerland. Authors’ contributions All authors have contributed to the development of the manuscript. AG and KCG co-designed the Post Webinar Training Evaluation Survey. TK and MSC conducted the literature review. CA conducted the data analysis, interpretation of results and the drafting of the manuscript. VQ, KCG and TK assisted in the interpretation of results. VQ and KCG partipated in substantive revisions to the draft manuscript. All authors have read and approved the manuscript. Competing interests The authors declare that they have no competing interests. Received: 19 February 2011 Accepted: 20 October 2011 Published: 20 October 2011 References 1. Rivera J, Rice M: A comparison of student outcomes & satisfaction betweenyu traditional & Web based course offerings. Online Journal of Distance Learning Administration 2002. 2. Schimming LM: Measuring medical student preference: a comparison of classroom versus online instruction for teaching PubMed. J Med Libr Assoc 2008, 96(3):217-222. 3. Goldberg HR, Haase E, Shoukas A: Redefining classroom instruction. Adv Physiol Educ 2006, 30(3):124-127. 4. Campbell M, Floyd J, Sheridan JB: Assessment of student performance and attitudes for courses taught online versus onsite. The Journal of Applied Business Research 2002, 18(2):45-51. 5. Carr: Online Psychology Instruction is effective but not satisfying, study finds. Chronicle of Higher Education 2002, 46(27)-A48, 2/5p. 6. Chen YC: Evaluating the learning effectiveness of using web-based instruction: An individual differences approach. International Journal of Information and Communication Technology Education 2005, 1(1):69-82. 7. Carswell L, et al: Distant education via the Internet: the student experience. British Journal of Education Technology 2000, 31(1):29-46. 8. Tuttle BD, Von Isenburg M, Schardt C: PubMed instruction for medical students: searching for a better way. Med Ref Serv Q 2009, 28(3):199-210. 9. Premkumar K, Ross AG, J L: Technology-enhanced learning of community health in undergraduate medical education. Can J Public Health 2010, 101(2):165-170. 10. Casebeer L, Brown J, Roepke N: Evidence-based choices of physicians: a comparative analysis of physicians participating in Internet CME and non-participants. BMC Med Educ 2010, 10:10-42. 11. Sisson SD, Hill-Briggs F, D L: How to improve medical education website design. BMC Med Educ 2010, 21:10-30. 12. Reisch LM, et al: Training, quality assurance, and assessment of medical record abstraction in a multisite study. Am J Epidemiol 2003, , 157: 546-551. 13. Liddy C, Wiens M, W H: Methods to achieve high interrater reliability in data collection from primary care records. Ann Fam Med 2011, 19(1):57-62. Submit your next manuscript to BioMed Central 14. Enger SM, et al: Breast cancer treatment of older women in integrated health care settings. J Clin Oncol 2006, 24:4377-4383. and take full advantage of: doi:10.1186/1756-0500-4-430 • Convenient online submission Cite this article as: Avila et al.: Webinar Training: an acceptable, feasible • Thorough peer review and effective approach for multi-site medical record abstraction: the BOWII experience. BMC Research Notes 2011 4:430. • No space constraints or color figure charges • Immediate publication on acceptance • Inclusion in PubMed, CAS, Scopus and Google Scholar • Research which is freely available for redistribution Submit your manuscript at www.biomedcentral.com/submit

Journal

BMC Research NotesSpringer Journals

Published: Oct 20, 2011

There are no references for this article.