|
 
 |
| SHORT COMMUNICATION |
|
| Year : 2022 | Volume
: 8
| Issue : 1 | Page : 22 |
|
Implementation of a digital health curriculum for health-care students
Karoliina X Yang1, Benjamin I Rosner2, Steven R Chan3, Ralph G Brindis4, Tiffany M Lee5, Raman R Khanna5, Andrew D Auerbach5
1 College of Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
2 Division of Hospital Medicine; Center for Clinical Informatics and Improvement Research; Institute for Health Policy Studies, University of California San Francisco, San Francisco, California, USA
3 Division of Hospital Medicine, University of California, San Francisco, San Francisco; Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, California, USA
4 Institute for Health Policy Studies, University of California San Francisco; Division of Cardiology, University of California, San Francisco, San Francisco, California, USA
5 Division of Hospital Medicine, University of California, San Francisco, San Francisco, California, USA
| Date of Submission | 07-Apr-2022 |
| Date of Decision | 29-Jun-2022 |
| Date of Acceptance | 18-Jul-2022 |
| Date of Web Publication | 27-Sep-2022 |
Correspondence Address:
Karoliina X Yang
University of Cincinnati, 2707 Edroy Court, Cincinnati, Ohio, 45209
USA
 Source of Support: None, Conflict of Interest: None  | Check |
DOI: 10.4103/digm.digm_16_22
Introduction: Digital health software technologies (DHSTs) have the potential to shift models of care delivery and improve patient outcomes. Health-care students lack training to prepare them for the growing breadth and depth of DHSTs that they will need to identify, assess, and use in their future clinical careers. The implementation of this pilot digital health curriculum aimed to provide health-care trainees skills to identify, assess, and use DHSTs in their future clinical careers. Materials and Methods: From June to August of 2021, the authors implemented a 10 weeks pilot educational internship in partnership with the Accelerated Digital Clinical Ecosystem (ADviCE), a multi-institutional collaborative created by physicians within University of California, San Francisco Health Informatics. Interns completed a customized digital health didactic program developed by program faculty, including a speaker series with clinical informaticists in the field, journal club discussions, and intern-led presentations. In addition, interns were given real-world educational opportunities to directly interact with DHST vendors in the process of creating a DHST registry. Interns also completed a postcourse survey to procure feedback about the pilot program's strengths and weaknesses. Results: Interns learned from a didactic educational curriculum that included more than 10 h of live lectures, 3 supplemental readings, and structured interactions with 74 DHST vendors. Based on a survey to assess interns' satisfaction with the internship, interns ranked the expert speaker series the highest with an average standard deviation ranking on a 5-point Likert scale of 4.6 (0.57) and journal club discussions the lowest with a ranking of 4.31 (0.96). Interns also gave feedback on recommending the internship to future students with a ranking of 4.6 (0.65). Conclusion: This first internship led to the positively favored and successful execution of a remote educational experience exploring digital health in medicine, while also giving interns direct interaction with a broad array of digital health companies.
Keywords: Digital health software technologies, Digital medicine, Health-care student, Health care, Medical education
How to cite this article:
Yang KX, Rosner BI, Chan SR, Brindis RG, Lee TM, Khanna RR, Auerbach AD. Implementation of a digital health curriculum for health-care students. Digit Med 2022;8:22 |
How to cite this URL:
Yang KX, Rosner BI, Chan SR, Brindis RG, Lee TM, Khanna RR, Auerbach AD. Implementation of a digital health curriculum for health-care students. Digit Med [serial online] 2022 [cited 2023 Jun 9];8:22. Available from: http://www.digitmedicine.com/text.asp?2022/8/1/22/357207 |
| Introduction | |  |
Digital health is a growing field described by the U.S. Food and Drug Administration (FDA) as including “diverse categories such as mobile health, health information technology, wearable devices, telehealth and telemedicine, and personalized medicine”.[1] Although there is evidence to show that some digital health software technologies (DHSTs) may improve patient outcomes in certain settings and medical specialties (e.g., cardiovascular disease, obesity, and mental health),[2],[3],[4] the implementation of DHSTs continues to face numerous barriers,[5],[6] not the least of which is lack of familiarity by the very clinicians prescribing or recommending their use to patients.
The regulatory landscape to assure the safety, efficacy, and security of DHSTs continues to evolve.[7] Current health-care providers and those in training who are recommending and prescribing these tools to patients shoulder an ongoing responsibility to be familiar with the evidence and to critically evaluate the increasing breadth and volume of DHSTs.[8],[9],[10] These sentiments, as reflected by recent efforts to implement greater digital health literacy as foundational foci among their health-care workforces, are recognized by major international health-care organizations, including the World Health Organization and National Health Services of the United Kingdom.[11],[12] In general, however, formal digital health-care curricula or training opportunities for health-care students are missing educational components that may serve to benefit future health-care providers and the patients they will serve.[13] Opportunities to teach the next generation of health-care leaders how to identify, think about, and evaluate digital health technologies are imperative for the future of digitally enabled clinical care.
This paper outlines the implementation of a pilot digital health curriculum in June to August of 2021, the aim of which was to provide health-care trainees the skills needed to understand and critically evaluate DHSTs for use in clinical settings.
| MAterials and Methods | | |
In 2018, physicians at the University of California, San Francisco (UCSF) Health Informatics group created the Accelerated Digital Clinical Ecosystem (ADviCE) to ensure that new DHSTs have an informed evidence base for safe integration in clinical operations and safe and effective use in clinical practice. In May 2021, this organization began to build a registry of existing DHSTs to fill gaps in DHST knowledge for health systems.
In the summer of 2021, we implemented a 10 weeks, remote internship for 16 students with an interest in digital health, offering them firsthand experience with DHSTs through contribution to the registry. We recruited undergraduate and graduate students from across the U.S. who showed a strong interest in medicine, digital health, and clinical informatics. To promote science, technology, engineering, and mathematics (STEM) opportunities for local high school students considering careers in health care, we collaborated with UCSF's Center for Science Education and Outreach, an organization that works with San Francisco Bay Area school district partner schools to support exploration of STEM careers among high school students and increase college matriculation rates for underrepresented populations.
We developed a digital health curriculum with three key goals: (1) to give interns experience with the various types of digital health tools (e.g., telehealth, mobile applications, DHSTs involving Electronic Health Record (EHR) integrations), (2) help interns apply a framework to assess the quality of DHSTs, and (3) understand how health systems use DHSTs in real-world settings.
The design of the curriculum included didactic presentations from faculty who are experts in the field of digital health, journal club sessions, and student-led presentations about existing digital health companies and their tools. In addition, using a structured rubric, interns gained first-hand experience critically evaluating DHSTs, interacting directly with digital health-care companies, and populating their learning into the DHST registry.
The curriculum was delivered in hour-long biweekly team meetings over a period of 10 weeks. Due to the distributed location of interns across the U.S., all meetings were conducted over Zoom, and sessions were recorded for additional viewing as needed.
Curriculum component 1: Digital health speaker series
We invited experts in digital health and clinical informatics to present lectures to interns. Lecture topics included descriptions of the fields of digital health and clinical informatics, implementation barriers to DHSTs, the evolving landscapes of DHST regulatory policy and reimbursement, as well as professional pathways to digital health and clinical informatics, how faculty are using DHSTs in their current jobs, and their responsibilities in their current roles. The list of speakers included:
- Chief Nursing Informatics Officer at UCSF.
- Associate Professor of Department of Epidemiology and Biostatistics at UCSF.
- Director of Telehealth at UCSF.
- Founder/President of Tour De Heart, a non-profit organization providing digital healthcare education.
- Chief Medical Informatics Officer at Mayo Clinic.
- Author of NYTimes best-selling book on digital medicine, The Digital Doctor: Hope, Hype and Harm at the Dawn of Medicine's Computer Age.
- Medical Director of Clinical and Quality Analysis at Partners Healthcare Systems.
- Senior Medical Officer of the National Cardiovascular Data Registry.
- Informaticist and Digital Health Researcher at the Center for Clinical Informatics and Improvement Research (CLIIR).
Curriculum component 2: Journal clubs
We selected research and thought pieces that were timely, provocative, and instructive, thereby providing an opportunity for large group, moderated discussions. Interns were assigned three papers to read and discuss during biweekly team meetings:
- JAMA commentary on evaluating digital health tool studies registered in ClinicalTrials. gov.[14]
- JAMA viewpoint on the use of EHRs as a public health tool for Ebola in the U.S.[15]
- Blog post on health-care interoperability obstacles with regards to real-time scheduling in health care.[16]
Paper discussions were led by an intern volunteer and moderated by a program faculty member who was present to answer questions and supplement/prompt discussion with additional information.
Curriculum component 3: Intern presentations on a digital health software technologies
Based on their direct interactions with DHST vendors, interns were required to research, assess, and present to program faculty and co-interns, a summary of a DHST vendor and tool based on the framework emphasized in lectures and readings. This framework included questions such as: What does the company do? What product does it make? How does the product affect health? If it affects health, does it make a health claim? What is the evidence behind the health claim? How does the product work? Does it integrate with an EHR? Does it connect with consumer digital devices (e.g., patient's phone or fitness device)? Discussions with peers and faculty following each presentation focused on understanding the DHST, limitations and challenges associated with the technology, and critical assessment of marketing claims and study-based evidence.
Curriculum component 4: Building the ADviCE digital health software technologies registry
Undergraduate and graduate interns were given real-world experience with DHST vendors to learn about their organizations and tools and to supplement ADviCE DHST registry. High school students were not offered the opportunity to independently interact with DHST vendors, a decision made based on their level of formal professional experience and development. After vetting registry candidates using frameworks discussed in the curriculum, interns held meetings with potential vendors to explain the goals of the organization's registry and to present the information collection form for the registry, referred to as the “Digital Health Common Application” (DHCA). Two DHCAs were created for information gathering: One specific to the company, and another specific to the product itself (DHST). Questions included: Does your company have a relationship with the U.S. FDA? How and for what purposes do you share patient-level, user-level, or anonymous data? How do you store and encrypt data? At what stage of development is your application? Who are the target users for your tool? What is the program goal of your tool or software platform? What features does your product include? What do your data show on the effectiveness of your application, patient safety, adverse effects, and usability?
Internship evaluation
We based metrics of success of our digital health curriculum on (1) the program's ability to offer at least 10 h of educational activity to interns over 10 weeks, (2) real-world exposure of interns to DHST vendors to allow them to translate their didactics into practice, and (3) the interns' assessment of the internship measured through an anonymous end-of-course evaluation survey with Likert scale questions (1 represents “strongly disagree” and 5 “strongly agree”).
| Results | | |
Intern didactics
Out of an applicant pool of 31, 16 interns were accepted into the pilot program: 6 undergraduate students and 5 graduate students studying medical sciences, public health, or clinical informatics program, and 5 high school students with interests in health-care careers. Characteristics of the intern class are shown in [Table 1].
For the speaker series, 10 speakers with expertise in digital health or clinical informatics each gave 40 min presentations, totaling 6.67 h of expert didactics and lectures. For the journal club, we estimated that each of the 3 papers took 30 min for interns to read and 30 min to discuss in team meetings with the program faculty, totaling 3 h. For student presentations, it was estimated to have taken interns 2 h for each student to complete research, create a presentation, and deliver the presentation. By the end of the 10 weeks curriculum, the program offered 11.67 didactic hours per intern.
Intern interaction with digital health software technologies vendors
By the completion of the internship, interns had collectively met with 74 DHST vendors and were able to collect in-depth information on the company and their digital health tools for 16 vendors as outlined in “Curriculum Component 4: Building the ADviCE DHST Registry.”
Intern program assessment
Fifteen (94%) interns completed the 10 weeks program. One (6%) intern discontinued during the 5th week of the program due to competing responsibilities.
Thirteen (81%) interns completed an end-of-internship exit survey in [Table 2] with questions ranked using a 5-point Likert scale. Interns agreed or strongly agreed with the statements “The summer internship helped me understand careers in digital health” (mean score 4.38), “Readings and online resources around topics in digital health were useful” (mean 4.31), and “I would recommend the summer internship to others” (mean score 4.6). Speakers and their topics were also rated highly. | Table 2: 5-point Likert-scale responses from end-of-internship assessment and from speaker assessments.
Click here to view |
| Discussion | | |
For this pilot digital health curriculum, we reached our goal of offering a digital health education which achieved set metrics, and which interns would recommend to others. Interns viewed the speaker series more favorably than journal club readings, suggesting that future programs may benefit from greater lecture or didactic foci. Zoom meetings proved to be a successful means of creating a positive remote learning experience, allowing for geographic flexibility for the program's faculty, interns, and speakers to join lectures and didactics throughout the U.S. Further, this internship provided an opportunity for high school students from underserved backgrounds to learn about different careers in digital health and medicine and receive mentorship from the program's faculty.
Although we benefited from assessing the interns' satisfaction with various parts of the curriculum, one limitation of this curriculum is the lack of an empiric test of interns' understanding of digital health following the curriculum. In future iterations of our program, we aim to create pre-and postevaluations assessing interns' understanding of digital health prior to and after the curriculum, allowing for better assessment of whether the curriculum achieves the set learning objectives. In addition, understanding baseline knowledge of digital health among the various educational levels of the interns could allow for more tailored and focused learning objectives for each group. Finally, the proportion of males enrolled in the program was substantially lower than females in this inaugural class of interns. This is one of the many demographic factors (including race, education level, and career interest) that will require additional investigation in future iterations of the program to optimize recruitment and enrollment, and to determine subgroup differences in internship satisfaction and learning outcomes.
As our organization plans our internship for future years, we aim to improve on this foundation to offer a high-quality and accessible educational experience around digital health. We hope to create a formal measure of knowledge assessment and learning based on refined learning objectives, with the potential to offer formal course credit for students. In addition to offering a completely remote curriculum delivery to allow for geographical flexibility, we look forward to more in-person learning opportunities (e.g., institute campus visits, DHST vendors' office tours). As we build a database of valuable educational material from expert speakers and selected readings, we may also develop an opportunity to create an asynchronous, online educational program that can be completed by students who may not have the opportunity to participate in longitudinal aspects of our internship. Finally, we hope to expand our reach to bring in more students of various education levels and students of underrepresented backgrounds to ensure that digital health literacy can be available to all.
| Conclusion | | |
Our innovative internship was successful in creating a remote educational experience which promoted a broad understanding of the role of digital health in medicine, while also giving interns direct interaction with a broad array of digital health companies. Future goals include refining learning objectives for the educational program, creating a more focused assessment of whether learning objectives are met, and identifying additional opportunities to expand the reach of educational content to improve digital health literacy for future leaders of medicine.
Acknowledgment
Special thanks to program speakers Bob Wachter, David Bates, Linda Branagan, Heidi Dohse, Courtney Lyles, Kay Burke, and Steve Peters.
Disclosures
Raman Khanna reports receiving licensing income from Voalte, Inc. Voalte had no role in the conception, design, analysis, or decision-making related to this project.
Andrew Auerbach, Benjamin Rosner, and Tiffany Lee report funding from the UCSF-Stanford Center of Excellence in Regulatory Science and Innovation (2U01FD005978-06). The UCSF-Stanford Center of Excellence in Regulatory Science and Innovation had no role in the conception, design, analysis, or decision-making related to this project.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
| References | | |
| 1. | |
| 2. | Widmer RJ, Collins NM, Collins CS, West CP, Lerman LO, Lerman A. Digital health interventions for the prevention of cardiovascular disease: A systematic review and meta-analysis. Mayo Clin Proc 2015;90:469-80.  |
| 3. | Beleigoli AM, Andrade AQ, Cançado AG, Paulo MN, Diniz MD, Ribeiro AL. Web-based digital health interventions for weight loss and lifestyle habit changes in overweight and obese adults: Systematic review and meta-analysis. J Med Internet Res 2019;21:e298. |
| 4. | Fu Z, Burger H, Arjadi R, Bockting CLH. Effectiveness of digital psychological interventions for mental health problems in low-income and middle-income countries: A systematic review and meta-analysis. Lancet Psychiatry 2020;7:851-64. |
| 5. | Desveaux L, Soobiah C, Bhatia RS, Shaw J. Identifying and overcoming policy-level barriers to the implementation of digital health innovation: Qualitative study. J Med Internet Res 2019;21:e14994. |
| 6. | Whitelaw S, Pellegrini DM, Mamas MA, Cowie M, Van Spall HG. Barriers and facilitators of the uptake of digital health technology in cardiovascular care: A systematic scoping review. Eur Heart J Digit Health 2021;2:62-74. |
| 7. | Diao JA, Venkatesh KP, Raza MM, Kvedar JC. Multinational landscape of health app policy: Toward regulatory consensus on digital health. NPJ Digit Med 2022;5:61. |
| 8. | Murray E, Hekler EB, Andersson G, Collins LM, Doherty A, Hollis C, et al. Evaluating digital health interventions: Key questions and approaches. Am J Prev Med 2016;51:843-51. |
| 9. | Banerjee R, George P, Priebe C, Alper E. Medical student awareness of and interest in clinical informatics. J Am Med Inform Assoc 2015;22:e42-7. |
| 10. | Poncette AS, Glauert DL, Mosch L, Braune K, Balzer F, Back DA. Undergraduate medical competencies in digital health and curricular module development: Mixed methods study. J Med Internet Res 2020;22:e22161. |
| 11. | |
| 12. | |
| 13. | Machleid F, Kaczmarczyk R, Johann D, Balčiūnas J, Atienza-Carbonell B, von Maltzahn F, et al. Perceptions of digital health education among European medical students: Mixed methods survey. J Med Internet Res 2020;22:e19827. |
| 14. | Auerbach AD. Evaluating digital health tools-prospective, experimental, and real world. JAMA Intern Med 2019;179:840-1. |
| 15. | Mandl KD. Ebola in the United States: EHRs as a public health tool at the point of care. JAMA 2014;312:2499-500. |
| 16. | |
[Table 1], [Table 2]
|
Search Pubmed for