Evaluate, but how? – Criteria for evaluating and reporting studies on health and medical apps
Sven Kernebeck MSc PH / Dr. rer. nat. Ursula Kramer / Dr. med. Marcus Redaelli / Univ.-Prof. dr med. Horst Christian Vollmar MPH
With digitalization in the healthcare system, great expectations are tied to increasing the quality of healthcare (Mathews et al., 2019), this particularly applies to diagnostics, prevention and the management of chronic diseases (Gordon, Landman, Zhang & Bates, 2020). Health and medicine apps (GuMA) are given a special role due to the ubiquitous availability of smartphones among the population. However, there has been a broad discussion for some time regarding the still limited evidence of GuMA, which is assessed as heterogeneous in various systematic reviews (Agarwal, Lefevre & Labrique, 2017; Felix et al., 2018; Mathews et al., 2019). In a systematic review by Bayambasuren et al. the authors searched for systematic reviews of randomized controlled trials (RCTs) with prescribable GuMAs that were freely available for download (Byambasuren, Sanders, Beller & Glasziou, 2018). The authors define prescriptive apps as apps that are currently available, that are effective and that are used on their own (stand-alone) without the involvement of healthcare professionals. The authors included a total of six systematic reviews in their analysis, from which they in turn extracted 23 RCTs on 22 different GuMAs. Only 11 of these 23 RCTs demonstrated a meaningful effect on health or on selected surrogate endpoints that could possibly be attributed to GuMA. Overall, however, the evidence of effectiveness was of limited quality, which makes it much more difficult for doctors to prescribe these apps (Mathews et al., 2019). The authors also criticized a high risk of bias in both the six included systematic reviews and the 23 extracted RCTs, which also contributes to a limitation of the results. Likewise, positive results in terms of effectiveness can be found in various areas of application. This can be stated, for example, for the self-management of hypertension (Li, Liang, Bu & Hesketh, 2020) or the self-management of mental illnesses (Linardon, Cuijpers, Carlbring, Messer & Fuller-Tyszkiewicz, 2019).
Read more
First publication date: 30.03.2021
Downloads
Abstract: Bewerten, aber wie? – Kriterien für die Evaluation und das Reporting von Studien zu Gesundheits- und Medizin-Apps
Background: Similar to the low quality-assured guidelines at the end of the 1990s, the German healthcare system is once again experiencing an inflation of low-quality-assured publications from digital applications against the background of digitalization. Against this background, weak evidence from health and medical apps is criticized in many places. In order to be able to assess the quality of studies and the transparency in reporting studies as service providers and users, uniform criteria are required, especially for doctors and scientists. Methodology: In September 2019 and additionally in February 2020, a literature search was carried out in PubMed and Google Scholar carried out. We were looking for frameworks and reporting guidelines that can be used in evaluations of health and medical apps. As comparison criteria, the framework, phases/items, methodology and focus were compared and evaluated in an exploratory manner. Result: Seven publications on frameworks and reporting guidelines were included. These differ in the intended focus and the scope and content of the evaluation criteria. The frameworks and reporting guidelines address studies conducted with health and medical apps, such as: B. the Mobile Health Evidence Reporting and Assessment (mERA) and the so-called DiGA guidelines for the fast-track procedure for digital health applications. In addition, method-related reporting guidelines should be highlighted, which primarily place a methodological focus on the application, for example in randomized controlled studies. Conclusion: Given the options for standards in the reporting of digital applications presented in this work, the guidelines presented by the BfArM could include additional items be expanded to give service providers and users more security and at the same time prevent compromises in quality standards.
Keywords (English)
Literature
Abbey, E., Maiken, H., Camilla, V., Lyngsie, H., Lisa, Utoft, N., Carsten, Kirsten, B., & Helle, P. (2018). Co-created Smartphone App to Improve the Quality of Life of Adolescents and Young Adults with Cancer (Kræftværket): Protocol for a Quantitative and Qualitative Evaluation. JMIR Res Protoc, 7(5), e10098. doi: 10.2196/10098Adu, MD, Malabu, UH, Callander, EJ, Malau-Aduli, AE, & Malau-Aduli, BS (2018). Considerations for the Development of Mobile Phone Apps to Support Diabetes Self-Management: Systematic Review. JMIR MHealth and UHealth, 6(6), e10115. doi: 10.2196/10115Agarwal, S., Lefevre, AE, & Labrique, AB (2017). A Call to Digital Health Practitioners: New Guidelines Can Help Improve the Quality of Digital Health Evidence. JMIR MHealth and UHealth, 5(10), e136. doi: 10.2196/mhealth.6640Agarwal, S., LeFevre, AE, Lee, J., L'Engle, K., Mehl, G., Sinha, C., … Group, W. mHealth TER (2016). Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ, 352, i1174. doi: 10.1136/bmj.i1174Ahmadiankalati, M., Steins-Loeber, S., & Paslakis, G. (2020). Review of Randomized Controlled Trials Using e-Health Interventions for Patients With Eating Disorders. Frontiers in Psychiatry, 11, 568. doi: 10.3389/fpsyt.2020.00568Allner, R., Wilfling, D., Kidholm, K., & Steinhäuser, J. (2019). Telemedicine projects in rural areas of Germany. A systematic evaluation using the “Model for the Evaluation of Telemedical Applications”. Journal of evidence, training and quality in healthcare, 141–142, 89–95. doi: 10.1016/j.zefq.2019.03.005Ann, M., Sarah, N, R., Thomas, C, K., Abby, D, G., Christopher, & Sutton,Stephen. (2016). IDEAS (Integrate, Design, Assess, and Share): A Framework and Toolkit of Strategies for the Development of More Effective Digital Interventions to Change Health Behavior. J Med Internet Res, 18(12), e317. doi: 10.2196/jmir.5927BfArM (2020a) Federal Institute for Drugs and Medical Devices. Digital Health Applications. In https://www.bfarm.de/DE/Medizinprodukte/DVG/_node.html (accessed on May 20, 2020) BfArM. (2020b). The fast-track procedure for digital health applications (DiGA) according to § 139e SGB V. A guide for manufacturers, service providers and users. In https://www.bfarm.de/DE/Medizinprodukte/DVG/_node.html (accessed on May 20, 2020) Byambasuren, O., Sanders, S., Beller, E., & Glasziou, P. (2018) . Prescribable mHealth apps identified from an overview of systematic reviews. Npj Digital Medicine, 1(1), 12. doi: 10.1038/s41746-018-0021-9Collins, L. (2018). Optimization of Behavioral, Biobehavioral, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST). Springer International Publishing.Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2012). Developing and evaluating complex interventions: the new Medical Research Council guidance. International Journal of Nursing Studies, 50(5), 587-592. doi: 10.1016/j.ijnurstu.2012.09.010EbM network. (2019). Statement of the German Network Evidence-Based Medicine on the draft of a law for better care through digitization and innovation (Digital Care Act - DVG) of May 15, 2019. Ekelend, A., Bowes, A., & Signe, F. (2010). Effectiveness of telemedicine: A systematic review of reviews. International Journal of Medical Informatics, 79, 736-771. doi: https://doi.org/10.1016/j.ijmedinf.2010.08.006Enam, A., Torres-Bonilla, J., & Eriksson, H. (2018). Evidence-Based Evaluation of eHealth Interventions: Systematic Literature Review. Journal of Medical Internet Research, 20(11), e10971. doi: 10.2196/10971Eysenbach, G., & Group, C.-E. (2011). CONSORT-EHEALTH: Improving and Standardizing Evaluation Reports of Web-based and Mobile Health Interventions. Journal of Medical Internet Research, 13(4), e126. doi:10.2196/jmir.1923Felix G, Indra J, Mark C, Samantha R, Neelam P, & John P (2018). What is an appropriate level of evidence for a digital health intervention? The Lancet, 392(10165), 2665-2667. doi: 10.1016/s0140-6736(18)33129-5 Gordon WJ, Landman A, Zhang H, & Bates DW (2020). Beyond validation: getting health apps into clinical practice. Npj Digital Medicine, 3(1), 14. doi: 10.1038/s41746-019-0212-zHennessy M, Heary C, Laws R, Rhoon L, Toomey E, Wolstenholme H , & Byrne, M. (2019). The effectiveness of health professional-delivered interventions during the first 1000 days to prevent overweight/obesity in children: A systematic review. Obesity Reviews, 20(12), 1691-1707. doi: 10.1111/obr.12924Hoffmann, TC, Glasziou, PP, Boutron, I., Milne, R., Perera, R., Moher, D., … Michie, S. (2014). Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ, 348(mar07 3), g1687-g1687. doi: 10.1136/bmj.g1687Kidholm, K., Clemensen, J., Caffery, LJ, & Smith, AC (2017). The Model for Assessment of Telemedicine (MAST): A scoping review of empirical studies. Journal of Telemedicine and Telecare, 23(9), 803-813. doi: 10.1177/1357633x17721815Kidholm, K., Ekeland, AG, Jensen, LK, Rasmussen, J., Pedersen, CD, Bowes, A., ... Bech, M. (2012). A MODEL FOR ASSESSMENT OF TELEMEDICINE APPLICATIONS: MAST. International Journal of Technology Assessment in Health Care, 28(1), 44–51. doi: 10.1017/s0266462311000638Kowatsch, T., Otto, L., Harperink, S., Cotti, A., & Schlieter, H. (2020). A design and evaluation framework for digital health interventions. It - Information Technology, 61(5–6), 253–263. doi: 10.1515/itit-2019-0019Kristian, K., K., J., Lise, Tue, K., Emilie, N., & B., H., Mette. (2018). Validity of the Model for Assessment of Telemedicine: A Delphi study. J Telemed Telecare, 24(2), 118-125. doi: 10.1177/1357633x16686553Krüger-Brand, & E., H. (2019). Digital Supply Act: Further refinement in detail. Dtsch Ärztebl.Li, R., Liang, N., Bu, F., & Hesketh, T. (2020). The Effectiveness of Self-Management of Hypertension in Adults Using Mobile Health: Systematic Review and Meta-Analysis. JMIR MHealth and UHealth, 8(3), e17776. doi: 10.2196/17776Linardon, J., Cuijpers, P., Carlbring, P., Messer, M., & Fuller‐Tyszkiewicz, M. (2019). The efficacy of app-supported smartphone interventions for mental health problems: a meta-analysis of randomized controlled trials. World Psychiatry, 18(3), 325-336. doi: 10.1002/wps.20673Mathews, SC, McShea, MJ, Hanley, CL, Ravitz, A., Labrique, AB, & Cohen, AB (2019). Digital health: a path to validation. NPJ Digital Medicine, 2(1), 38. doi: 10.1038/s41746-019-0111-3Moher, D., Schulz, KF, Simera, I., & Altman, DG (2010). Guidance for developers of health research reporting guidelines. PLoS Medicine, 7(2), e1000217. doi: 10.1371/journal.pmed.1000217NICE. (2019). Evidence standards framework for digital health technologies. In: https://www.nice.org.uk/about/what-we-do/our-programmes/evidence-standards-framework-for-digital-health-technologies (accessed on December 10, 2020) Sauerland, p. (2019). Digital health interventions - when is evidence needed? KVH-Journal, 4/2019. Thurnheer, SE, Gravestock, I., Pichierri, G., Steurer, J., & Burgstaller, JM (2018). Benefits of Mobile Apps in Pain Management: Systematic Review. JMIR MHealth and UHealth, 6(10), e11231. doi: 10.2196/11231Tønning, ML, Kessing, LV, Bardram, JE, & Faurholt-Jepsen, M. (2019). Methodological Challenges in Randomized Controlled Trials on Smartphone-Based Treatment in Psychiatry: Systematic Review. Journal of Medical Internet Research, 21(10), e15362. doi: 10.2196/15362Vilardaga, R., Casellas-Pujol, E., McClernon, JF, & Garrison, KA (2019). Mobile Applications for the Treatment of Tobacco Use and Dependence. Current Addiction Reports, 6(2), 86-97. doi: 10.1007/s40429-019-00248-0Wang, Y., Min, J., Khuri, J., Xue, H., Xie, B., Kaminsky, LA, & Cheskin, LJ (2020). Effectiveness of Mobile Health Interventions on Diabetes and Obesity Treatment and Management: Systematic Review of Systematic Reviews. JMIR MHealth and UHealth, 8(4), e15400. doi: 10.2196/15400WHO. (2020). mHealth: new horizons for health through mobile technologies: second global survey on eHealth. Global observatory for eHealth Series, 3.
Additional Information
- Issue: Issue 02/2021 Publication Date: 30.03.2021 All issues as PDF Subscribe to print magazine Order single issue Open Access