TY - JOUR AB - BACKGROUND: Large language models (LLMs) hold promise for supporting clinical tasks, particularly in data-driven and technical disciplines such as radiation oncology. While prior evaluation studies have focused on examination-style settings for evaluating LLMs, their performance in real-life clinical scenarios remains unclear. In the future, LLMs might be used as general AI assistants to answer questions arising in clinical practice. It is unclear how well a modern LLM, locally executed within the infrastructure of a hospital, would answer such questions compared with clinical experts. OBJECTIVE: This study aimed to assess the performance of a locally deployed, state-of-the-art medical LLM in answering real-world clinical questions in radiation oncology compared with clinical experts. The aim was to evaluate the overall quality of answers, as well as the potential harmfulness of the answers if used for clinical decision-making. METHODS: Physicians from 10 departments of European hospitals collected questions arising in the clinical practice of radiation oncology. Fifty of these questions were answered by 3 senior radiation oncology experts with at least 10 years of work experience, as well as the LLM Llama3-OpenBioLLM-70B (Ankit Pal and Malaikannan Sankarasubbu). In a blinded review, physicians rated the overall answer quality on a 5-point Likert scale (quality), assessed whether an answer might be potentially harmful if used for clinical decision-making (harmfulness), and determined if responses were from an expert or the LLM (recognizability). Comparisons between clinical experts and LLMs were then made for quality, harmfulness, and recognizability. RESULTS: There were no significant differences between the quality of the answers between LLM and clinical experts (mean scores of 3.38 vs 3.63; median 4.00, IQR 3.00-4.00 vs median 3.67, IQR 3.33-4.00; P=.26; Wilcoxon signed rank test). The answers were deemed potentially harmful in 13% of cases for the clinical experts compared with 16% of cases for the LLM (P=.63; Fisher exact test). Physicians correctly identified whether an answer was given by a clinical expert or an LLM in 78% and 72% of cases, respectively. CONCLUSIONS: A state-of-the-art medical LLM can answer real-life questions from the clinical practice of radiation oncology similarly well as clinical experts regarding overall quality and potential harmfulness. Such LLMs can already be deployed within the local hospital environment at an affordable cost. While LLMs may not yet be ready for clinical implementation as general AI assistants, the technology continues to improve at a rapid pace. Evaluation studies based on real-life situations are important to better understand the weaknesses and limitations of LLMs in clinical practice. Such studies are also crucial to define when the technology is ready for clinical implementation. Furthermore, education for health care professionals on generative AI is needed to ensure responsible clinical implementation of this transforming technology. AU - Dennstädt, F.* AU - Schmerder, M.* AU - Riggenbach, E.* AU - Mose, L.* AU - Bryjova, K.* AU - Bachmann, N.* AU - Mackeprang, P.H.* AU - Ahmadsei, M.* AU - Sinovcic, D.* AU - Windisch, P.* AU - Zwahlen, D.R.* AU - Rogers, S.* AU - Riesterer, O.* AU - Maffei, M.* AU - Gkika, E.* AU - Haddad, H.* AU - Peeken, J.C. AU - Putora, P.M.* AU - Glatzer, M.* AU - Putz, F.* AU - Hoefler, D.* AU - Christ, S.M.* AU - Filchenko, I.* AU - Hastings, J.* AU - Gaio, R.* AU - Chiang, L.* AU - Aebersold, D.M.* AU - Cihoric, N.* C1 - 75609 C2 - 58246 TI - Comparative evaluation of a medical large language model in answering real-world radiation oncology questions: Multicenter observational study. JO - J. Med. Internet Res. VL - 27 PY - 2025 SN - 1439-4456 ER - TY - JOUR AB - BACKGROUND: It is necessary to harmonize and standardize data variables used in case report forms (CRFs) of clinical studies to facilitate the merging and sharing of the collected patient data across several clinical studies. This is particularly true for clinical studies that focus on infectious diseases. Public health may be highly dependent on the findings of such studies. Hence, there is an elevated urgency to generate meaningful, reliable insights, ideally based on a high sample number and quality data. The implementation of core data elements and the incorporation of interoperability standards can facilitate the creation of harmonized clinical data sets. OBJECTIVE: This study's objective was to compare, harmonize, and standardize variables focused on diagnostic tests used as part of CRFs in 6 international clinical studies of infectious diseases in order to, ultimately, then make available the panstudy common data elements (CDEs) for ongoing and future studies to foster interoperability and comparability of collected data across trials. METHODS: We reviewed and compared the metadata that comprised the CRFs used for data collection in and across all 6 infectious disease studies under consideration in order to identify CDEs. We examined the availability of international semantic standard codes within the Systemized Nomenclature of Medicine - Clinical Terms, the National Cancer Institute Thesaurus, and the Logical Observation Identifiers Names and Codes system for the unambiguous representation of diagnostic testing information that makes up the CDEs. We then proposed 2 data models that incorporate semantic and syntactic standards for the identified CDEs. RESULTS: Of 216 variables that were considered in the scope of the analysis, we identified 11 CDEs to describe diagnostic tests (in particular, serology and sequencing) for infectious diseases: viral lineage/clade; test date, type, performer, and manufacturer; target gene; quantitative and qualitative results; and specimen identifier, type, and collection date. CONCLUSIONS: The identification of CDEs for infectious diseases is the first step in facilitating the exchange and possible merging of a subset of data across clinical studies (and with that, large research projects) for possible shared analysis to increase the power of findings. The path to harmonization and standardization of clinical study data in the interest of interoperability can be paved in 2 ways. First, a map to standard terminologies ensures that each data element's (variable's) definition is unambiguous and that it has a single, unique interpretation across studies. Second, the exchange of these data is assisted by "wrapping" them in a standard exchange format, such as Fast Health care Interoperability Resources or the Clinical Data Interchange Standards Consortium's Clinical Data Acquisition Standards Harmonization Model. AU - Stellmach, C.* AU - Hopff, S.M.* AU - Jaenisch, T.* AU - Nunes de Miranda, S.M.* AU - Rinaldi, E.* AU - NAPKON Working Group (Kraus, M. AU - Anton, G.) C1 - 72036 C2 - 56351 TI - Creation of standardized common data elements for diagnostic tests in infectious disease studies: Semantic and syntactic mapping. JO - J. Med. Internet Res. VL - 26 PY - 2024 SN - 1439-4456 ER - TY - JOUR AB - Background: Telemedicine is defined by three characteristics: (1) using information and communication technologies, (2) covering a geographical distance, and (3) involving professionals who deliver care directly to a patient or a group of patients. It is said to improve chronic care management and self-management in patients with chronic diseases. However, currently available guidelines for the care of patients with diabetes, hypertension, or dyslipidemia do not include evidence-based guidance on which components of telemedicine are most effective for which patient populations.Objective: The primary aim of this study was to identify, synthesize, and critically appraise evidence on the effectiveness of telemedicine solutions and their components on clinical outcomes in patients with diabetes, hypertension, or dyslipidemia.Methods: We conducted an umbrella review of high-level evidence, including systematic reviews and meta-analyses of randomized controlled trials. On the basis of predefined eligibility criteria, extensive automated and manual searches of the databases PubMed, EMBASE, and Cochrane Library were conducted. Two authors independently screened the studies, extracted data, and carried out the quality assessments. Extracted data were presented according to intervention components and patient characteristics using defined thresholds of clinical relevance. Overall certainty of outcomes was assessed using the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) tool.Results: Overall, 3564 references were identified, of which 46 records were included after applying eligibility criteria. The majority of included studies were published after 2015. Significant and clinically relevant reduction rates for glycated hemoglobin (HbA(1c);<=-0.5%) were found in patients with diabetes. Higher reduction rates were found for recently diagnosed patients and those with higher baseline HbA(1c) (>8%). Telemedicine was not found to have a significant and clinically meaningful impact on blood pressure. Only reviews or meta-analyses reporting lipid outcomes in patients with diabetes were found. GRADE assessment revealed that the overall quality of the evidence was low to very low.Conclusions: The results of this umbrella review indicate that telemedicine has the potential to improve clinical outcomes in patients with diabetes. Although subgroup-specific effectiveness rates favoring certain intervention and population characteristics were found, the low GRADE ratings indicate that evidence can be considered as limited. Future updates of clinical care and practice guidelines should carefully assess the methodological quality of studies and the overall certainty of subgroup-specific outcomes before recommending telemedicine interventions for certain patient populations. AU - Timpel, P.* AU - Oswald, S.* AU - Schwarz, P.E. AU - Harst, L.* C1 - 58620 C2 - 48362 CY - 130 Queens Quay E, Ste 1102, Toronto, On M5a 0p6, Canada TI - Mapping the evidence on the effectiveness of telemedicine interventions in diabetes, dyslipidemia, and hypertension: An umbrella review of systematic reviews and meta-analyses. JO - J. Med. Internet Res. VL - 22 IS - 3 PB - Jmir Publications, Inc PY - 2020 SN - 1439-4456 ER - TY - JOUR AB - BACKGROUND: Mobile apps are an evolving trend in the medical field. To date, few apps in an oncological context exist. OBJECTIVE: The aim was to analyze the attitude of health care professionals (HCPs) toward telemedicine, mHealth, and mobile apps in the field of oncology. METHODS: We developed and conducted an online survey with 24 questions evaluating HCPs' general attitude toward telemedicine and patients using medical mobile apps. Specific questions on the possible functionality for patients and the resulting advantages and disadvantages for both the patients' and HCPs' daily clinical routine were evaluated. RESULTS: A total of 108 HCPs completed the survey. In all, 88.9% (96/108) considered telemedicine useful and 84.3% (91/108) supported the idea of an oncological app complementing classical treatment. Automatic reminders, timetables, and assessment of side effects and quality of life during therapy were rated as the most important functions. In contrast, uncertainty regarding medical responsibility and data privacy were reasons mostly named by critics. Most (64.8%, 70/108) were in favor of an alert function due to data input needing further clarification, and 94% (66/70) were willing to contact the patient after a critical alert. In all, 93.5% (101/108) supported the idea of using the collected data for scientific research. Moreover, 75.0% (81/108) believed establishing a mobile app could be beneficial for the providing hospital. CONCLUSIONS: A majority of HCPs are in favor of telemedicine and the use of oncological apps by patients. Assessing side effects can lead to quicker response and thus lower inconvenience for patients. Clinical data, such as life quality and treatment satisfaction, could be used to evaluate and improve the therapy workflow. Eventually, a mobile app would enhance the patients' relationship to their treating department because they are in permanent contact.   AU - Kessel, K.A. AU - Vogel, M.M. AU - Schmidt-Graf, F.* AU - Combs, S.E. C1 - 50370 C2 - 42177 TI - Mobile Apps in Oncology: A survey on health care professionals' attitude toward telemedicine, mHealth, and oncological apps. JO - J. Med. Internet Res. VL - 18 IS - 11 PY - 2016 SN - 1439-4456 ER -