We conducted this meta-research study following selected principles of the scoping review methodology according to PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) [9], for example, broad inclusion criteria and descriptive synthesis. We performed a structured, quantitative content analysis on a random sample of 250 abstracts from systematic reviews and meta-analyses researching the effects of DHIs.

Table 1 provides an overview of the key aspects in each methodological step.

The search strategy was developed with the aim to include existing systematic reviews and meta-analyses which address (1) a broad range of DHI technologies and (2) a broad range of potential associated outcomes in (3) any health care setting.

A total of 33 DHI-related and 41 outcome-related search terms were used and—depending on the database searched—applied as Medical Subject Headings terms to subject headings or as plain search terms on titles and abstracts. A complete documentation of all search terms is available in Multimedia Appendix 1.

A comprehensive literature search was conducted for the time frame 2011 to October 2021 for a preceding umbrella review project, which was updated in October 2023 for more in-depth research.

To capture a large heterogeneity of care settings and DHI application scenarios, the search included all areas of health care—from inpatient hospital care through outpatient and community-based care to self-care, without any geographic boundaries.

The search was limited to systematic reviews or meta-analyses published in English or German.

We searched 5 leading databases—Scopus, AISeL, EBSCO or CINAHL, Cochrane Library, and MEDLINE via PubMed.

The search results were uploaded to Covidence (Veritas Health Innovation Ltd), a software for managing and streamlining systematic reviews. All potential studies for further analyses, such as evidence and gap mapping, were screened in pairs by UB and Jan-Oliver Kutza, and Johannes Thye and Moritz Esdar, and discrepancies were resolved through discussion or consultation with a senior reviewer (JDL or Ursula Hübner). A complete documentation of all inclusion and exclusion criteria is available in Multimedia Appendix 2.

The initial abstract review and screening process was completed in February 2024. After all abstracts were screened, 2528 eligible studies for subsequent analyses were exported to Microsoft Excel, additional duplicates were identified, checked, and omitted, and a random sample of 250 studies was drawn, using the random number assignment function in Excel, by the lead author (UB). The sample size of 250 abstracts was chosen as the basis for the detailed data extraction and quantitative analysis, to represent about 10% (250/2528) of eligible studies, and to have a 95% confidence level that the actual value is within ±6% of the measured or surveyed value when exploring potential relationships between study characteristics reported in the abstracts and the likelihood to find conclusive evidence in the full texts.

In line with our research questions, a set of 16 data extraction fields and variables was developed. Next to meta-information on study design (particularly if meta-analysis was performed and randomized controlled trial [RCT]–focus), publication year, search period in years, databases searched, the total number of studies identified by the search strategy, the number of included studies, the journal, and statements regarding evidence quality and quality assessment tools, these were the descriptions related to the PICO elements problem and population, intervention, comparator, and outcome, as well as the results and conclusions, representing the scope of evidence.

All information was exclusively identified and extracted from titles and abstracts.

This decision was based on the relevance of abstracts as the main communicative interface in the screening process for evidence synthesis and decision-making contexts. Abstracts are increasingly used in automated screening and mapping tools, making them a critical unit of analysis.

Out of 21,161 initial records, 2528 studies remained after screening. From these, a random sample of 250 was drawn for detailed analysis.

Figure 1 shows the PRISMA flowchart.

We analyzed 9 study characteristics for the discussion in the context of their influence on evidence recognition and indications of possible methodological deficits.

Table 2 provides an overview, showing the number (n) of the 250 abstracts in which a study characteristic was reported together with the relative shares (%).

The level of specification of PICO elements was analyzed in the context of the quality of current research questions in the field of DHIs. Each PICO element plays a distinct role in framing the scope and analytical precision of a review. Their individual specifications were therefore examined separately.

Table 3 and Figure 2 display the level of specification of PICO elements according to the classification scheme (Table S1 in Multimedia Appendix 3) across the sample. A key finding is that no single PICO element was highly specified in more than half of the abstracts.

Research focus on 1 medical problem (P1), such as the disease, condition, or clinical objective, was highly specified in 50.4% (126/250) of the abstracts. The intervention (I), referring to the technology used, was specified to some degree in all abstracts, with levels of specification distributed from low (58/250, 23.2%) to high (90/250, 36%). Outcomes (O) were relatively well-defined in 43.6% (109/250) of abstracts, whereas the remaining lacked a clear focus on predefined outcomes. Notably, 35.2% (88/250) of abstracts did not specify the population (P2), referring to the user group of the intervention, and 38.8% (97/250) lacked any specification for the comparison (C) element, referring to the health care setting applicable to the intervention. The population and comparison were highly specified in only 13.2% (33/250) and 19.2% (48/250) of abstracts, respectively.

We hypothesized that low PICO specification in research objectives, methods, and questions is associated with low evidence recognition. Therefore, our study aimed to examine the relationship between PICO specification and the availability of conclusive evidence described in the abstracts.

To this end, criteria for the likelihood of conclusive evidence were defined based on literature and a qualitative review and thematic analysis of all results and conclusions reported in the abstracts (Table S2 in Multimedia Appendix 3).

Table 4 provides a summary of these criteria and the results of their application to the sample.

To assess whether methodological focus correlates with result strength (our second research question), we examined the relationship between the overall level of PICO specification, based on a cumulative PICO score (minimum=0, maximum=12; 0‐3=very low, >3‐6=low, >6‐9=medium, and >9=high), and the likelihood of conclusive evidence being reported. Table 5 and Figures 3 and 4 show the results.

Among the 30 abstracts (30/250, 12%) with a high PICO score, only 4 specified all 4, 20 specified 3, and 6 specified only 2 PICO elements highly. Of those 101 with a medium PICO score, only 6 specified 3, and 46 specified 2 PICO elements highly. The 103 abstracts with a low PICO score included only 8 with 2 PICO elements and 44 with 1 PICO element being highly specified.

This means that very few systematic reviews (4/250; 1.6%) have a clear focus on 1 problem or population, 1 intervention, 1 comparative setting, and 1 outcome. On the other hand, 47.6% (119/250) demonstrate a low or very low level of PICO specification, indicating lack of research focus and a very broad scope.

Only 6 abstracts in the sample reported highly conclusive positive evidence on highly specified PICO-based research questions (6/19, 31.6% of abstracts indicating highly conclusive evidence; 6/250, 2.4% of the entire sample).

Figure 3 and Figure 4 visualize the increasing likelihood of conclusive evidence with rising PICO specification. In Figure 3, PICO specification is represented by the PICO score.

Figure 4 illustrates the distribution of the combined shares for very low and low, and medium and high PICO specification within each evidence class, highlighting that methodological clarity alone does not guarantee robust findings.

While the combined medium and high PICO specification is associated with more robust findings, about two-fifths of the abstracts that remained inconclusive or yielded a low likeliness of conclusive evidence did so regardless of the medium or high level of PICO specification.

Correlation analysis confirmed a weak to medium positive relationship with statistical significance (ρ=0.26; P<.001) between overall PICO specification (ie, PICO score) and conclusive evidence. Higher specification of the outcome of interest (PICO element O; ρ=0.27; P<.001) and of the problem (PICO element P1; ρ=0.17; P=.008) were most associated with conclusive evidence being reported. In contrast, specification of the PICO elements population (P2), intervention (I), and comparison (C) showed no or only weak relationships with conclusive evidence.

To illustrate the implications for future review practice, we have aligned the key insights of this study with the typical phases of a systematic review process, adapted from the core methods of conducting a Cochrane review [12,13].

Figure 5 highlights where critical flaws most frequently occur and also offers targeted suggestions to strengthen methodological coherence and improve evidence recognition, particularly in the context of complex interventions, such as DHIs.

In the interpretation of the above data, it should be noted that sample sizes (n) on which percentages are based vary. Refer to Tables 2, 4, and 5 and Multimedia Appendix 4 for details.

Research question: Our findings indicate inadequate application of PICO in formulating research questions, with 47.6% (119/250) of abstracts displaying low or very low PICO specification, and 63.6% (159/250) featuring poor or inconclusive evidence.

Although statistically significant, the correlation between PICO specification and conclusive evidence remains modest (ρ=0.26; P<.001). Outcome and problem specification showed the strongest associations with conclusive evidence, indicating their central role in the methodological quality of systematic reviews. Despite this apparent relationship, more than 40% of abstracts with poor (58/135) or inconclusive (10/24) evidence exhibit medium to high PICO specification.

Eligibility criteria: Currently, 36.5% (62/170) of systematic reviews rely exclusively on RCTs despite the availability of several alternatives, such as pre-post studies, nonrandomized controlled trials, cohort studies, time series, and case studies, that may also be appropriate, suggesting a need to expand the scope of study designs considered.

Search strategy: Despite more than 65 databases being available, 33.2% (67/202) of authors only used 3 or fewer databases. The number of studies retrieved and screened varies greatly, indicating significant variation in search strategy quality, particularly search periods and the formulation of search terms, operators, and search strings. Interestingly, no apparent relationship exists between PICO specification levels in the abstracts and search results. In total, 6 of the 10 lowest search results with less than 150 identified studies show poor PICO specification, while 6 of the 10 highest search results are associated with a high level of PICO specification.

Screening criteria: Despite generally large numbers of studies being identified and screened, only a small percentage (median 1.2%, IQR 0.6%‐3.1%; mean 4.2%, SD 8%) and absolute number (median 17; mean 25) is finally included, with 27.7% (65/235) of systematic reviews including 10 or less studies. In this context, it is noteworthy that 24.8% (62/250) of systematic review abstracts are not structured along background, methods, results, and conclusions, with missing details having a potential impact on the screening decision.

Critical appraisal tools: There is a host of tools available for different study designs (eg, systematic reviews, mixed methods, and primary research), research focus (eg, care practice, diagnostic accuracy, and health economics), and appraisal focus (eg, RoB and quality), of which 29 were identified in the sample. The choice of appraisal tool and the extreme variability in their criteria can significantly influence the exclusion of relevant studies, especially due to harsh appraisal ratings. Even for included studies, it is noteworthy that, if reported in the abstract, study or evidence quality is often described as low (19/42, 45%) and RoB as high (11/23, 48%). On the other hand, Blum et al [14] found no association with study quality for their results.

Evidence synthesis: Meta-analysis, the most desirable output of a systematic review to derive conclusive evidence, is featured in only 30% (75/250) of studies. Moreover, 13.2% (33/250) of abstracts include explicit comments on considerable heterogeneity or variability of study designs, populations, interventions, and outcomes, often stating that this prevented meta-analysis [15-25]. Alternative synthesis methods, such as thematic analysis [26] or statistical result aggregation [27,28], are underused. Higher-level syntheses, such as umbrella reviews of systematic reviews, are still rare (10/250, 4%).

Conclusion validity: In total, 54% (135/250) of systematic review abstracts reported neither meta-analysis nor any significant results; 9.6% (24/250) reported no evidence at all or no directional conclusion or equally contradictive effects. Although our analysis did not delve deeply into outcome effects, we stress the importance of selecting outcomes sensitive to change, as noted by Shen et al [29].

This meta-research study aimed to identify key methodological factors that limit conclusive evidence recognition in systematic reviews on DHIs. Our findings show that the overall level of PICO specification in the abstracts was low or very low in nearly half of the sample, and that two-thirds of the reviews yielded inconclusive or weak evidence. Although higher levels of PICO specification, particularly regarding outcomes and clinical problems, were moderately associated with the likelihood of conclusive findings, these factors alone were not sufficient to guarantee evidence clarity.

This observation supports our initial assumption that PICO-structured research questions provide an important methodological foundation. However, it also reveals the limitations of assuming a linear relationship between methodological structure and evidentiary strength. Indeed, about two-fifths of reviews with low or inconclusive evidence had medium to high PICO specification. This suggests that even clearly defined research questions may fail to translate into robust evidence if other aspects of the review process are flawed.

Among these aspects, we identified recurring issues in search strategy design (eg, limited database use and vague search terms), restrictive eligibility criteria (eg, exclusive reliance on RCTs), and inconsistent use of quality appraisal tools. These elements interact with question formulation and may explain why even well-specified PICO frameworks can fail to yield strong evidence. Notably, substantial heterogeneity in study designs and outcomes further impedes evidence synthesis, particularly when meta-analyses are not feasible or misapplied.

Our findings underscore the need for a more integrated methodological approach. Rather than focusing solely on formal structures, review authors should align research questions, search strategies, eligibility and screening criteria, critical appraisal, evidence synthesis methods, and conclusions to the specific characteristics of digital health technologies. This is particularly relevant for DHIs, which often involve complex, context-sensitive mechanisms of action that challenge conventional evaluation models.

Research question: While high PICO specification does not guarantee conclusive results, a well-defined scientific starting point is critical [30]. In formulating the research question, both too specific and too broad specification of PICO elements appear more likely to lead to inconclusive results. We conclude, at a medium level of specification, that is, clearly defined and differentiated categories, in at least 2 or 3 PICO elements, would be optimal for many systematic reviews.

Eligibility criteria: Broad inclusion criteria, represented by low PICO specification, relate to substantial heterogeneity [30]. Conversely, narrow inclusion criteria, such as focusing solely on RCTs, also add challenges. Despite RCTs being the gold standard for intervention evaluations [31], they might not be fully appropriate in evaluating DHIs, due to long time frames, high costs, rigid protocols, and DHI specificities, such as individual tailoring and interaction effects that RCTs can insufficiently address [32-34].

Search strategy: A comprehensive and sensitive search strategy in multiple databases is recommended [30]. Our findings highlight the necessity for more methodological rigor in comprehensive search strategies to mitigate the inconsistent retrieval of relevant studies. This concerns particularly searches across multiple databases and search terms and strings that are more aligned with the research questions. Exponential growth in the number of publications and increasing quality of studies over time leads us to suggest restricting search periods for future systematic reviews to a maximum of 10 years.

Screening criteria: Our findings suggest that screening criteria might not effectively capture all relevant research to ensure comprehensive evidence synthesis, necessitating a reassessment in the screening process. Given the limitations observed with broad and narrow eligibility criteria, more consideration of inclusion and exclusion criteria, especially a focus on RCTs, seems to be required to balance comprehensiveness and specificity.

While the abstract structure and length may reflect specific journal guidelines, evolving reporting guidance should ensure that all standard structural components of a systematic review abstract (ie, background, methods, results, and conclusions) are included.

Critical appraisal tools: With 121 different tools published, “there is no ‘gold standard’ for any study design, nor is there any widely accepted generic tool that can be applied equally well across study types” [35]. This issue emphasizes the importance of selecting the most appropriate available tools and the need for more standardized, pragmatic tools that align with the objectives of systematic reviews on DHI effects.

The appraisal process may demand an assessment or validation and consolidation or adaptation of available tools to ensure it is capturing pertinent research without unnecessarily excluding valuable studies due to overly stringent or irrelevant criteria of appraisal tools.

Evidence synthesis: Incorporating alternative synthesis methods beyond meta-analysis is recommended. Meta-analysis is often challenged by substantial heterogeneity and variability in research questions, study designs, interventions, and outcomes, and incomparable data from the primary research studies. Recognizing the challenges of meta-analyses, systematic reviews should more often explore thematic analyses and other quantitative aggregations to ensure a comprehensive evidence base. Purely narrative approaches or summary tables of primary studies lack the quantitative aggregation needed for conclusive reviews.

Conclusion validity: Researchers should strive for outcome measures that are sensitive to change and aim to capture both relevant proximal and distal intervention effects to ensure meaningful and legitimate conclusions. In interpreting findings, conclusions that suggest limited evidence of benefits due to outcomes comparable with standard or conventional care may neglect the significance of parity when additional benefits, such as cost reduction, improved timeliness, and enhanced access to care, are considered. This is particularly pertinent to telemedicine, telerehabilitation, and telehealth.

This study has several limitations that should be considered when interpreting the findings.

First, all data were extracted exclusively from abstracts. We acknowledge that abstracts may not fully capture the methodological details of the full texts; our analysis, therefore, reflects reported, not necessarily executed, methodological rigor. However, abstracts are the primary entry point for study screening and selection, making them critical for evidence recognition.

Second, the analysis was descriptive and exploratory in nature. No causal claims can be made, and the moderate correlations observed should be interpreted as indicative rather than predictive.

Third, data extraction and analysis were based on a random sample rather than a full systematic review of all eligible studies. Thus, while the sample was randomly drawn and diverse in terms of publication outlets and topics, it may not fully represent the entire body of systematic reviews on DHIs.

Finally, although a structured coding scheme was applied, some degree of subjective judgment was inevitable in the classification of PICO elements and evidence conclusiveness.

Future studies should complement abstract-level analyses with full-text reviews, interrater reliability testing, and more refined models to account for complex interactions between review components.

This sample-based meta-research study provides empirical insight into why systematic reviews on DHIs often fail to yield conclusive evidence.

The findings underscore the crucial role of optimally specifying the PICO elements of interest in DHI research, revealing that inadequate PICO specification correlates somewhat with suboptimal evidence recognition.

However, while clearer specification—especially of outcomes and clinical problems—was moderately associated with stronger evidence, this alone did not guarantee evidentiary clarity.

Our findings suggest that methodological coherence across all review stages is a necessary condition to ensure conclusions and evidence-informed decisions in the digitalization of healthcare are both valid and meaningful.

In summary, beyond question formulation, issues in search strategy, eligibility criteria, and synthesis design also play a critical role in shaping review outcomes. To support the underlying processes, we propose a structured, PICO-based categorization framework that is aligned with this research, builds on well-established categories, and provides clear and differentiated definitions and associated terms. Such a framework could inform the research question, and the search strategy and screening process by providing terms for search strings and inclusion or exclusion criteria. Our proposed framework could ultimately enable comprehensive and continuous evidence and gap mapping for DHIs, especially if the increasing power of artificial intelligence tools is leveraged in the research process. Thus, it may help to improve the quality and transparency of systematic reviews—especially in the complex and dynamic field of digital health.