Background The aim of the INSPECT-SR project is to develop a tool to identify problematic RCTs in systematic reviews. In Stage 1 of the project, a list of potential trustworthiness checks was created. The checks on this list must be evaluated to determine which should be included in the INSPECT-SR tool. Methods We attempted to apply 72 trustworthiness checks to RCTs in 50 Cochrane Reviews. For each, we recorded whether the check was passed, failed or possibly failed, or whether it was not feasible to complete the check. Following application of the checks, we recorded whether we had concerns about the authenticity of each RCT. We repeated each meta-analysis after removing RCTs flagged by each check, and again after removing RCTs where we had concerns about authenticity, to estimate the impact of trustworthiness assessment. Trustworthiness assessments were compared to Risk of Bias and GRADE assessments in the reviews. Results 95 RCTs were assessed. Following application of the checks, assessors had some or serious concerns about the authenticity of 25% and 6% of the RCTs, respectively. Removing RCTs with either some or serious concerns resulted in 22% of meta-analyses having no remaining RCTs. However, many checks proved difficult to understand or implement, which may have led to unwarranted scepticism in some instances. Furthermore, we restricted assessment to meta-analyses with no more than 5 RCTs (54% contained only 1 RCT), which will distort the impact on results. No relationship was identified between trustworthiness assessment and Risk of Bias or GRADE. Conclusions This study supports the case for routine trustworthiness assessment in systematic reviews, as problematic studies do not appear to be flagged by Risk of Bias assessment. The study produced evidence on the feasibility and impact of trustworthiness checks. These results will be used, in conjunction with those from a subsequent Delphi process, to determine which checks should be included in the INSPECT-SR tool.

This paper provides guidance and tools for conducting open and reproducible systematic reviews in psychology. It emphasizes the importance of systematic reviews for evidence-based decision-making and the growing adoption of open science practices. Open science enhances transparency, reproducibility, and minimizes bias in systematic reviews by sharing data, materials, and code. It also fosters collaborations and enables involvement of non-academic stakeholders. The paper is designed for beginners, offering accessible guidance to navigate the many standards and resources that may not obviously align with specific areas of psychology. It covers systematic review conduct standards, pre-registration, registered reports, reporting standards, and open data, materials and code. The paper is concluded with a glimpse of recent innovations like Community Augmented Meta-Analysis and independent reproducibility checks.

This systematic living meta-review investigated presence of open science practices such as FAIR data adherence (findability, accessibility, interoperability, and reusability), preregistration of protocols and reproducibility (e.g., searches) in educational systematic reviews through the years 2019-2023. Additionally, we assessed systematic reviews for adherence to methodological standards (Methodological Expectations of Campbell Collaboration Intervention Reviews (Wang et al., 2021), using the tool ROBIS—Risk of Bias in Systematic Reviews (Whiting et al., 2016). 

Results show that very few systematic reviews are assessed as low risk of bias, preregister a protocol, are possible to reproduce (e.g., searches and where exactly primary data is collected) and very few organize their data according to FAIR, although there are some reviews that adhere to the (much needed) rigorous standards and share data that can be reused. This poster will address these issues on how the community can improve reproducibility practices.

This systematic living meta-review investigated the presence of open science practices (e.g., preregistration of protocols), reproducibility (e.g., describing searches and where exactly primary data is collected), and FAIR data adherence (findability, accessibility, interoperability, and reusability) in educational systematic reviews through the years 2019-2023. Additionally, we assessed the systematic reviews for adherence to MECCIR standards (Methodological Expectations of Campbell Collaboration Intervention Reviews; Wang et al., 2021), using the tool ROBIS (Risk of Bias in Systematic Reviews; Whiting et al., 2016).

Results show that very few systematic reviews could be assessed as low risk of bias, had preregistered a protocol, were possible to reproduce, and very few organized their data according to FAIR. There were however some reviews that adhered to the (much needed) rigorous standards and shared data that can be reused. This poster will address these issues further and discuss how the community can improve reproducibility practices.