This dissertation investigates how metascientific approaches can enhance the quality and reliability of evidence syntheses in educational psychology. Prompted by the replication crisis, widespread questionable research practices, and the growing dependence on systematic reviews and meta-analyses in education, this work critically examines current research standards and advances innovative solutions rooted in open science.

Study I evaluates the methodological validity and reproducibility of the influential research synthesis Visible Learning by John Hattie. The study reveals several methodological flaws that contest the assumptions of the findings and the failure of being able to reproduce the statistics serves as a warning example of the presence of the replication crisis. 

Study II evaluates the risk of bias and transparency in systematic reviews conducted in educational psychology. Alarmingly, most included systematic reviews were judged as high risk of bias and across the entire sample, there was a lack of data sharing, preregistered protocols, and reproducible primary research data. 

Study III is a proof of concept of a registered report in educational psychology, the study aims to investigate the evidence of a writing intervention by conducting a systematic review. By adhering to the state-of-the-art conducting standards in systematic reviews, this protocol covers all aspects needed to produce reliable evidence as well as being reproducible. 

In Study IV, an innovative open-source Community-Augmented Meta-Analysis combined with a database is developed. The study presents solutions to several well-known problems in systematic reviews by allowing the research community to update, store, calculate, and share educational interventional data in a convenient way.

The findings of the included studies highlight significant gaps in research rigor and transparency, underscoring the necessity of fundamental change to adhere to current standards and modern research practices. 

By incorporating methodological tools such as preregistration, open science, risk of bias assessments and FAIR data principles, this dissertation calls for a paradigm shift in the synthesis and application of evidence in educational psychology. Ultimately, it seeks to promote more trustworthy, transparent, and impactful research to better inform educational policy and practice.