We have conducted an implementation science project exploring the systematic effect of equipping primary school teachers to incorporate digital learning materials (DLM) into classroom activities. As part of the implementation science framework used in the study–Active Implementation Framework–we drew upon a mixed-methods approach collecting data from a series of interviews, observations, surveys, log-books, and analysis of the data collected from the DLM.

This study explores 34 teachers’ needs, first, and understanding, then, of the integration of DLM into teaching and learning activities developed over time and during the four stages of the implementation project. Teachers participated in a series of workshops based on their expressed needs and then systematically used the DLM, integrating them, during the workshops, into classroom activities. This research is solely based on the explorative thematic analysis of the interview data collected at three intervals, before, during and towards the end of the implementation. All interviews were conducted with Zoom and recorded. We seek to answer the research question: What expectations about digital learning material in relation to teachers’ teaching emerge from the interview data, and how do they differ from a temporal perspective?

The researchers have used ChatGPT-4o to analyse the interview data for two reasons. First, to understand if ChatGPT-4o does a sufficient and maybe even surprising job at capturing the intricacy of practitioners’ concepts about the scope of the research. Second, to check for the accuracy of the analysis by listening to the interviews and going back to the transcribed raw data at random times to build a human-machine trust for future research purposes. It is crucial to pinpoint that the researchers have transcribed all the interviews by Whisper, and then listened to all the interviews while decoding each speaker.

The preliminary results indicate that:1) Teachers anticipated digital tools would enhance student engagement but recognized the need to adjust their teaching methods to integrate these technologies effectively.2) Teachers encountered difficulties such as technical problems, inadequate training, and challenges in keeping students focused and engaged with digital platforms.3) Variations in students' digital literacy affected their ability to engage with digital materials, with some excelling while others struggled, leading to uneven learning experiences.4) The temporal evolution of teacher perspectives on digital learning shows that while initially hesitant, many teachers' attitudes toward digital tools shifted over time as they adapted to and, in some cases, came to value the flexibility these tools offered. Finally, the analysis conducted through ChatGPT-4o was checked for accuracy by randomly choosing 10 events for which the researchers asked ChatGPT-4o to point to specific codes from the interviews and the specific speaker. This resulted in an accurate analysis, which was also supported by the researcher who had listened to all the interviews.

This study demonstrates the positive effects of systematic professional development interventions, the core of our project. It could indicate the impact of professional development on teachers' competence and confidence in using digital tools. But also that ChatGPT-4o can be used for conducting a thematic analysis, especially when the researcher is faced with large interview raw datasets.

Background Clinical education in Emergency services (EMS) is unique due to its dynamic environment, brief patient encounters, and unpredictable cases. EMS provides valuable learning opportunities for nursing students, fostering person-centered care approaches and a variation of clinical training and learning. Formative feedback is crucial todevelop knowledge and skills. Multisource feedback (MSF) offers a comprehensive assessment by incorporating feedback from various individuals, promoting self-reflection and targeted learning. MSF has not, to our knowledge, been systematically evaluated in the context of EMS, and therefore, the aim of the study was to describe nursing students’ experiences with MSF during their clinical education in the EMS, using a digital instrument as a facilitating tool.

Methods A qualitative design with an inductive approach was used. Data were collected in 2021, using focus group interviews (n = 4) with 31 final-semester nursing students in Stockholm, Sweden, who had conducted clinical education in the EMS and received MSF through a digital instrument. Data were analyzed using reflexive thematic analysis, guided by Braun and Clarke’s methodology.

Results Three themes revealed: feedback from sources familiar with the student’s learning objectives, feedbackfrom sources unfamiliar with the learning objectives, and general perceptions of MSF in the EMS. Students valued self-reflection and feedback from peers and supervisors for personal and professional growth. Patient feedback was challenging due to their limited contextual understanding and emotional states, while feedback from other healthcare professionals was appreciated but hindered by the healthcare professionals’ workload and timing constraints. Overall, students appreciated MSF’s diverse perspectives, enriching their learning, performance, and development.

This study is part of a longitudinal, interdisciplinary implementation research project examining how systematic professional development shapes teachers' digital teaching practices. As schools and other educational organisations continue to adopt Digital Learning Materials (DLM), it is essential to understand the ways in which these shifts affect educators' perspectives and practices. This specific study addresses the research question: What discourses on digital learning material emerge in schools involved in a digitalisation project when a traditional (manual) discourse analysis is applied? We are particularly interested in how discourses are used and constructed by teachers and principals within pedagogical practice. 

Data consist of group interviews conducted at three points during the course of the project: at the beginning of the project, after one and a half years, and finally after two and a half years. In each phase, a total of 47 teachers, and school principals from five schools across four municipalities in Sweden were interviewed. 

The results of the analysis reveal both (A) emerging discourses on conceptualization and idealization of DLM and (B) how the discursive field evolves during the project. Within A) Emerging discourses centred on: DLM as a pedagogical tool; DLM as an effectivization tool; DLM as the limit in the Digital/Analog divide. Within B) Discourses of the digital-analog; Discourses of DLM; Discourses of DLM in teaching.

To conclude, the first part of the results provides a broad overview of the discourses that were present throughout the project. The focus here is on DLMs as part of the teaching context and how DLMs gain significance through how the analog teaching is conducted. The second part of the results becomes more detailed through its division of data into three sampling points and shows how different discourses become hegemonic, suppressed, and transformed throughout the project.

The promise of Learning Analytics Dashboards in education is to collect, analyze, and visualize data with the ultimate ambition of improving students’ learning. Our overview of the latest systematic reviews on the topic shows a number of research trends: learning analytics research is growing rapidly; it brings to the front inequality and inclusiveness measures; it reveals an unclear path to data ownership and privacy; it provides predictions which are not clearly translated into pedagogical actions; and the possibility of self-regulated learning and game-based learning are not capitalized upon. However, as learning analytics research progresses, greater opportunities lie ahead, and a better integration between information science and learning sciences can bring added value of learning analytics dashboards in education.

There is a significant amount of data available about students and their learning activities in many educational systems today. However, these datasets are frequently spread across several different digital services, making it challenging to use them strategically. In addition, there are no established standards for collecting, processing, analyzing, and presenting such data. As a result, school leaders, teachers, and students do not capitalize on the possibility of making decisions based on data. This is a serious barrier to the improvement of work in schools, teacher and student progress, and the development of effective Educational Technology (EdTech) products and services. Data standards can be used as a protocol on how different IT systems communicate with each other. When working with data from different public and private institutions simultaneously (e.g., different municipalities and EdTech companies), having a trustworthy data pipeline for retrieving the data and storing it in a secure warehouse is critical. In this study, we propose a technical solution containing a data pipeline by employing a secure warehouse—the Swedish University Computer Network (SUNET), which is an interface for information exchange between operational processes in schools. We conducted a user study in collaboration with four municipalities and four EdTech companies based in Sweden. Our proposal involves introducing a data standard to facilitate the integration of educational data from diverse resources in our SUNET drive. To accomplish this, we created customized scripts for each stakeholder, tailored to their specific data formats, with the aim of merging the students’ data. The results of the first four steps show that our solution works. Once the results of the next three steps are in, we will contemplate scaling up our technical solution nationwide. With the implementation of the suggested data standard and the utilization of the proposed technical solution, diverse stakeholders can benefit from improved management, transportation, analysis, and visualization of educational data.

Background Assessment of undergraduate students using assessment instruments in the clinical setting is known to be complex. The aim of this study was therefore to examine whether two different assessment instruments, containing learning objectives (LO`s) with similar content, results in similar assessments by the clinical supervisors and to explore clinical supervisors’ experiences of assessment regarding the two different assessment instruments.

Method A mixed-methods approach was used. Four simulated care encounter scenarios were evaluated by 50 supervisors using two different assessment instruments. 28 follow-up interviews were conducted. Descriptive statistics and logistic binary regression were used for quantitative data analysis, along with qualitative thematic analysis of interview data.

Result While significant differences were observed within the assessment instruments, the differences were consistent between the two instruments, indicating that the quality of the assessment instruments were considered equivalent. Supervisors noted that the relationship between the students and supervisors could introduce subjectivity in the assessments and that working in groups of supervisors could be advantageous. In terms of formative assessments, the Likert scale was considered a useful tool for evaluating learning objectives. However, supervisors had different views on grading scales and the need for clear definitions. The supervisors concluded that a complicated assessment instrument led to limited very-day usage and did not facilitate formative feedback. Furthermore, supervisors discussed how their experiences influenced the use of the assessment instruments, which resulted in different descriptions of the experience. These differences led to a discussion of the need of supervisor teams to enhance the validity of assessments.

Conclusion The findings showed that there were no significant differences in pass/fail gradings using the two different assessment instruments. The quantitative data suggests that supervisors struggled with subjectivity, phrasing, and definitions of the LO´s and the scales used in both instruments. This resulted in arbitrary assessments that were time-consuming and resulted in limited usage in the day-to-day assessment. To mitigate the subjectivity, supervisors suggested working in teams and conducting multiple assessments over time to increase assessment validity.

Halfway through a four-year research project supported by implementation science and the Active Implementation Frameworks (AIF), this article reports on the status of the initial two implementation stages. Our research investigates the impact of systematically preparing educators and educational institutions to integrate digital learning materials and learning analytics dashboards to enrich teaching practices and improve student performance outcomes.

Furthermore, it seeks to establish a foundation for the use of innovative and validated educational technology (EdTech) through sustainable implementation strategies, evidence-based evaluation, and continuous redesign of digital learning materials. By adopting this comprehensive approach, we aim to enhance the knowledge base regarding effective digital innovation integration within educational environments.

We argue that applying implementation science in educational settings facilitates the adoption of effective innovations, promotes evidence-based decision-making, and helps identify and address obstacles to change. Our ongoing research underscores the transformative impact of implementation science in education. Thus far, we have highlighted the crucial role of teacher perspectives and the necessity of co-designing technology aligned with teaching and learning objectives.

This nuanced approach refutes the notion of a one-size-fits-all solution or a quick fix achievable in a single academic year. Instead, it advocates a dynamic, collaborative model that acknowledges the multifaceted nature of implementation. Our journey has reaffirmed the dedication of teachers, showcasing their readiness to invest time and effort when their professionalism is respected, and their input is genuinely valued and acted upon.

Research topic and aim

The generation and use of digital data and analyses in education comes with promises and opportunities, especially where digital materials allow use of Learning Analytics (LA) as a tool in Data- Based Decision-Making (DBDM). An exploratory systematic scoping review was conducted regarding the use and impact of LA and DBDM in classroom practice to outline aspects related to Digital Learning Material (DLM), teacher usage, and student learning in the context of K-12 mathematics education.

Theoretical framework

LA implies, analysing data to understand and optimise learning and learning environments (Siemens & Baker, 2012). In this study we discuss LA as “a sophisticated form of data driven decision making” (Mandinach & Abrams, 2022, p. 196). Data driven decision making or DBDM is a process used by teachers to make decisions based on data, to implement improvement actions and evaluate these innovations (Schildkamp & Kuiper, 2010). LA in DLM can offer learners adaptive functions embedded in DLMs or provide learners (and teachers) compiled student assessments in relation to learning goals extracted from learning activities (Wise, Zhao & Hausknecht, 2014). We focus on LA-use based on digital data for student learning, for teaching and for teachers’ DBDM.

Methodological design

The methodology used the five-stage framework (Arksey & O’Malley, 2005), identifying the research question, identifying relevant studies, study selection, charting the data, collating, summarizing, and reporting the results. For the analysis, thematic summary and synthesis was used to answer:

RQ1: How are analyses of digital data from DLM used in mathematics education?

RQ2: How do analyses of digital data from DLM impact teaching and learning?

Expected conclusions/findings

3653 records were identified whereof 15 studies were included. Results show that LA-research is an emerging field where LA-applications is used across many contents and curricula standards of K-12 mathematics education. LA-usage supports a wide variety of teachers’ data use. However, teaching by DBDM focused on supervision and guidance. LA-usage have a positive effect on student learning where high-performing students benefit most. Finally, we suggest a definition of an additional class of LA, which we introduce as Guiding analytics for learners.

Relevance to Nordic educational research

Research on using LA and DBDM is essential to support teachers and school leaders to meet today’s demands of utilising data, to be aware of possible unwanted consequences, and to use technology to enhance active learners and students’ ownership of learning.

Introduction

The generation and use of digital data and analyses in education comes with promises and opportunities, especially where digital materials allow use of Learning Analytics (LA) as a tool in Data-Based Decision-Making (DBDM). LA implies, analysing educational data to understand and optimise learning and learning environments (Siemens & Baker, 2012). In this paper we discuss LA as “a sophisticated form of data driven decision making” (Mandinach & Abrams, 2022, p. 196) as we explore how LA is used to support mathematics teaching and learning with digital materials in classroom practice. Data driven decision making or DBDM has been defined by Schildkamp and Kuiper (2010) as “systematically analyzing existing data sources within the school, applying outcomes of analyses to innovate teaching, curricula, and school performance, and, implementing (e.g., genuine improvement actions) and evaluating these innovations” (p. 482). DBDM is a key for the interpretation of LA, and can use any form of data, but in this review, the term DBDM is restricted to digital data. Using LA as a tool for DBDM could streamline data, making it more readily interpretable. However, questions remain about how usage can translate into practice (Mandinach & Abrams, 2022). 

Quality of technology integration is not merely about technology use, but also about pedagogical use (Ottestad & Guðmundsdottir, 2018), about transformation and amplification of teaching as well as learning through use of technology (Consoli, Desiron & Cattaneo, 2023). LA within Digital Learning Material (DLM) can offer learners adaptive functions seamlessly embedded in DLMs or, provide learners (and teachers) compiled student assessments in relation to learning goals extracted from learning activities (Wise, Zhao & Hausknecht, 2014). The role of the teacher in student learning is clearly of central importance (Hattie & Yates, 2013; Yackel & Cobb, 1996), and teachers have a key responsibility to make digital technology a recourse in teaching to support student learning (Scherer, Siddiq & Tondeur, 2019). 

This paper present findings from an exploratory systematic scoping review which was conducted regarding the use and impact of LA and DBDM in classroom practice to outline aspects related to Digital Learning Material (DLM), teacher usage, and student learning in the context of K-12 mathematics education. 

A scoping review was deemed most appropriate since it can be performed even if there is limited number of published primary research (Gough, Oliver & Thomas, 2017), fitting new research areas such as LA, as it provides “a technique to ‘map’ relevant literature in the field of interest” (Arksey & O’Malley, 2005, p. 20), as well as combine different kinds of evidence (Gough, et al., 2017).

Method

The methodology used the five-stage framework (Arksey & O’Malley, 2005), identifying the research question, identifying relevant studies, study selection, charting the data, collating, summarizing, and reporting the results. The databases ACM Digital Library, ERIC, PsycINFO, Scopus and Web of Science were chosen as they cover a wide range of topics within both technology and educational science to answer:

RQ1: How are analyses of digital data from DLM used in mathematics education?

RQ2: How do analyses of digital data from DLM impact teaching and learning?

The key elements of the research questions, Participants, Phenomena of Interest, Outcome, Context, Type of Source of Evidence (Arksey & O’Malley, 2005) were used to create the eligibility criteria. Publications that were included reported qualitative and/or quantitative data and were connected to the use of DLM and LA based on digital data involving students (between 6–19 years old) and teachers in mathematics K-12 education. The search was limited to papers published from 2000 up-to-date (March 2023) in English, Swedish or Norwegian. Exclusion criteria were developed to ensure consistency within the selection process.

Each record was screened by two reviewers and the relevance were coded according to the inclusion criteria. An independent researcher outside of the review group was consulted to design and validate the results of an inter-rater reliability test. The calculated inter-rater reliability score was 0.822, greater than 0.8, indicating a strong level of agreement (McHugh, 2012). After further screening 57 records were assessed to be eligible. At this stage the review pairs swapped batches and preformed data extraction showing, authors, year, title, location, aim, population, digital technology, method, intervention, outcomes, and key findings was performed for each record. 

The final selection of 15 articles was made by group discussion and consensus. Discussions mainly centred around four components (use, analysis, learning and teaching). The heterogeneity in our sample demanded a configurative approach to the synthesis to combine different types of evidence (Gough et al., 2017). A thematic summary provided the analysis with a narrative approach to answer RQ1. To explore RQ2 more deeply, a thematic synthesis was performed (Gough et al., 2017). The analysis focused on LA-usage based on digital data for student learning, for teaching, and for teachers’ DBDM. PRISMA Extension for Scoping Reviews (PRISMAScR) (Tricco, Lillie, Zarin, O'Brien, Colquhoun, Levac et al., 2018) was used as guidelines for reporting the results.

Preliminary results

3653 records were identified whereof 15 studies were included. Results show that LA-research is an emerging field, where LA-applications is used across many contents and curricula standards of K-12 mathematics education. LA were mainly based on continuously collected individual student log data concerning student activity in relation to mathematical content. Eight of the studies included embedded analytics and all 15 studies included extracted analytics, but accessibility varied for students and teachers. Overall, extracted analytics were mainly mentioned as a function for teacher-usage, available as tools for formative assessment, where analytics need to be translated by teachers into some kind of pedagogical action (i.e., into teaching).

LA-usage supports a wide variety of teachers’ data use, and while mathematics teachers seemed to have a positive attitude towards LA-usage, some teachers were unsure of how to apply it into their practice. The thematic synthesis yielded two themes regarding teaching, which showed that teaching by DBDM focused on Supervision and Guidance. Results indicate extracted analytics is more commonly used for Supervision than guidance. 

Results regarding learning suggest that LA-usage have a positive effect on student learning, where high-performing students benefit most. The included studies examine students’ digital learning behaviour, by describing sequences of actions related to LA, learning outcomes and student feelings. Hereby, through the thematic synthesis, we capture parts of students’ studying-learning process and how it can be affected by LA usage. Finally, we suggest a definition of an additional class of LA, which we introduce as Guiding analytics for learners.

Going forward, research on using LA and DBDM is essential to support teachers and school leaders to meet today’s demands of utilising data, to be aware of possible unwanted consequences, and to use technology to enhance active learners and students’ ownership of learning.

The generation, use, and analysis of educational data comes with many promises and opportunities, especially where digital materials allow usage of learning analytics (LA) as a tool in data-based decision-making (DBDM). However, there are questions about the interplay between teachers, students, context, and technology. Therefore, this paper presents an exploratory systematic scoping review to investigate findings regarding LA usage in digital materials, teaching, and learning in K–12 mathematics education. In all, 3,654 records were identified, of which 19 studies met all the inclusion criteria. Results show that LA research in mathematics education is an emerging field where applications of LA are used in many contexts across many curricula content and standards of K–12 mathematics education, supporting a wide variety of teacher data use. Teaching with DBDM is mainly focused on supervision and guidance and LA usage had a generally positive effect on student learning with high-performing students benefiting most. We highlight a need for further research to develop knowledge of LA usage in classroom practice that considers both teacher and student perspectives in relation to design and affordances of digital learning systems. Finally, we propose a new class of LA, which we define as guiding analytics for learners, which harnesses the potential of LA for promoting achievement and independent learning.