Evaluation of digital educational media with eduCheck digital
eduCheck digital is a project commissioned by the 16 German federal states to develop legal and technical standards, as well as an associated evaluation procedure, for digital educational media. The project’s objectives are therefore (1) to develop defined quality standards in the individual evaluation areas and (2) to test the feasibility of a joint evaluation procedure across the federal states.
The assessment areas cover law & data protection, IT security & technology, interoperability, accessibility and the use of AI. Through a media classification system, providers can determine which assessment criteria are relevant to their product; these are then verified using defined tests. The cross-state project, funded by the DigitalPakt Schule, is coordinated by the Institut für Film und Bild in Wissenschaft und Unterricht gGmbH (FWU). In the long term, the assessment within the eduCheck digital framework is intended to provide guidance for ministries, teachers and school authorities, as well as for learners and parents. To this end, a reliable, transparent and standardised assessment process for media providers is to be established.
Assessment area “Interoperability”
A central component of Fraunhofer FOKUS’s research and development work is the conceptual development of testing criteria in the field of “Interoperability”. In doing so, we systematically examine existing standards and technical specifications to identify which requirements are suitable for subsequent automated testing. We implement these in a test framework consisting of a test suite and test scenarios. Our experience from numerous projects enables us to formulate practical criteria that are both technologically sound and compatible with real-world educational infrastructures.
Test suite process
The architecture of the test suite describes a modular system for executing and evaluating test scenarios. The aim is to map tests in a digitally structured manner in line with the eduCheck test criteria and to adapt them flexibly to different requirements.
The test suite natively supports various types of test scenarios:
The architecture is designed to be scalable: additional test servers can be flexibly integrated. To this end, dynamic management of the connected systems is provided, which efficiently controls the execution and forwarding of tests.
A central component of the test suite is the storage and evaluation of test results. In the future, it is planned to make these available in various formats to enable versatile use.
The use cases range from debugging and quality assurance in development to comprehensive verification of compliance requirements.
Architecture of the test suite
The starting point for all test processes is the Test Criteria Manager (TCM). This is where the tests and test sequences are defined and managed. Based on these sequences, the required input parameters are recorded, checked and converted into a technically executable test structure. This results in standardised test artefacts, known as tasks, which translate business requirements into concrete technical processes. The actual test scripts can either be stored directly in the TCM or remain in connected services. In the latter case, the TCM primarily handles the control of parameters and the assignment of tests to the respective responsible execution services.
The distribution and control of test processes is handled by the Dynamic Service Handler. It ensures that each test sequence is forwarded to the appropriate service. Different types of tests can be supported in parallel. For example, manual tests can be integrated, in which users are guided step by step through defined test procedures. At the same time, automated tests can be implemented, such as for checking APIs or system structures, which run entirely on the server side and without direct user interventio
A client-side automated test run, in the eduCheck digital context, is carried out by a test automation framework such as Playwright or Appium. This framework passes the required input data to the Reference Interoperability Client (RIC), which then checks whether the defined standards and criteria are met. At the same time, the RIC visually presents the results, including logs, screenshots and other artefacts, so that the test run can be transparently traced. This information is then structured and processed by the test framework.
The Test Protocol Manager (TPM) is responsible for the final processing of the test results. It is responsible for converting all collected data into a standardised and reusable format. The results are stored in a database and form the basis for evaluations and future analyses. In the future, it is also planned to be able to export the test results in various formats and make them easily available.
We are delighted to be involved in this exciting project.
Fraunhofer Institute for Open Communication Systems