The MiLEST Toolbox

Model-in-the-Loop for Embedded System Testing

At the early stage of new system functionalities development a model serves as a primary means for including the novelties, yet there is no code, no hardware, and thus no real reference signals for testing. In MiLEST FOKUS proposes a new method for the stimulation and evaluation of embedded hybrid systems behavior which breaks down requirements into characteristics of specific signal features. A novel understanding of a signal is defined that enables us to describe it in an abstract way based on its properties – e.g., decrease, constant, maximum.

top

MiLEST Integration in the W-Model

MiLEST applies to the software built into embedded systems. In particular, it refers to the software models from which systems are built. A model-based approach to functional black-box testing is used to provide a test model. This is in marked contrast to current test methods that are generally dedicated solutions specialized for specific testing contexts. The test method we propose is realized in MATLAB®/Simulink®/Stateflow® environment.

top

MiLEST Architecture

Technically, MiLEST is a Simulink add-on built on top of the MATLAB engine that represents an extension towards model-based testing activities. MiLEST consists of a library including callback functions, transformation functions, and other scripts. The testing library is divided into four composite parts.

top

Model-based Testing

MiLEST Model-based Testing (MBT) is testing in which the entire test specification is derived in whole or part from both the requirements and an SUT model that describes selected functional aspects of the SUT. In this context, the term entire test specification covers both the abstract test scenarios substantiated with concrete sets of test data and expected SUT outputs. It is organized in a set of test cases.

top

Test Development Process

A simplified technically-oriented MiLEST test development process is shown in Figure X. The Model-based Development paradigm assumes that the System Under Test (SUT) model is already available and that the input/output interfaces are clearly defined and accessible. Besides the analysis of the SUT specification, proper functional, dynamic testing also requires systematic selection of test stimuli, appropriate test evaluation algorithms, and obviously an execution or simulation environment. If the above assumptions hold, a pattern for generation of a test harness model can be applied on the SUT model as denoted in step I. This is done automatically with a MiLEST transformation function, giving a concrete frame for test specification. Further on, the test engineer refines the test specification design (step II) using the concept of validation function patterns. Structures for test stimuli and concrete test signals are then generated. This step (step III) occurs automatically with application of the transformations. The test control design can be added automatically too (step IV). Finally, the tests may be executed and test results obtained in the form of verdicts (step V). At the same time the quality of the test system specification produced is also assessed.

top

Hierarchical Test System

The MiLEST specifications reflect the structure of the system requirements. The test development process and the abstractly aligned test system enable us to apply the concepts of signal feature generation and detection mechanisms while also building test specifications systematically. In this context, the test system is understood as a hierarchically structured test model (also called the ‘test design’). Since the aim of the test system is not to provide the means for testing a single signal property but for validating complete SUT models independently of their complexity, structuring the test models in a proper way contributes to the scalability and reusability of the solution. Moreover, traceability of the test development artifacts is possible and transformation potentials can be identified The structure of the test system consists of four different levels that can be built systematically. This makes it less error-prone while also leaving the test engineers plenty of scope for developing the complete test specification.

top

Features

• Systematic, consistent functional test specification
• Signal feature – oriented paradigm
• Graphical test design
• Test process automation
• automatic test data generation
• online automatic test evaluation
• Model-in-the-Loop test execution
• Reusable test patterns
• Abstract and concrete views

top

Benefits

• Testing in early design stages
• Test of hybrid systems including temporal and logical dependencies
• Traceability of test cases to the requirements
• Traceability of verdicts to the root faults
• Increased test coverage and test completeness
• Assured test quality of the test specification

top

Publications

MiLEST is original work undertaken between September 2005 and August 2008 at the Fraunhofer Institute for Open Communication Systems, Competence Center – Modeling and Testing for System and Service Solutions, the Technical University of Berlin, Faculty of Electrical Engineering and Computer Science, and the University of California, San Diego, Department of Computer Science and Engineering. It has been financed by several research grants and the doctoral fellowship awarded to the author by Studienstiftung des deutschen Volkes (english: German National Academic Foundation). Portions of this work have been already presented elsewhere. They resulted in the following publications:

• ZANDER-NOWICKA, J., SCHIEFERDECKER, I., MARRERO PÉREZ, A.: Automotive Validation Functions for On-line Test Evaluation of Hybrid Real-time Systems, In Proceedings of the IEEE 41st Anniversary of the Systems Readiness Technology Conference (AutoTestCon 2006), Pages: 799-805, IEEE Catalog Number: 06CH37750C, ISBN: 1-4244-0052-X, ISSN: 1088-7725, Anaheim, CA, U.S.A. 2006.
• ZANDER-NOWICKA, J., SCHIEFERDECKER, I., FARKAS, T.: Derivation of Executable Test Models From Embedded System Models using Model Driven Architecture Artefacts - Automotive Domain, In Proceedings of the Model Based Engineering of Embedded Systems II (MBEES II), Editors: Giese, H., Rumpe, B., Schätz, B., TU Braunschweig Report TUBS-SSE 2006-01, Dagstuhl, Germany. 2006.
• ZANDER-NOWICKA, J., MARRERO PÉREZ, A., SCHIEFERDECKER, I.: From Functional Requirements through Test Evaluation Design to Automatic Test Data Retrieval – a Concept for Testing of Software Dedicated for Hybrid Embedded Systems, In Proceedings of the IEEE 2007 World Congress in Computer Science, Computer Engineering, & Applied Computing; The 2007 International Conference on Software Engineering Research and Practice (SERP 2007), Editors: Arabnia, H. R., Reza, H., Volume II, Pages: 347-353, ISBN: 1-60132-019-1, Las Vegas, NV, U.S.A. CSREA Press, 2007.
• ZANDER-NOWICKA, J., MARRERO PÉREZ, A., SCHIEFERDECKER, I., DAI, Z. R.: Test Design Patterns for Embedded Systems, In Business Process Engineering. Conquest-Tagungsband 2007 – Proceedings of the 10th International Conference on Quality Engineering in Software Technology (CONQUEST 2007), Editors: Schieferdecker, I., Goericke, S., ISBN: 3898644898, Potsdam, Germany. Dpunkt.Verlag GmbH, 2007.
• ZANDER-NOWICKA, J.: Reactive Testing and Test Control of Hybrid Embedded Software, In Proceedings of the 5th Workshop on System Testing and Validation (STV 2007), in conjunction with ICSSEA 2007, Editors: Garbajosa, J., Boegh, J., Rodriguez-Dapena, P., Rennoch, A., Pages: 45-62, ISBN: 978-3-8167-7475-4, Paris, France. Fraunhofer IRB Verlag, 2007.
• ZANDER-NOWICKA, J., XIONG, X., SCHIEFERDECKER, I.: Systematic Test Data Generation for Embedded Software, In Proceedings of the IEEE 2008 World Congress in Computer Science, Computer Engineering, & Applied Computing; The 2008 International Conference on Software Engineering Research and Practice (SERP 2008), Editors: Arabnia, H. R., Reza, H., Volume I, Pages: 164-170, ISBN: 1-60132-086-8, Las Vegas, NV, U.S.A. CSREA Press, 2008.
• ZANDER-NOWICKA, J.: Model-Based Testing of Real-Time Embedded Systems for Automotive Domain, In Proceedings of the International Symposium on Quality Engineering for Embedded Systems (QEES 2008); in conjunction with 4th European Conference on Model Driven Architecture – Foundations and Applications, Pages: 55-58, ISBN: 978-3-8167-7623-9, Berlin. Fraunhofer IRB Verlag, 2008.
• ZANDER-NOWICKA, J., MOSTERMAN, J. P., SCHIEFERDECKER, I.: Quality of Test Specification by Application of Patterns, In Proceedings of the 2nd International Workshop on Software Patterns and Quality (SPAQu 2008); in conjunction with 15th Conference on Pattern Languages of Programs (PLoP 2008), co-located with OOPSLA 2008, Nashville, TN, U.S.A. to appear, 2008.

The authors tend to protect their invention in the market. Hence, the proposed algorithms for software quality assurance, the corresponding test methodology, and the resulting test execution framework, called MiLEST have been submitted to the United States Patent and Trademark Office (USPTO) in 2008 as a patent application.