Market researches show the growing importance of IoT: Gartner forecasts 20,4 billion networked products worldwide by 2020 (January 2017). The European Union also recognizes the expansion of IoT. Between 2014 and 2017, the EU invested nearly € 200 million through the Horizon2020 initiative in IoT research, innovation and deployment. However, the EU not only recognizes the opportunities of digitalization and networking but also the accompanied risks of attacks against insecure or vulnerable technologies.

How does this risk of attacks arise? A few years ago, manufacturers of devices just like toasters were not confronted with the potential that their devices could be a target of hacker attacks. However, in October 2016, the malware Mirai Botnet exploited a high number of networked devices: Networks were overloaded on purpose using unchanged default logins in a massive DDoS (Distributed Denial of Service) attack. As a result, services such as Amazon, Twitter or Netflix had major disruptions or even outages.

Threats such as Mirai – a botnet consisting of insecure IoT devices such as toasters or coffee machines – show that there is an urgent need for action regarding quality assurance and certification in the IoT area. It is necessary to test devices accordingly concerning functionality, interoperability, robustness, security and trustworthiness. However, so far there is no holistic solution to the problem available. Test tools in the IoT community already exist, but they are often incomplete, poorly maintained or only available commercially. Meanwhile, the security and (data) privacy of IoT products and applications cannot be neglected.

A new tool for IoT testing and certification is the Eclipse IoT-Testware. It is part of the IoT-T project funded by the Federal Ministry for Economic Affairs and Energy (BMWi). The aim of this IoT -Testware is to assist companies in the quality assurance and certification of their IoT-based solutions and products.

Based on the test methodology practiced at ETSI (European Telecommunications Standards Institute), a systematic approach characterizes the IoT-Testware. Automated test suites are developed for IoT protocols and services, using standardized and proven technologies such as TTCN-3 which is an established standard for testing protocols. Evolving since 1998, TTCN-3 has been applied for compliance testing of VoIP, 4G and IPv6. With the introduction of the IoT-Testware, TTCN-3 is applied for the first time in conformance test suites for CoAP and MQTT protocols. In addition to HTTP, new protocols like CoAP and MQTT are the foundation for the communication between networked things. The conformance tests aim at robustness, reliability and dynamics of open environments. IoT-Testware pursues a holistic approach with an open source strategy covering and extending the goals of other tools.

Test Tool Eclipse Titan

The test suites created with the Eclipse IoT-Testware are implemented and executed with the testing tool “Eclipse Titan”. Eclipse Titan is applied as a central environment to integrate test cases described in TTCN-3. Originally developed by Ericsson as an in-house tool, “Titan” now has been freely available as open source software for all TTCN-3 users for several years. Among others, Ericsson itself has already developed implementations of test access points (so-called test ports) for the MQTT and CoAP protocols. These are used in the respective IoT-Testware test suites.

IoT Test Methods

One of the most important tasks in IoT testing is to define suitable test suites. Based on user scenarios and system architectures of IoT products as well as on selected test targets, the access points of the test system components need to be defined. For each test it is required to divide the sequence and to assign them to parallel test components, which perform the defined test steps or simulate an environmental behavior.

The specific use of the IoT test methods depends on the respective test objects and objectives. Such test objects are for example:

• Microcontroller
• Protocol implementations
• IoT Gateways
• Cloud Server
• User devices
• as well as IoT infrastructures and solutions.

After testing, developers are able to trace the individual test cases back to the standard conformance statements. Through this, developers not only obtain the success rate, but also a direct reference to the respective conformance statements of the standard.

Holistic Solution through the IoT-Testware

In previously existing test tools often only limited aspects were considered, which may not lead to a sufficient quality statement. Particularly for SMEs and start-ups with limited resources, the young IoT sector lacks a cost-effective and comprehensive solution, which needs to check quickly and reliably and to prove quality, security, scalability and interoperability.

Overall, the Eclipse IoT-Testware will close existing gaps in quality testing for IoT devices as well as solutions and allows for certification. The currently developed conformance tests for MQTT and CoAP represent only an initial part of this. Another important aspect of the IoT-Testware is its open-source nature.

The IoT-Testware thus supports a comprehensive certification and is the basis for awarding an IoT seal of quality. Especially with a certificate, trust in the product itself and its safety can be strengthened significantly. Due to the extremely fast growth rate of IoT systems, device manufacturers are trying to bring their products to market as quickly as possible. Security and privacy requirements are (yet) not a high priority, which in turn has a negative impact on consumer confidence. Appropriate standards verified by a holistic IoT-Testware can be used to grant certificates and restore confidence in tested products.

Certification with the IoT-Testware is currently not available to the industry. Nevertheless, test labs and interested developers can already benefit by using the implemented test cases, thus improving their own quality control.