RACOMAT – Risk Assessment COMbined with Automated Testing

The Challenge: Risks of complex ICT systems

Networked information and communication systems (ICT) conquer our daily lives and change the industry. Critical infrastructures, such as power grids or the banking system, are nowadays already heavily dependent on them. The importance of ICT systems will continue to rise with autonomous means of transport (e.g. autonomous cars and trains, drones) in the near future. The direct, existential well-being of persons is thereby increasingly entrusted to information and communication systems. The requirements for their security and reliability are correspondingly high.

In order to capture and minimize the risks, security-critical ICT systems should be subjected to careful risk management in accordance with the established industry standard ISO 31000. However, for complex systems, risk management can be very complex and difficult. While the subjective assessment of experienced experts can be an acceptable method for risk analysis in small scale, other approaches need to be chosen as size and complexity increase. One possibility for a more objective analysis is the use of security testing according to ISO 29119. However, the testing itself can also be complicated and expensive, in particular if unintended, unknown behavior is to be analyzed. Even highly unsafe systems provide many harmless test results, as long as the "wrong" test cases are created and executed.

The Idea

Combine risk assessment and security testing

One way to deal with these difficulties is to combine the different approaches and try to use the strengths. In a complex system, it is first necessary to carry out a high-level assessment of the risks based on experience and literature. In order to make this initial risk assessment more precise, it is possible to use security testing exactly where the first high-level risk image shows the greatest uncertainties. The objective test results can then be used to extend, refine or correct the previous risk image. Economically applicable to large complex ICT systems, however, this method is only provided with adequate tool support.

The RACOMAT Tool

RACOMAT is a risk management tool developed at Fraunhofer FOKUS which in particular combines risk assessment with security tests. The security test can be integrated directly into event simulations, which the RACOMAT tool uses to calculate risks. The RACOMAT Tool enables extensive automation from risk modeling to security testing. Existing databases, such as known threat scenarios, are used by the RACOMAT tool to ensure a high degree of reuse and to avoid errors.

The RACOMAT tool supports a component-based, compliant risk assessment. The tool uses intuitively understandable risk graphs to model and visualize a risk image. For risk analysis, known methods such as Fault Tree Analysis (FTA), event tree analysis (ETA) and the CORAS method can be used in combination in order to benefit from the different strengths of the individual processes.

The RACOMAT tool calculates how much effort is required for security testing in order to improve the quality of the risk image by reducing uncertainties, starting from a total budget for the risk assessment. The tool provides recommendations on how to use these resources. RACOMAT identifies and prioritizes relevant tests.

Automatic Security Testing

In order to be able to automate risk-based testing, a low-level risk assessment is required. For this, RACOMAT allows to model relations between artefacts from the risk analysis and elements of the investigated system. Specifically, the tool uses Threat Interfaces. These can represent components with their input and output interfaces as well as the associated potential weaknesses and hazards.

RACOMAT supports the allocation of potential vulnerabilities and threats with existing expertise from existing libraries such as MITER CAPEC and CWE or BSI IT Grundschutz. Experience values for important safety features - such as probabilities and consequences - are already included in this. Depending on the interface and technology, various potential vulnerabilities and threats are more relevant. The RACOMAT Tool automatically suggests the most relevant vulnerabilities and threat scenarios for identified Threat Interfaces. Thus, the analyst has only to go through a manageable checklist to ensure that nothing is overlooked.

The RACOMAT Tool performs automatic or at least semi-automatic testing using the Security Test Pattern. For typical attack scenarios, RACOMAT provides a catalog of predefined test patterns that can be instantiated without manual effort to test selected components. If there are no matching test patterns yet, new test patterns can be created in the RACOMAT Tool.

The observed test results are used in particular to calculate probabilities that attackers can trigger certain unintended events.

Dependencies and simulations of events

Through the intuitive drag and drop user interface, the RACOMAT tool allows direct relationships between artifacts of the risk analysis and the system model to be created. For typical artefacts, typical relations can also be used for the fully automatic expansion of the model. A precise modeling of dependencies between unintended events is possible with the RACOMAT tool. Weighted relations and gates are available to express how basic events can trigger other events.

The RACOMAT Tool uses Monte Carlo simulations to calculate probabilities of occurrence for dependent events. Randomly distributed input values are used in a large number of simple event simulations in order to be able to determine approximate entry probability values even in the case of complex dynamic interrelations, even if the exact calculation of the solution would be too difficult.

Iteratives Risiko Management
Iterative Risk Management Fraunhofer FOKUS

The RACOMAT Process

The RACOMAT Tool suggests an iterative approach to risk assessment. An initial, coarse risk image is gradually improved in several rounds. The concepts of risk-based security testing (RBST) and test-based risk assessment (TBRA), i.e. the improvement of risk assessment using safety tests, are combined:

1. Develop the initial risk model based on literature, empirical values and expert assessment

2. Use event simulations to calculate the consequences and overall risks of threats

3. Select threat scenarios with the greatest uncertainties that are to be analyzed more closely using security tests

4. Generate required test cases

5. Perform tests and use the results to improve the risk model

6. Once again, use an improved risk model to carry out event simulations in order to determine overall risks more precisely

Continue with step 3 until the budget for the risk analysis is exhausted

7. Final risk evaluation and measures to reduce unacceptably high risks

Support for higher management

Event simulations are not restricted to technical safety-critical events in the RACOMAT Tool. The tool is also suitable for simulating business processes. In the risk assessment, this can be particularly useful for analyzing the indirect effects of technical incidents on the business processes.
A more compact presentation with a focus on the economic aspects makes sense for further risk management, i.e. for risk assessment and risk handling. The RACOMAT Tool provides the management with a dashboard and functions to plan and control risk management. Final, the risks are expressed as costs per time, so they are also tangible for non-technical managers.

Domains-specific assistance

The use of domains-specific information provides further opportunities for support for large organizations. A plug-in has already been developed for the finance and banking sector, which actively supports the modeling of the business scenarios in BPMN and the association with the domains-specific technical infrastructure.

In future, wizards for other domains should also be developed and deployed as plug-ins for the RACOMAT tool.