The Security Testing Improvements Profile (STIP)

Software testing is the most important measure of quality assurance in software development. Security testing plays currently a minor role, but we are sure that this will change within the next five years. Studies show that in industry, testing processes are often perceived as too costly and laborious. Testing processes are often difficult to control and improvements of the processes with respect to quality, cost efficiency and stringency are often requested.
Security testing key areas
Figure 1: Security testing key areas Fraunhofer FOKUS

STIP principles

Fraunhofer has developed an assessment scheme called Security Testing Improvements Profile (STIP), that is dedicated to assess security testing processes. It can be used stand alone or in addition to established test process assessment approaches. STIP is based on the general ideas of TMMi and TPI. Thus, we have defined a set of key areas that we considered relevant for security testing. The key areas describe major aspects or activities in a security testing process.

The key areas are grouped in four main groups and defined to be self-contained and distinct so that each of the areas represents a relevant aspect of a security testing process. For each of the key areas we have defined a performance scale with up to four levels that are hierarchically organized and build on each other. The levels can be used to evaluate concrete security testing processes with respect to their performance in the belonging key area. Each level with a higher number represents an improvement for the underlying security testing process. Each higher level is better than its prior level in terms of time (faster), money (cheaper) and/or quality (better).

Evaluation results of one of the DIAMOND case studies
Figure 2: Evaluation results of one of the DIAMOND case studies Fraunhofer FOKUS

Application to the DIAMONDS Case Studies

The STIP approach has been used to evaluate all of the case studies in the DIAMONDS project. To explicitly show the progress that has been made during the DIAMONDS project, we carried out two assessments for each case study. The first assessment explicitly considers the application of the DIAMONDS techniques & tools and thus provides us an impression of the security testing processes in the case studies at the end of the DIAMONDS project. The second assessment intentionally disregards the results DIAMONDS and thus gives us an impression of the maturity of the testing processes before DIAMONDS.

Figure 2 shows the results of the case study on testing the application software for bank note sorting machines. The score before the project started is denoted in red and after the project in blue. We can observe that the case study advanced in nearly every aspect of security testing. The case study gained from nearly all the relevant innovations of the DIAMONDS project with the exception of monitoring. The biggest gains were made in the areas Fuzzing and Test generation where the case study was used as a driver for the research project.

Service Offer

Originally, STIP has been developed as an objective, detailed analysis and evaluation of the DIAMONDS research and development in case studies. After the project, it has been extended to cover the assessment and improvement of security testing processes. It shows how tools, techniques and methodologies fit together and provide recommendations for others on how to practically integrate our results to improve security-testing processes at hand. If you are interested in evaluating your processes please contact us.