IMS Testing

Before you release your hardware/software you need to test your product in real world like environment and by running of systematic step-by-step tests. The IMS Playground constitutes such an environment, where various experiments can be run.
FOKUS is developing systematic tests to test functionality, interoperability and conformance of IMS components, including definition of test purposes, choice of the method, test case specification, execution of tests and evaluation of results.

Another group of test comprises performance tests and benchmarking. The actual work concentrates on signaling paths in IMS: SIP and DIAMETER messages, with end user simulation and measurements of response time, rate (throughput) in TPS (transactions pro sec) and resource utilization. Different kinds of tests are possible:

• benchmarking – for comparison
• load/stress – how systems perform under load
• capacity testing – max load the system can handle before failing
• scalability testing – to plan capacity improvements
• interoperability testing – to verify the compatibility and conformance

SIPNuke – The IMS Load Generator

SIPNuke is at its core an NGN load generator designed to simulate realistic load conditions. We believe that stateless load generators are incapable of recreating traffic profiles with the required level of accuracy in order to test NGN systems and produce results that are representative and comparable with real-life exploitations. In the context of emerging Service Delivery Platforms, Web 2.0 and Service Oriented Architectures, simplistic measurements like SIP transactions handled or BHCA are no longer sufficient for assessing the composite performance and behavior of an entire system. As such SIPNuke was built to easily emulate millions of subscribers in parallel while also offering ultimate output performance and full-flexibility in scenario specification.

Developed at NGNI as an IMS testing tool, it is used in projects where flexible scenarios and high traffic levels are required. SIPNuke has been successfully validated in many NGN/IMS related projects. A notable example is the MAMS project where SIPNuke has assessed the performance limits of our Open Source IMS Core implementation. A full performance report was written concluding that our Open Source IMS Core implementation has leading performance levels. For this report, the traffic generation capabilities of SIPNuke outperformed currently available alternatives.

SIPNuke's distinctive characteristics are, as stated, its efficiency at generating loads for determining performance levels of server platforms and its extensible design.

In addition to the set of features that are present in most SIP/IMS testing tools, our solution includes additional functionality needed for the projects we are working on:

• powerful scenario specification language
• record/playback of scenarios
• message templates
• debugging of scenarios
• multiple subscribers (essentially 250.000 for 1GB of RAM)
• configurable transport protocols (multiple UE IPs and ports, different proxy addresses per UE)
• completely open support for defining traffic distribution models
• multi-instance support (synchronization with a central process)
• RTP support
• integrated AS support

MAMS Benchmark Results

The tests that were conducted as part of the MAMS project were mainly focused on determining limit performance numbers and testing stability of the IMS under significant traffic loads on individual scenarios.

Generally, results have been collected in the form of charts that measured the latency and error rate for different scenario rates.

The tests were grouped into the following categories:

• Registration
• Page-Mode Messaging
• Two-Party Calls
• Service Triggering
• Loopback Tests

As an example, the messaging tests resulted in the following two charts.

First, the performance limits and under-stress behavior of the IMS network were analyzed using a ramp-like scenario rate:

A second test was run in order to confirm that the system under test (SUT) behavior was stable, by generating a constant load. SIPNuke was able to precisely hold high very throughput rates with predictable behavior.

Interoperability Testing

FOKUS implements the IMS NNI Interworking Test Specifications standardized and provided by ETSI. Maintaining this set of tests makes it possible to assess compatibility of any IMS core network implementation by testing against the Open Source IMS Core. In this context, the Open Source IMS Core is already being used by a large community as a reference for NGN core networks. As such, SIPNuke and the Open Source IMS Core together offer an optimum platform for acceptance testing as well as providing excellent opportunities to perform automated testing and identify regressions easily.

The interoperability (IOP) tests are run using the SIPNuke testing tool developed at FOKUS and we are exploring the possibility of building further performance and functionality testing, with the same application as a starting point.

ETSI IOP

The ETSI Interworking Test Specification's primary role is to select a minimal set of tests that allows IMS deployers to be confident about an IMS implementation's basic adherence to basic standards. It is extremely useful for determining the level of compatibility between two IMS service providers. The test suite is structured into 6 major groups:

• Group 1.1: General Capabilities
• Group 1.2: IMS-ALG
• Group 1.3: Initial dialog or standalone request procedures
• Group 1.4: Registration procedures
• Group 1.5: Special Cases
• Group 1.6: Subsequent or target refresh requests on a dialog procedures

Several tests are quite similar in the way they are described, and thus would benefit heavily from a way to specify them in a modular fashion, reusing as much as possible. This goal has been easily reached using our testing tool and it has significantly cut down the amount of work needed to arrive at a functional state.

ETSI – IOP Standards

SIPNuke as a base for IOP testing

The code for the mentioned test suite reuses common parts that have been developed for different projects and that is required for every use case. Furthermore, templates for the messages exchanged are reused for all tests, so that 17 templates are used for 63 tests. Furthermore, it is unlikely that, as new tests are added, new templates will need to be written.

Running tests in an automated fashion is easy and it is trivial to add report generation based on templates. This makes it possible to track the progress of IMS implementations with respect to standards conformance. Provisioning of users (such as database initialization) and setting up or disabling Application Servers (AS) for certain tests can be done on the fly as well, so that tests can be run fully unattended.

IMS Benchmarking

Benchmarking allows the comparison of performance of different implementations. For load generation of SIP or DIAMETER state-full traffic standard test notation TTCN-3 can be used and TTCN-3 test execution environment (TTworkbench form TestingTech). FOKUS is developing Abstract Test Suite for IMS benchmarking in TTCN-3. Various important QoS parameters like transactions/calls pro second, delay of important messages, number of supported users, how many failures acceptable are defined and measured.


NGN Testing with SIPNuke – PDF

IMS Benchmarking – Project Description