Written by Johanna Sochos | June 10, 2020
Before going further, let us briefly revisit our test concept for measuring interactivity by sending a continuous stream of UDP packets from client to server and back with an underlying realistic traffic pattern for a given application type. With this test concept, we can measure key technical parameters of a network, including
- Data rates of TX and RX
- Round-trip latency of the packets
- Packet delay variation (jitter) as the latency variation over time
- Packet loss/packet error rate
Our interactivity score unites these key measures into a QoE model, which gives a single score for the interactivity of a network for a target application.
Interactivity test accuracy
First, we want to discuss the measurement accuracy we have achieved with our implementation, as the interactivity test targets ultra-reliable low latency communication (URLLC) applications made possible by 5G networks. To evaluate the measurement accuracy that can be achieved in an ideal network channel, we have created a reference setup with extremely low latency.
The smartphone with the interactivity test running on a QualiPoc handheld was connected directly via Ethernet (over a USB interface) to a Linux server where the session reflector was located. The whole setup was standalone and not connected to the Internet so that no background data could interfere with the test.
In this reference setup, we conducted more than 1000 tests with the eGaming real-time pattern and achieved very low latencies, packet delay variation, and a high interactivity score. Specifically, we measured an average two-way latency of 1.26 ms ± 0.01 ms. The average interactivity score is 99.6 %. Figure 1 below illustrates the reference setup and shows the measurement results of one of the tests done in this setup for accuracy analysis.
Figure 1: Reference setup for accuracy analysis (left); measurement results in the low latency reference setup
The analysis of the outlier ratios shows that only 1:5000 packets arrive more than 10 ms later than the median RTT. These slower packets are probably caused by processing time differences on Android, as it is not a real-time operating system.
Overall, the measurement accuracy is high enough to measure round-trip latencies of down to 1 ms and thus is suitable for URLLC measurements.
Interactivity test results under good 4G conditions
As a first real field example, we want to show a typical result in good network conditions in 4G/LTE with a fast, well-connected in-country cloud server. In such conditions, we see median round trip latencies of approx. 40 ms down to 25 ms. Under good network conditions, no packet is lost or delayed beyond the packet delay budget of 2 × 50 ms, and the channel QoS is therefore 100 %.
The interactivity score is lower than the maximum of 100%, with values between 75 % and 85 % due to a rather long round-trip latency for real-time applications. However, even in this range, a fair to good real-time eGaming experience can be expected.
Figure 2: Overall interactivity test results in good 4G conditions as presented in SmartAnalytics
In the picture below, every green dash marks the individual round-trip latency of one packet. It gives an immediate impression on the distribution of latencies over time. The quasi-continuous data flow of the eGaming traffic pattern emulates real application load patterns. This leads to a broader and much more realistic distribution of measurements than a ping test could deliver.
Figure 3: Per-packet round-trip latencies and intermediate results of one interactivity test in good 4G conditions
Example of good NR 5G EN-DC conditions
Now that 5G networks are emerging, let us look at how they perform regarding latency and interactivity compared to 4G networks. Measurements in 5G EN-DC mode with dual connectivity in good network conditions show that the latency is typically only a few ms less than in 4G networks. In addition, the packet delay variation is also a few ms less compared to LTE. The interactivity score is thus higher and in the range of 85 % to 90 %, which indicates a good to very good real-time eGaming experience.
Figure 4: Interactivity test results in good 5G EN-DC conditions.
The reason for the rather small difference in round-trip latency between measurements in 4G and 5G EN-DC is that the underlying radio technology and core network are still the same. Therefore, there is no native benefit to be expected for latency measurements.
Very interesting are the results of an in-house benchmarking campaign conducted in January 2020 in two European countries with five different providers in total. The 10th percentile of the round-trip latency is a measure for the “best-case scenario” latency.
Providers A, B, D, and E have 5G EN-DC deployed, and we measure a decrease in latency of a few ms for 5G compared to 4G if looking at the 10% fastest transmissions. Provider C, however, who has not yet rolled out 5G, can compete with the 5G latency of the other networks due to the higher bandwidth allocation for provider C’s 4G network.
Table 1: Benchmarking results of measuring latency with the interactivity test in two European countries in January 2020.
We expect that a clear benefit of the 5G technology will only be visible once the standalone mode with the dedicated core network is available.
Example of 4G/5G with carrier aggregation
The interactivity test is especially valuable to detect short-term problems in the packet flow for highly interactive applications like a temporal build-up of the latency, lost and highly delayed packets, and longer interruptions. These kinds of impairments are usually invisible to most types of application tests.
There are many interesting examples to demonstrate the detection of short-term problems in the packet flow. We will look into one that presents several representative events in a single test. It was conducted with the eGaming real-time traffic pattern and started in the 4G technology. In the third second of the test, when the bandwidth demand increases, a second 4G carrier was added.
This simple network action often leads to temporarily increased latencies beyond the allowed maximum delay of 100 ms; the packet loss is marked as red intervals in figure 5 below. The connection recovers after the completion of network reconfiguration. Similar problems occur during handovers, and also when the 5G carrier is added. In the last two seconds of the test in stable 5G network conditions, the slight drop in the round-trip latency compared to 4G is visible.
Figure 5: Interactivity test results in the presence of carrier aggregation and handovers
Overall, the median round-trip latency is still in a tolerable range, but the high packet delay variation and packet loss are resulting in an interactivity score of 0 %. In these network conditions, it is not possible to play a real-time game.
Figure 6: Overall interactivity test results in the presence of carrier aggregation and handovers.
Carrier aggregation and aggressive cell changes maximize data throughput but at the expense of latency and real-time interactivity. This finding will have to be addressed by network providers as the importance of interactive applications is on the rise.
Rate interactivity and identify transmission bottlenecks
5G will enable a new set of services, not only for human users but also for industrial and automotive purposes. Key factors for acceptance and use are the ultra-high reliability and the ability for real-time interaction. For these future applications, it is key to measure and to rate the interactivity of a network and to efficiently identify bottlenecks in the transmission.
- The interactivity test combines testing round-trip latency, packet delay variation, packet error rate, and proofing bit rate in one single test
- It can measure very short latencies with high accuracy and is URLLC-ready
- The configuration based on realistic traffic patterns is very flexible and scalable and can easily be extended to a large range of application types
- It provides a full set of detailed results down to per-packet analysis
- The smart integrative scoring delivers a representative quality indication for a given use case such as eGaming
- The implementation based on the standardized TWAMP protocol ensures the very flexible configuration of the responding servers with existing components anywhere in the network
Read more about interactivity test in part 1 - 8 of this series and download our white paper "Interactivity test":
Interactivity test: QoS/QoE measurements in 5G (part 1)
Interactivity test: Concept and KPIs (part 2)
Interactivity test: Distance to server impact on latency and jitter (part 4)
Interactivity test: Impact of changing network conditions (part 5)
Interactivity test: Packet delay variation and packet loss (part 6)
Interactivity test: Dependency of packet latency on data rate (part7)
