Testing is essential to ensure private network performance

The 5 phases of mobile network testing in a smart factory

Figure 1 below shows the five test phases of network testing in smart factories or any other business-critical private networks. These phases are indispensable for achieving a high-quality mobile network and verifying the strict reliability and latency requirements.

Engineering – phase 0 / ongoing
Engineering activities are typically aimed at verifying the RF and application performance of a novel network technology, as well as features and services in the early phases of introduction based on lab tests and field trials, where the interaction of the network and devices is in focus.
In this phase features are required to stimulate specific network behavior (e.g. by forcing the device or by creating specific application layer traffic patterns) to help experts to understand the network behavior under different conditions.
The Engineering phase often precedes network rollout, can happen in the field as well as in the lab and is typically done by experts.

ROMES is a powerful software platform for network engineering and optimization which enables a high degree of control of device and measurement parameters and delivers detailed results from all layers. Deep insights can be achieved with detailed analysis with the ROMES TTI view that provides a deep dive view even down to HARQ processes between the devices and the network to identify root causes of potential problems

Rollout preparation – phase 1
In some countries, factory owners have access to the 5G spectrum solely dedicated to campus or private networks. The first task for these factory owners is to ensure that the new spectrum is interference-free. The R&S®TSMx network scanners, together with a handheld interference hunting spectrum analyzer and monitoring receiver – such as the R&S®Spectrum Rider FPH and R&S®PR200 portable monitoring receiver, respectively – are an excellent choice for sustainable rollout preparation.

Site acceptance and performance tuning – phase 2
If the private network is deployed, its network performance needs to be tuned to fulfill the required customer KPIs. For this task, passive RF and active tests have to be executed and documented finally in an acceptance report:
During site acceptance, the operation of newly deployed base stations will be tested and validated. This phase includes simple, functional tests, such as download (DL) and upload (UL) tests and round-trip latency measurements, over-the-air (OTA) RF spectrum analysis, and signal decoding to verify the PCI, SSB, and SIB information of 5G and LTE anchor signals. Signal decoding also helps in troubleshooting specific parameters in case of issues or unexpected results.

The smartphone-based troubleshooter QualiPoc Android executes functional DL, UL, and ping tests; the handheld spectrum analyzer R&S®Spectrum Rider FPH performs OTA spectrum measurements, and the 5G site testing solution ( R&S®5G STS ) helps tuning the network performance in case of issues.

Performance & service level verification – phase 3
The private network is now deployed, has the correct network performance (customer KPIs are met) and the network is in its operational phase. It is important that the performance level (in terms of coverage and other KPIs such as data throughput and latency) is regularly checked because every modification in the network, e.g. moving a metal shelf or robot on the factory shop floor, will affect the signal propagation characteristics and can affect the coverage and network performance.
QualiPoc Android tests the latency and real-time capability of communication links by combining the emulated traffic behavior, latency, and continuity when running the unique interactivity test.
This can be done either manually (a person walks through the whole network area with an R&S®Freerider 4 backpack or a R&S®5G STS setup in a shoulder bag and measures and documents the performance) or automatically with SmartMonitor (data collection probes are spread throughout the whole private network to autonomously execute and analyze tests continuously (24/7/365) to also allow trend analysis). With this continuous performance monitoring the private network can be predictively maintained and kept to an optimum performance before an issue occurs.
The web-based SmartMonitor application manages a fleet of specifically tailored RF probes that are distributed throughout the factory, as well as in automated guided vehicles (AGV) and autonomous mobile robots (AMR). The probes continually check the connectivity levels, including latency, and report the results to a central unit (the SmartMonitor dashboard) displaying results and probe status in real time.
Any deviation from normal conditions or trends are visible immediately. The data analytics software suite SmartAnalytics analyzes the data offline to identify trends and to detect anomalies.

Troubleshooting – phase 4
If a problem occurs in the private network (acceptance after rollout does not show good performance, KPI not met after some operation time, negative trend in certain areas, etc.), technically skilled persons should do troubleshooting. This activity often contains detailed passive RF and active tests on site with the goal to identify the root cause of such network issues and certainly correct the failures and optimize the network.
All kinds of mobile network testing tools mentioned above can be used to troubleshoot network issues.

• Network scanners (R&S®TSMx) measure signal levels (RSRP) and quality (SINR) of access points and can detect degradations compared to the acceptance phase on physical level.
• QualiPoc SW capabilities running together with industry modules or industry routers via the diagnostic port can conduct dedicated measurements targeting certain areas, test types or traffic patterns that show a degradation.
• ROMES with the deep insights real-time analysis capabilities (ROMES TTI view) is also a powerful troubleshooting tool.
• With the data analytics software suite SmartAnalytics, the measurement results can be visualized and underperforming tests or areas analyzed by the provided drill-down possibilities down to per test or even per IP packet level which greatly facilitates the identification of the root cause of the network issue.

How to test latency in a smart factory network

In part 1 of this article series about smart factories, I already touched upon the difference between round-trip latency and one-way latency. Before going into more detail about these differences, let’s quickly review the use cases for round-trip latency and one-way latency, respectively.

• AR/VR use cases, for example, need low round-trip latency because if the technician wearing AR/VR glasses moves his head, the image depicting instructions needs to update very quickly to ensure proper operation. Experts also forecast a fast-paced development of new VR use cases in various environments that will increase the demand for higher data throughputs and low latencies.
• Robot control, on the other hand, requires low one-way latency. The system sends an order to the robot, and the robot has to act immediately (e.g., stop movement); it is just one-way communication.

Round-trip latencies are traditionally measured using ping messages. By definition, ping is a basic Internet software utility to verify that a particular Internet node exists and can accept requests. The accessibility of the Internet node is confirmed by its feedback, the ping time. However, ping has some inherent disadvantages for accuracy, particularly for very low latency values.

Other arguments for no longer using ping include emulating the typical traffic behavior of the communication to robots and other entities in a smart factory. For this, the preferred protocol is the Two-Way Active Measurement Protocol (TWAMP) that has been specified by the Internet Engineering Task Force (IETF) to verify service-level agreements (SLA) regarding times/latencies.

In the illustration figure 2 below, the measurement device sends a configurable User Datagram Protocol (UDP) stream of unique packets emulating a realistic traffic profile of a use case class to an (active) server (uplink) that immediately reflects (TWAMP reflector) it to the device (downlink).

The measurement device’s QualiPoc Android software runs the interactivity test to analyze the round-trip latency, packet delay variation, and packet error rate to calculate an interactivity score for this specific use case class based on its particular QoS requirements.

The interactivity test component resulting from the round-trip latency forms an S-shaped curve, as shown in figure 3. The packet delay variation and packet error rate components are factors [0, 1] that scale down the S-curve.

The parametrization of the interactivity test and interactivity score can be individual for each application class (e.g., AR/VR remote support or robot control in a smart factory) depending on the latency and real-time requirements for the specific application class. The interactivity test forms the basis for a scalable QoE model that can be adapted to all kinds of interactive applications.

One-way latency measurements require the synchronization of sender and receiver. In a prototype implementation, we provided a time synchronization via GPS-locked PPS to both sides (QualiPoc Android and the TWAMP server) and measured one-way latency in both directions. First results in a well-optimized public LTE network show a very asymmetrical result (DL with much shorter latency than UL).

More investigations are being made, particularly in real 5G-based industrial deployments. Stay tuned and, in the meantime, watch the latest on-demand webinar about URLLC - how reliable are private 5G networks today?