Implementing Software Testing for a Robotics Project

The MFSP application is a robotics application developed using the ROS2 framework, where several ROS2 packages work together with various other technologies and frameworks to form a digital twin simulation, where a selection of different types of robots can be spawned inside a simulated world and their properties can be examined in varying environments. 

Introduction 

As I am taking my final steps towards graduating with a Bachelor’s in Information and Communication Technology, I had the opportunity to write my thesis on the practical application of unit and integration testing for the project at hand. In this blog post I will be describing the test suite implementation, while shedding light on the intricacies of establishing a reliable test environment in a challenging robotics setting. 

Motivation for Comprehensive Testing 

As software driven systems are becoming increasingly present in everyday society, the importance of software testing has risen correspondingly. In the world of Robotics, a comprehensive test suite is a vital component of a modern, secure, and reliable application. Ensuring that all components of a system perform according to their specifications lays the foundation for a robust and dependable software product. Building on top of this concept, it is also crucial that the individual components work in harmony with each other. 

Additionally, while the ROS2 framework is generally well documented with the core concepts clearly explained alongside easy-to-follow examples, the documentation is very light when it comes to testing. This sparked an interest in conducting research into the available tools and methods for applying a test suite for our project. 

Testing a ROS2 application 

In a ROS2 application, the bulk of the source code is found in the implementation of the Nodes. As Nodes are the central computational units, their tasks should be well defined, and they should each have a specific responsibility within the system. While the individual Nodes are responsible for their own tasks, completing and executing these tasks typically involves communicating with other Nodes present in the system. Think of a Node whose job is to navigate a robot to a given set of co-ordinates. At the very least, completing this task would require communicating with a Node responsible for turning and spinning the robot's wheels. Assuming this task must be completed in an environment with obstacles, the involvement of a Node responsible for sensing the environment is also introduced. This simplified case example shows that while each of the Nodes presented have their own responsibilities, fulfilling them requires communication with other Nodes as well. 

Implementing Unit Tests 

As a first step in asserting the expected behavior of a ROS2 application, it is crucial to ensure that the individual Nodes are implemented correctly. This is where Unit Testing comes into play. In addition to verifying that the various methods that define the behavior of the Nodes are working properly, it is also essential to validate the existence and correct configuration of the communication interfaces, which are paramount for enabling data exchange within the system. The operation of every Node within the application was assessed through individual unit testing, where each unit test consisted of an array of test cases evaluating a specific property of the given Node. As the focus of the unit tests was to explicitly verify the ROS2 specific operations of the Nodes, the inherit complexities of a robotics application were significantly reduced with the use of mocking, a process of replacing a given functionality with simulated behavior. This approach not only allows for isolated testing of individual components without the need for a fully functional environment, but it also streamlines the testing process and enhances the reliability of the test results by eliminating dependencies on external factors. 

As a result of the implementation of the unit tests, some minor defects in the ROS2 source code were located and refactored. Having verified that the unit-tested components are now working as expected, the next natural step was to proceed to integration testing.  

Implementing Integration Tests 

As the focus shifts from evaluating the individual nodes to evaluating the integration of these nodes, the complexity of the testing environment increases. To test the integration of the nodes, a separate executable file was required for each active node within the test. These files were responsible for evaluating the availability of the communication interfaces invoked during the node-to-node communications, in addition to verifying that the data exchanged by the nodes does not get corrupted in transmission. At this point, the ROS2 framework “launch_testing” was added to the tool case. This framework allows launching multiple processes simultaneously, while monitoring their execution and exit codes. Additionally, the ”launch_testing” framework adds the possibility of running tests both concurrently with the executing processes, as well as after the processes have terminated. This allowed for versatility in the test case design. 

In addition to requiring the execution of multiple node processes simultaneously, the previous approach of mocking certain functionalities could not be applied here. Namely, the unit tests used mocking to a great extent in replacing the counterparts for various communication channels. While this is a perfectly valid strategy in unit testing, in the integration tests the goal was to verify that the nodes can successfully communicate directly with each other.  

With both the unit and integration tests in place, the application now has an added layer of reliability when ensuring the system's inner operations. 

Running the tests 

Since the unit tests are responsible for evaluating the execution of individual nodes in an isolated environment, the tests are located within their respective packages. This means that each tested package in the MFSP application contains a ”test/” directory where the unit tests reside, making it straightforward to find and manage tests specific to that package and its Nodes. This approach allows for immediate consideration of any changes made to the package in the test cases, while also ensuring that modifications in other packages do not impact the outcomes of the tests. 

The integration tests, on the other hand, occur in an environment where multiple packages are invoked during testing. Therefore, a separate package was constructed for storing these tests. This dedicated integration test package contains all the necessary configuration files, test files and launch files required to set up and run the integration tests. Isolating the integration tests in their own package further enhances the separation of the different testing levels, simplifying both future development and maintenance. 

To streamline the testing process across multiple packages, we leverage the ”colcon” tool, which facilitates both the building and the testing of a ROS2 application. The ”colcon test” command, in particular, allows us to run all the tests from every package with a single command. This simplifies the testing workflow by automatically discovering and executing each test within the application.  

Conclusion 

The addition of the tests described in this blog post further enhanced the quality of the developed application by providing reliable metrics about software performance. The developed test suite lays a solid foundation for future higher-level testing, as the expertise gathered during the test development process can be expanded on. 

Developing a comprehensive test suite for a ROS2 application requires a profound understanding of the robotic environment, as well as technical skills to effectively utilize the testing tools available. The testing of the MFSP application is still in its initial stages, with many future additions already planned. However, we are off to a great start with what has already been implemented, setting the stage for ongoing improvements and further development. 

For further insights and information, don't hesitate to reach out to us!

Our services: