Summary of Contents: This paper explores the difficulty of testing embedded software applications that operate in networked environments and distributed configurations. It then describes several innovations in software development tools – including full system simulation, application-aware and automated debugging, and application profiling – that help developers address these challenges.

Embedded systems developers face unique challenges when testing and debugging network devices and distributed applications. Even in the simplest case, building a standalone device that operates in a network environment, thorough testing involves attaching a prototype to a physical network and concocting a wide range of test scenarios to prove correct interaction with the types of systems that the device will encounter once deployed. In a small shop, it might suffice to simply plug the system into the office network. But these days, especially in larger businesses, the IT department won’t let just anything happen on the LAN. And in the early days of development, a hardware prototype may not be available. Testing large scale distributed systems is likely to require a dedicated test network and a full complement of prototype and off-the-shelf equipment, along with the software infrastructure to create and automate useful test cases. This kind of test harness comes only at great cost in time, money and complexity added to the development process.

Fortunately, recent innovations in development tools are beginning to address these challenges. A new generation of testing, debugging and analysis technology makes it considerably easier to build quality into networked and distributed systems with early testing, custom debugging and powerful analysis of software. System simulation techniques offer a way to start testing at the earliest stages of development and create complex network topologies without the need for hardware. Extensible debuggers provide a means to access, manipulate and display target data in a way that is appropriate to the application being tested. And application profiling exposes the runtime behavior of the system by gathering application-specific data and storing it in a database for later analysis. Rather than attempting to provide a one-size-fits-all solution to the problem of testing networked systems, these tools offer facilities and building blocks that embedded systems developers can use to create the ultimate test harnesses for their own applications.

System simulation refers to a group of techniques for developing embedded systems on a workstation, or a network of workstations, without the need for target hardware. This involves behavioral simulation of peripherals, networks and user interfaces to provide a complete virtual environment for the application being developed. While it is possible to create models that simulate hardware at the register transaction level, it is generally effective to simulate peripherals at the device driver API level and networks at the protocol level. A system simulation environment provides off-the-shelf simulation models of standard technologies and facilities for creating application-specific models and automating test scenarios. The simulation environment also provides the infrastructure necessary to control and synchronize multiple applications and application instances running on one or more workstations. An additional benefit of system simulation is the ability to create virtual prototypes of human-machine interfaces attached to the same application logic that is used in production. One can then interact with these virtual devices, either directly or via scripted testing suites, to ensure the correct behavior of the application, as well as the ergonomics and ease of use of the human interface.

Figure 1 shows an example of a mobile phone simulation in the EDGE SimTest environment. The two virtual phones are connected over a simulated GSM network. While the phones initiate and receive calls to and from one another, network protocol messages can be monitored in a data window. Keypad input can be automated using the Python scripting language to create regression testing suites. A third phone or a piece of network equipment might run on a second workstation and interact with the phones depicted here.

Figure 1 - A system simulation environment

The system simulation approach offers major benefits for developing and testing networked and distributed applications. Testing can begin before hardware is available, it can take place anywhere, it can be thoroughly automated, and development hardware costs can be reduced. Complex Internet and wireless topologies can be modeled and tested much earlier and for much lower cost than using a dedicated test network. And human-machine interfaces can be prototyped, validated and changed with dramatically reduced costs and turnaround times compared with physical prototypes.

Extensible debuggers are another innovation that can aid in testing networked systems. Traditional debuggers are best at analyzing software in terms of its source code. Extensions might be available that elucidate operating system data structures, but application-specific information can usually be viewed only as raw data. Extensible debuggers offer the ability to customize the way that target data is handled according to what the data is. This may be as simple as interpreting a data structure and displaying the contents as a human-friendly text description. Or it could mean piping multimedia data to an audio player, a video player or a photo viewer. The important fact to note is that the target data can be manipulated and displayed in a way that is sensible and efficient for the application or the problem at hand.

The key to enabling this kind of application-aware debugging is to expose all of the debugger’s functionality, both on the target and on the host, to the user through a scripting interface. The same technology can be used to create sophisticated hardware initialization routines, to automate debugging with regression testing suites, or even to remotely monitor a system over the web. The capabilities are open-ended.

Figure 2 shows an example of a custom debugging view within the EDGE Debugger. In this case, an Ethernet packet sniffer has been implemented inside the debugger to analyze TCP/IP traffic. A similar approach might be used to graph bandwidth utilization or to validate streamed multimedia content.

Figure 2 An application-aware debugging view

Application profiling is another method of analyzing an application on its own terms, but with the focus on a system’s behavior over time. To avoid confusion, it should be noted that software can be profiled at several different levels. Many developers will first think of source profiling, where a system’s runtime behavior is analyzed in terms of the functions and instructions in its source code. There is also system profiling, in which operating systems events are captured. System profiling is quite useful for performance and memory analysis, as well as analysis of real time logic.

Application profiling is the additional capability to capture and analyze application-specific events. In a networked system, these might be network protocol events like the sending and receipt of particular types of messages. The events are captured at runtime and stored in a database. From there, the data can be graphed, displayed in tabular format, and further manipulated and reported. Depending on the needs of the application, a data set might represent a few minutes or a few days of runtime. One compelling use of application profiling in networked systems would be the benchmarking of network throughput under various conditions to ensure quality of service for streaming voice or media data.

Taken together, these three technologies – system simulation, application-aware debugging and application profiling – point to a new direction in general purpose development tools that are robust and extensible enough to enable sophisticated custom solutions to complex problems in specific application domains. The testing of networked and distributed systems constitutes just one such problem, but it is a particularly good example of the benefits of such an approach. By combining off-the-shelf tools with domain expertise, developers can build the ultimate network test harness more easily and at lower cost than ever before.

by Madison Turner, Technology Analyst, Mentor Graphics Corporation

January 3, 2008

[back to top]

Comments on this article? Send them to comments@fpgajournal.com