Description
The combination of Programmable Logic (PL) and software-programmable CPU cores on FPGAs / SoCs, such as the AMD MPSoC family offers a high potential for parallel processing. This is achieved by programmable logic described in Hardware Description Language (e.g. VHDL) and control applications written for Embedded Linux (Petalinux, Yocto) or bare-metal environments.
Keeping the oversight of the system level complexity can become very challenging.
Taking the right decisions on which functions to deploy in Programmable Logic or implement in software is a complex undertaking for the system architect. Theoretical performance calculations and system-level simulations can be great tools for a reliable development process. However, reality can be sometimes very surprising due to the variety of IP-blocks and parameters involved. Some IP-cores might be even a black-box where the timing or performance is not specified in detail. Typical challenges include inefficient AXI interconnect configurations, excessive memory wait states, polling-based software architectures, unnecessary data transfers between PS and PL, unbalanced HW / SW partitioning, multi-core contention on shared memory resources, and many more.
So, how to deal with intransparent performance bottlenecks?
As a solution approach, targeted system-level profiling of software and hardware in combination gives pragmatic guidance for optimizations. Important tools, among others, are AXI Performance Monitors (APM) for analyzing the data flow inside the PL and PS, as well as software execution tracer tools like Jaeger.
Methodology
Our profiling methodology includes AXI Performance Monitoring in PS and PL, multi-core software profiling under PetaLinux / Yocto, analysis of interrupt and polling behaviour, memory bandwidth and latency measurements, shared-memory and cache analysis, performance visualization using Prometheus and Grafana, and automated analysis and reporting using custom scripting. Particular attention is paid to AXI interconnect structures, memory access patterns and communication overhead between programmable logic and software.
Petalinux / Yocto typically comes with only rudimentary driver support for AXI Performance Monitors (APM) and no user-space application (CLI / GUI). Our Linux driver for the APM implements the necessary adjustments for proper functionality. The user-space application bridges the gap between the driver and the Prometheus metric framework. By using an external Time-Series Database (TSDB), the storage requirements for embedded devices are minimal. The collected data can be filtered and visualized by tools like Grafana. A long retention time is key to visualize trends in performance or utilization over time and to provide early warnings through the devolopment cycle.
Inside PS, multiple APMs are located on strategic positions, the PS-DDR controller alone has six separate APM channels. By monitoring the read and write bandwidths, an estimation of the overall PS RAM bus load can be gained, which helps to detect bottlenecks. Since the overall bandwidth is access-dependent (random vs. sequential), the bandwidth itself does necessarily not show a direct limit. For deeper insight, the latency metric is helpful. An unusually high latency indicates an intermediate overuse of the controller’s performance.
The monitoring through APMs can be complemented with Jaeger as a software execution tracer. This provides the possibility to monitor the execution time of software functions. Together, all this information can be used to optimize a system or to review implementation strategies, like shifting a software function into a hardware-offloading IP-core.
Architecture Overview
Conclusion
System-level profiling provides practical guidance for optimization efforts and helps focus engineering resources on the areas with the highest impact.
Profiling by AXI Performance Monitors can yield pragmatic guidance for optimizations at the right places. Typical optimization results include a reduction of AXI wait cycles, improvements in memory throughput and PL / PS interface communications, lower CPU utilization, reduced interrupt latencies, increased real-time determinism, and lower power consumption.
In many cases we found that AXI interconnect structures can become very complex and can introduce a lot of performance-reducing wait cycles, so they often provide a lot of potential for optimization. On the software side it is worth checking the total amount of accesses to PL hardware. Unnecessary duplicate accesses can often introduce large performance losses and again provide a lot of potential for optimization. Visualization tools such as Grafana provide intuitive before / after comparisons and support data-driven optimization decisions.
Bandwidth, Bus Efficiency, and Latency Visualisation from Grafana