Backpressure
Designing reliable parallel systems
with shared resources
Are you experiencing rare errors in your designs where multiple video streaming pipelines compete about access to a shared resource such as DDR-RAM memory or a high-speed serial link? You participated in different training courses, asked colleagues, checked out expert communities, but still, there remains this unreliable behavior that is so hard to track down?
You are not alone: Have you ever heard about the notion of backpressure?
Figure 1: Parallel video & data processing pipelines experiencing backpressure
The Root Cause: Explaining the Backpressure Trigger
In most cases where multiple parallel pipelines are processing video or other streaming data in parallel, there are certain resources in the design which have to be shared by the parallel pipelines. For instance, a DDR5-RAM memory controller or a PCIe high-speed serial interface can be such a shared resource. Since the resource is shared, it may not be able to serve any of the competing streaming pipelines at any instant in time long enough. Instead, the shared resource may need to signal to a requesting pipeline that it has to wait until a minimum amount of streaming data of a currently serviced pipeline has been processed. Such signaling to wait is applying backpressure to that pipeline that has to wait. Waiting means that intermediate buffering of data along the pipeline must be provided, which can e.g. be achieved by inserting FIFO-type buffers between IP blocks along the pipeline. From a system architecture point of view, designing a proper architecture that can handle backpressure for 100% of situations is a quite complex task. Fortunately, there are at least partially analytical methods to design the intermediate FIFO buffer sizes, and there are well established interfaces focusing on simple handshaking through a minimum set of signals.
AXI4-Stream
A prominent example is the AXI4-Stream interface with the TVALID, TREADY and TLAST signals. When designing or integrating IP blocks with AXI4-S interfaces the expert designer still has to provide proper handling of these signals under all circumstances. Experience with a multitude of video processing pipeline designs and associated IP blocks shows that even though experts have designed the interface handling, backpressure can lead to rare logical errors that let a pipeline stall and get stuck forever or at least until it is reset.
How We Handle Our Backpressure Challenges:
1
Expert design
A practical approach is to start with expert design.
2
Expert code reviews
It is also recommended having an expert code review while planning the design (1).
3
Deterministic test cases in simulation
Various backpressure scenarios are applied as test cases.
4
Random test cases in simulation
Various backpressure scenarios are applied as test cases.
5
Reset considerations
As a special but extremely important use case, proper reseting of pipeline subsystems to initial states should be considered as step 5. This needs to be reflected in design and respective test cases and is usually not at all trivial. A systematic top-level approach to stop further input and to gracefully flush pipelines and intermediate buffers may be one practical approach among several valid scenarios.
6
Subsystem or system integration
In a further step 6. subsystem or system integration is applied on the FPGA embedded target.
In case of rare errors, ILAs can be applied to trigger and log the situation, analyze it and then simulation can be augmented by additional test cases. Regression test-benches result from this process of steady improvement. It should be mentioned that moving to step 6. too early may not be valid for ASIC designs. In this case the effort for the steps 1. through 4. has to be considerably higher than for an FPGA design where a new image can normally be generated within a few hours or days after testing on the embedded target.
