Low-Latency Vision Pipelines
Efficient camera-to-display and AI pipelines with Embedded Linux
Description
For certain vision-based applications not only resolutions and frames per second (FPS) are demanding, but also very low latencies in the video processing pipelines. Examples are control loops for unmanned aerial vehicles (UAVs), augmented reality glasses with cameras and displays, and many more. PLC2 has gained a lot of experience in multiple development service projects involving demanding video processing pipelines.
Methodology
As explained in our use case »Multi-Input Recording« Automotive applications have pushed the simplification of camera interfaces. Nowadays, 2-wire cables that can handle DC power supply, video data transfer, configuration channels, and even discrete GPIO signals through clever modulation techniques are widely used.
SerDes chips convert from camera signals to MIPI-CSI and an FPGA first converts from MIPI to an internal protocol such as AXI-Stream. When vision-enhancing algorithms, such as contrast improvement, are applied to displays integrated into glasses, very low latency is crucial to ensure that users do not experience disturbing delays.
Images are typically stored in local DDR-RAM to decouple input from output timing. Under certain timing constraints and by implementing the decoupling in a clever fashion, the overall latency can be reduced to far less than one frame period. Data can then be processed in video pipelines under the control of Embedded Linux and output with low latency on displays, e.g. for the purpose of enhancing contrast or for augmenting the camera images with artificial guidance.
The image depicts an augmented reality use case where only a latency that is perceivable by the human user is acceptable, i.e. a few milliseconds.
Furthermore, if incoming video data is to be used as input in control loops including AI-applications such as classification, recognition, and tracking, then low latency of the overall video processing pipeline is absolutely crucial.
Conclusion / Summary
PLC2 has gained profound experience with vision-based applications including Embedded Linux GStreamer and V4L video processing pipelines to achieve very low latencies from cameras to displays in augmented reality glasses or in control loops utilizing AI.
