Current video stream latency and a way to reduce it
Fig.1 Live stream latency testing
Recently we had an inquiry whether our cameras are capable of streaming low latency video. The short answer is yes, the camera’s average output latency for 1080p at 30 fps is ~16 ms. It is possible to reduce it to almost 0.5 ms with a few changes to the driver.
However the total latency of the system, from capturing to displaying, includes delays caused by network, pc, software and display.
In the results of the experiment (similar to this one) these delays contribute the most (around 40-50 ms) to the stream latency – at least, for the given equipment.
Goal
Measure the total latency of a live stream over network from 10393 camera.
Setup
- Camera: NC393-F-CS
- Resolution@fps: 1080p@30fps, 720p@60fps
- Compression quality: 90%
- Exposure time: 1.7 ms
- Stream formats: mjpeg, rtsp
- Sensor: MT9P001, 5MPx, 1/2.5″
- Lens: Computar f=5mm, f/1.4, 1/2″
- PC: Shuttle box, i7, 16GB RAM, GeForce GTX 560 Ti
- Display: ASUS VS24A, 60Hz (=16.7ms), 5ms gtg
- OS: Kubuntu 16.04
- Network connection: 1Gbps, direct camera-PC via cable
- Applications:
- gstreamer
- chrome, firefox
- mplayer
- vlc
- Stopwatch: basic javascript
Notes
Table 1: Transfer times and data rate
| Resolution/fps | Image size1, KB | Transfer time2, ms | Data rate3, Mbps |
|---|---|---|---|
| 720p/60 | 250 | 2 | 120 |
| 1080p/30 | 500 | 4 | 120 |
1 – average compressed (90%) image size
2 – time it takes to transfer a single image over network. Jitter is unknown. t = Image_size*1Gbps
3 – required bandwidth: rate = fps*Image_size
Camera output latency calculation
All numbers are for the given lens, sensor and camera setup and parameters. Briefly.
Sensor
Because of ERS each row’s latency is different. See tables 2 and 3.
Table 2: tROW and tTR
| Resolution | tROW1, us | tTR2, us |
|---|---|---|
| 720p | 22.75 | 13.33 |
| 1080p | 29.42 | 20 |
| full res (2592×1936) | 36.38 | 27 |
1 – row time, see datasheet. tROW = f(Width)
2 – time it takes to transfer a row over sensor cable, clock = 96MHz. tTR = Width/96MHz
Table 3: Average latency and the whole range.
| Resolution | tERS avg1, ms | tERS whole range2, ms |
|---|---|---|
| 720p | 8 | 0.01-16 |
| 1080p | 16 | 0.02-32 |
1 – average latency
2 – min – last row latency, max – 1st row latency
Exposure
tEXP < 1 ms – typical exposure time for outdoors. A display is bright enough to set 1.7 ms with the gains maxed.
Compressor
The compressor is implemented in fpga and works 3x times faster but needs a stripe of 20 rows in memory. Thus, the compressor will finish ~20/3*tROW after the whole image is read out.
tCMP = 20/3*tROW
Summary
tCAM = tERS + tEXP + tCMP
Since the image is read and compressed by fpga logic of the Zynq and this pipeline has been simulated we can be sure in these numbers.
Table 4: Average output latency + exposure
| Resolution | tCAM, ms |
|---|---|
| 720p | 9.9 |
| 1080p | 17.9 |
Stopwatch accuracy
Not accurate. For simplicity, we will rely on the camera’s internal clock that time stamps every image, and take the javascript timer readings as unique labels, thus not caring what time they are showing.
Results
 Fig.2 1080p 30fps |
 Fig.3 720p 60fps |
GStreamer has shown the best results among the tested programs.
Since the camera fps is discrete the result is a multiple of 1/fps (see this article):
- 30 fps => 33.3 ms
- 60 fps => 16.7 ms
| Resolution/fps | Total Latency, ms | Network+PC+SW latency, ms |
|---|---|---|
| 720p@60fps | 33.3-50 | 23.4-40.1 |
| 1080p@30fps | 33.3-66.7 | 15.4-48.8 |
Possible improvements
Camera
Currently, the driver waits for the interrupt from the compressor that indicates the image is fully compressed and ready for transfer. Meanwhile one does not have to wait for the whole image but start the transfer when the minimum of the compressed is data ready.
There are 3 more interrupts related to the image pipeline events. One of them is “compression started” – switching to it can reduce the output latency to (10+20/3)*tROW or 0.4 ms for 720p and 0.5 ms for 1080p.
Other hardware and software
In addition to the most obvious improvements:
- For wifi: use 5GHz over 2.4GHz – smaller jitter, non-overlapping channels
- Lower latency software: for mjpeg use gstreamer or vlc (takes an extra effort to setup) over chrome or firefox because they do extra buffering
Links
- Latency in live network video surveillance
- Wifi latencies, 2.4GHz & 5GHz
- This video compares different displays.
- About ERS
Updates
Table 6: Camera ports
| mjpeg | rtsp | |
|---|---|---|
| port 0 | 2323 | 554 |
| port 1 | 2324 | 556 |
| port 2 | 2325 | 558 |
| port 3 | 2326 | 560 |
GStreamer pipelines
- For mjpeg:
~$ gst-launch-1.0 souphttpsrc is-live=true location=http://192.168.0.9:2323/mimg ! jpegdec ! xvimagesink
- For rtsp:
~$ gst-launch-1.0 rtspsrc is-live=true location=rtsp://192.168.0.9:554 ! rtpjpegdepay ! jpegdec ! xvimagesink
VLC
~$ vlc rtsp://192.168.0.9:554
Chrome/Firefox
Open http://192.168.0.9:2323/mimg
Leave a Reply