Broadcast h264 video without transcoding from the Logitech C920

The delay was no more than 0.3 seconds, taking into account the fact that the camera is connected to a weak BeagleBone Blue with a processor load of no more than 30% and transferring the video stream via a wifi router to the laptop.

Initially, the task was to transmit video from the robot to a large computer without loading the processor on the robot and without delays. In the process of googling cameras and programs for this task, it was found that there are cameras with hardware video encoding on the camera itself , after which the Logitech C920 camera was acquired. And the tests and experiments began, during which two variants of translation were found without recoding.

All further actions are checked on Ubuntu 16.04, the second option may work in Windows

Broadcast hardware-encoded video from the Logitech C920 camera via cvlc and v4l2

clvc - server video broadcast via vlc from the command line.

Hardware video encoding

Set pixelformat cameras to H264

v4l2-ctl --device=/dev/video1 --set-fmt-video=width=800,height=600,pixelformat=1

Test H264 using vlc so it will not reset the settings:

 cvlc v4l2:///dev/video1 --demux h264

If you try this, vlc will reset the settings before playback:

 cvlc v4l2:///dev/video1:chroma=h264

Playback works. We get the H264 stream from the camera. Now let's transmit over the network via HTTP:

 cvlc v4l2:///dev/video1:chroma=h264:width=800:height=600 --sout '#standard{access=http,mux=ts,dst=localhost:8080,name=stream,mime=video/ts}' -vvv

(on the network you need to use the IP address of your local device instead of localhost)

Run receiver:

 mplayer http://localhost:8080/

So MPEG-TS allows mplayer to connect to the stream at any time with a slight delay (probably waiting for a keyframe or sth).

If you are using mux = avi, mime = video / avi, you need to start mplayer while you are running vlc, otherwise mplayer will not start playing.

Ffmpeg error

If you try ffmpeg instead of vlc for playback, you will probably encounter this problem: ffmpeg.org/trac/ffmpeg/ticket/1387
As soon as you start ffmpeg, it will switch to “Video: rawvideo (YUY2 / 0x32595559)”
While he is taking MJPEG, he will not work with H.264
The same thing happens when using qv4l2: the start of capture in H264 mode actually gives you the YUY2 video window

Playback hardware decoded video

Mplayer is able to play the framebuffer, make sure that X is not working and enter:

 mplayer http://192.168.1.100:8080/ -fs -framedrop -vo fbdev

You can also play in the X window if you want to: run X from the command line:

 Xorg -retro &> /dev/zero

There are a lot of fbdev error messages that we silently reject in / dev / zero

Playing a video stream from a LAN vlc source (see the section above):

 DISPLAY=":0" mplayer http://192.168.1.100:8080/ -fs -framedrop

If we do this, we will notice that we are not using Xvideo or any hardware acceleration yet.
linux-sunxi.org/CedarX seems to provide us with a VPU library, which, unfortunately, is the unprotected thing Allwinner
But there is a hacked VLC that can use it: linux-sunxi.org/VLC
Compiled successfully

Raspberry pi

In case you experience artifacts in the picture, and streams from Raspberry Pi use:

 cvlc v4l2:///dev/video0:chroma=h264 ...

You may need to update the firmware:

 apt-get install rpi-update raspi-config rpi-update reboot

Summary of cvlc and v4l2 broadcast

Tests showed that the processor load on Beaglebone Blue is no more than 30%, which is very good for a weak processor, but it is not clear how to push this video into ROS.

If the wifi bandwidth is not enough, the video delay increases greatly, in theory it can be solved by switching from tcp to udp, the cvlc documentation mentions the possibility of broadcasting by udp.

Broadcast hardware-encoded video from the Logitech C920 camera via Gstreamer

An original developer article supporting this method in Gstreamer.
Install gstreamer on Ubuntu 16.04

 sudo apt-get install libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools sudo apt-get install libgstreamer-plugins-base1.0-dev

Source uvch264src allows you to:

At the same time capture hardware-encoded h264 video stream and video stream finder.
Adjust hardware video encoding (bitrate, keyframes, etc.)

The secondary viewfinder source is an interesting concept. In addition to the main H.264 stream, you get a low-resolution secondary stream in MJPG or YUV format. I'm not sure if this is a feature of the camera, driver, or gstreamer source block, but I hope it will be useful for our Beaglebone cameras, provided the function does not use a processor that is too large to convert between raw and jpg.

Below is an example of a pipeline that displays high-resolution H.264 stream and viewfinder stream:

 gst-launch-1.0 -v -e uvch264src device=/dev/video0 name=src auto-start=true src.vfsrc ! queue ! video/x-raw,format=YUY2,width=320,height=240,framerate=10/1 ! xvimagesink sync=false src.vidsrc ! queue ! video/x-h264,width=1280,height=720,framerate=30/1 ! h264parse ! avdec_h264 ! xvimagesink sync=false

The H.264 primary stream is decoded using the avdec_h264 block, available through the gst-libav package, see notes on the oz9aec wiki . The viewfinder panel can be connected to the fake link if it is not needed, but I do not think that it can be turned off (you can turn off all the same). At least, this is the impression that I read the original KaKaRoTo blog post .

You will probably be overwhelmed when you see a large number of parameters that can be configured for this source block:

 gst-inspect-1.0 uvch264src

List of parameters uvch264src

Factory Details:
Rank none (0)
Long-name UVC H264 Source
Klass Source / Video
Description UVC H264 Encoding camera source
Author Youness Alaoui <youness.alaoui@collabora.co.uk>

Plugin Details:
Name uvch264
Description UVC compliant H264 encoding cameras plugin
Filename /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstuvch264.so
Version 1.8.3
License LGPL
Source module gst-plugins-bad
Source release date 2016-08-19
Binary package GStreamer Bad Plugins (Ubuntu)
Origin URL launchpad.net/distros/ubuntu/+source/gst-plugins-bad1.0

Gobject
+ ---- GInitiallyUnowned
+ ---- GstObject
+ ---- GstElement
+ ---- GstBin
+ ---- GstBaseCameraSrc
+ ---- GstUvcH264Src

Implemented interfaces:
Gstchildproxy

Pad Templates:
SRC template: 'vidsrc'
Availability: Always
Capabilities:
video / x-raw
format: {I420, YV12, YUY2, UYVY, AYUV, RGBx, BGRx, xRGB, xBGR, RGBA, BGRA, ARGB, ABGR, RGB, BGR, Y41B, Y42B, YVYU, Y444, v210, v216, NV12, NV21, NV16 , Others, others, i4, arg64, rg16, bg16, rgb15, uyvp, rg10, rgb, iyv1, arg64, rglp , GBR, GBR_10LE, GBR_10BE, NV12_64Z32, A420_10LE, A420_10BE, A422_10LE, A422_10BE, A444_10LE, A444_10BE}
width: [1, 2147483647]
height: [1, 2147483647]
framerate: [0/1, 2147483647/1]
image / jpeg
width: [1, 2147483647]
height: [1, 2147483647]
framerate: [0/1, 2147483647/1]
video / x-h264
width: [1, 2147483647]
height: [1, 2147483647]
framerate: [0/1, 2147483647/1]
stream-format: {byte-stream, avc}
alignment: au
profile: {high, main, baseline, constrained-baseline}

SRC template: 'imgsrc'
Availability: Always
Capabilities:
Empty

SRC template: 'vfsrc'
Availability: Always
Capabilities:
video / x-raw
format: {I420, YV12, YUY2, UYVY, AYUV, RGBx, BGRx, xRGB, xBGR, RGBA, BGRA, ARGB, ABGR, RGB, BGR, Y41B, Y42B, YVYU, Y444, v210, v216, NV12, NV21, NV16 , Others, others, i4, arg64, rg16, bg16, rgb15, uyvp, rg10, rgb, iyv1, arg64, rglp , GBR, GBR_10LE, GBR_10BE, NV12_64Z32, A420_10LE, A420_10BE, A422_10LE, A422_10BE, A444_10LE, A444_10BE}
width: [1, 2147483647]
height: [1, 2147483647]
framerate: [0/1, 2147483647/1]
image / jpeg
width: [1, 2147483647]
height: [1, 2147483647]
framerate: [0/1, 2147483647/1]

Element Flags:
no flags set

Bin Flags:
no flags set

Element Implementation:
Has change_state () function: 0x7ff438f22ba0

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
SRC: 'vfsrc'
Src: 'imgsrc'
SRC: 'vidsrc'

Element Properties:
name: The name of the object
flags: readable, writable
String. Default: "uvch264src0"
parent: The parent of the object
flags: readable, writable
Object of type "GstObject"
async-handling: Asynchronous state changes
flags: readable, writable
Boolean. Default: false
message-forward: Forwards all children messages
flags: readable, writable
Boolean. Default: false
mode: The capture mode (still image capture or video recording)
flags: readable, writable
Enum "GstCameraBin2Mode" Default: 2, "mode-video"
(1): mode-image - Still image capture (default)
(2): mode-video - Video recording
zoom: Digital zoom factor (eg 1.5 means 1.5x)
flags: readable, writable
Float. Range: 1 - 3.402823e + 38 Default: 1
max-zoom: Digital zoom factor (eg 1.5 means 1.5x)
flags: readable
Float. Range: 1 - 3.402823e + 38 Default: 10
ready-for-capture: Informs this element is ready for starting another capture
flags: readable
Boolean. Default: true
post-previews: If capture preview images should be posted to the bus
flags: readable, writable
Boolean. Default: true
preview-caps: (NULL means ANY)
flags: readable, writable
ANY

preview-filter: A custom preview filter to process preview image data
flags: readable, writable
Object of type "GstElement"
auto-start
flags: readable, writable
Boolean. Default: false
colorspace-name: The name of the colorspace element
flags: read, write, can be equal only to the state NULL or READY
String. Default: "videoconvert"
jpeg-decoder-name: The jpeg decoder element
flags: read, write, can be equal only to the state NULL or READY
String. Default: “jpegdec”
for the PTS synchronization (-1 = unlimited)
flags: read, write, can be NULL, READY, PAUSED or PLAYING
Integer. Range: 0 - 2147483647 Default: 0
num-buffers: Number of buffers to output before sending EOS (-1 = unlimited)
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
device: Device location
flags: readable, writable
String. Default: "/ dev / video0"
device-name: Name of the device
flags: readable
String. Default: ""
initial bitrate: Initial bitrate in bits / second (static control)
flags: read, write, can be equal only to the state NULL or READY
Unsigned Integer. Range: 0 - 4294967295 Default: 3,000,000
slice-units: slice units (static control)
flags: read, write, can be equal only to the state NULL or READY
Unsigned Integer. Range: 0 - 65535 Default: 4
slice-mode: slice-units property (static control)
flags: read, write, can be equal only to the state NULL or READY
Enum "UvcH264SliceMode" Default: 3, "slice / frame"
(0): ignored - Ignored
(1): bits / slice - Bits per slice
(2): MBs / slice - MBs per Slice
(3): slice / frame - Slice Per Frame
iframe-period: Time between IDR frames in milliseconds (static control)
flags: read, write, can be equal only to the state NULL or READY
Unsigned Integer. Range: 0 - 65535 Default: 10000
usage-type: The usage type (static control)
flags: read, write, can be equal only to the state NULL or READY
Enum "UvcH264UsageType" Default: 1, "realtime"
(1): realtime - Realtime (video conferencing)
(2): broadcast - Broadcast
(3): storage - Storage
(4): ucconfig0 - UCConfig 0
(5): ucconfig1 - UCConfig 1
(6): ucconfig2q - UCConfig 2Q
(7): ucconfig2s - UCConfig 2S
(8): ucconfig3 - UCConfig 3
entropy: Entropy (static control)
flags: read, write, can be equal only to the state NULL or READY
Enum "UvcH264 Entropy" Default: 0, "cavlc"
(0): cavlc - CAVLC
(1): cabac - CABAC
enable-sei: Enable SEI picture timing (static control)
flags: read, write, can be equal only to the state NULL or READY
Boolean. Default: false
num-reorder-frames: Number of B frames between the references frames (static control)
flags: read, write, can be equal only to the state NULL or READY
Unsigned Integer. Range: 0 - 255 Default: 0
preview-flipped: Horizontal flipped image for non H.264 streams (static control)
flags: read, write, can be equal only to the state NULL or READY
Boolean. Default: false
leaky-bucket-size: Size of the leaky bucket size in milliseconds (static control)
flags: read, write, can be equal only to the state NULL or READY
Unsigned Integer. Range: 0 - 65535 Default: 1000
rate-control: Rate control mode (static & dynamic control)
flags: read, write, can be NULL, READY, PAUSED or PLAYING
Enum "UvcH264RateControl" Default: 1, "cbr"
(1): cbr - Constant bit rate
(2): vbr - Variable bit rate
(3): qp - Constant QP
fixed-framerate: Fixed framerate (static & dynamic control)
flags: read, write, can be NULL, READY, PAUSED or PLAYING
Boolean. Default: false
max-mbps for the maximum processing rate
flags: readable
Unsigned Integer. Range: 0 - 4294967295 Default: 0
level-idc: Level IDC (dynamic control)
flags: read, write, can be NULL, READY, PAUSED or PLAYING
Unsigned Integer. Range: 0 - 255 Default: 40
peak-bitrate: dynamic control
flags: read, write, can be NULL, READY, PAUSED or PLAYING
Unsigned Integer. Range: 0 - 4294967295 Default: 3,000,000
average-bitrate: dynamic control
flags: read, write, can be NULL, READY, PAUSED or PLAYING
Unsigned Integer. Range: 0 - 4294967295 Default: 3,000,000
min-iframe-qp: The minimum quantization step for I frames (dynamic control)
flags: read, write, can be NULL, READY, PAUSED or PLAYING
Integer. Range: -127 - 127 Default: 10
max-iframe-qp: The minimum quantization step for I frames (dynamic control)
flags: read, write, can be NULL, READY, PAUSED or PLAYING
Integer. Range: -127 - 127 Default: 46
min-pframe-qp: The minimum quantization step size for frames (dynamic control)
flags: read, write, can be NULL, READY, PAUSED or PLAYING
Integer. Range: -127 - 127 Default: 10
max-pframe-qp: The minimum Quantization step for frames (dynamic control)
flags: read, write, can be NULL, READY, PAUSED or PLAYING
Integer. Range: -127 - 127 Default: 46
min-bframe-qp: The minimum quantization step size for frames (dynamic control)
flags: read, write, can be NULL, READY, PAUSED or PLAYING
Integer. Range: -127 - 127 Default: 10
max-bframe-qp: The minimum quantization step size for frames (dynamic control)
flags: read, write, can be NULL, READY, PAUSED or PLAYING
Integer. Range: -127 - 127 Default: 46
ltr-buffer-size: Total number of Long-Term Reference frames (dynamic control)
flags: read, write, can be NULL, READY, PAUSED or PLAYING
Integer. Range: 0 - 255 Default: 0
ltr-encoder-control: Number of LTR frames
flags: read, write, can be NULL, READY, PAUSED or PLAYING
Integer. Range: 0 - 255 Default: 0

Element Actions:
"Get-enum-setting": gboolean user_function (GstElement * object,
gchararray arg0
gpointer arg1,
gpointer arg2);
"Get-boolean-setting": gboolean user_function (GstElement * object,
gchararray arg0
gpointer arg1,
gpointer arg2);
"Get-int-setting": gboolean user_function (GstElement * object,
gchararray arg0
gpointer arg1,
gpointer arg2,
gpointer arg3);
"Start-capture": void user_function (GstElement * object);
“Stop capture”: void user_function (GstElement * object);

Many of these parameters can be used to configure parameters in H.264 compression. I find it quite surprising, given that the compression is performed by a chip inside the camera. The following example increases the default data transfer rate from 3 to 5 Mbps and reduces the default keyframe interval from 10 to 3 seconds:

 gst-launch-1.0 -v -e uvch264src initial-bitrate=5000000 average-bitrate=5000000 iframe-period=3000 device=/dev/video0 name=src auto-start=true src.vfsrc ! queue ! video/x-raw,format=YUY2,width=320,height=240,framerate=10/1 ! xvimagesink sync=false src.vidsrc ! queue ! video/x-h264,width=1280,height=720,framerate=30/1 ! h264parse ! avdec_h264 ! xvimagesink sync=false

On the other hand, there are no parameters for adjusting the usual webcam parameters, such as contrast, brightness, focus, etc. Thus, to configure them, we still need external tools, such as v4l2.ctl, or the variant mentioned in next section.

v4l2src

It looks like we can also use the good old v4l2src to capture H.264 encoded streams from Logitech C920:

 gst-launch-1.0 -v -e v4l2src device=/dev/video1 ! queue ! video/x-h264,width=1280,height=720,framerate=30/1 ! h264parse ! avdec_h264 ! xvimagesink sync=false

This is probably due to the evolution of both gstreamer and V4L2. This option looks simpler if you do not need to change the H.264 compression settings, and this gives us access to camera settings, such as brightness and contrast. This also has the advantage of being in a good package of plug-ins, which may be better preserved in the future.

Broadcast via Gstreamer

On the receiving computer, launch the receiver:

 gst-launch-1.0 -v udpsrc port=6666 ! application/x-rtp, encoding-name=H264 ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! autovideosink

On the device with the camera, we launch the transfer of hardware-encoded video:

 gst-launch-1.0 uvch264src name=src auto-start=true src.vidsrc ! video/x-h264,width=160,height=120,framerate=30/1 ! h264parse ! rtph264pay ! udpsink host=192.168.1.196 port=6666

For help in writing this code, a special thanks to Amomum

Summary of delayed video streaming via Gstreamer

Even with a resolution of 1920 * 1080, the processor load on Beaglebone Blue did not exceed 30%, and the video delay was no more than 0.3 seconds. Yes, since the video in this version was broadcast on udp, then with the loss of packets, the picture falls apart to the next keyframe, but we can configure it to send keyframes more often.

ROS tasks

It remains now only to stuff the received video in ROS, if anyone can help with this, write in a personal.

Source: https://habr.com/ru/post/415567/

All Articles