Sunxi-Cedrus

Sunxi-Cedrus is an effort to bring hardware-accelerated video decoding and encoding support for Allwinner SoCs to the mainline Linux kernel. Additional userspace components that interface with the kernel driver are also provided, for typical GNU/Linux-based systems.

Components
Support for Sunxi-Cedrus is implemented through various components, either in kernel space or userspace:
 * the Sunxi-Cedrus V4L2 M2M kernel driver
 * the libVA-Cedrus VAAPI backend

Additional userspace components are also available, for development purposes:
 * the cedrus-frame-test standalone tool, that allows testing the Sunxi-Cedrus VPU driver
 * the sun4i-specific libdrm, with support for allocating buffers in the MB32-tiled NV12 format used by the VPU
 * the dump libVA backend, that allows dumping metadata and slices from videos

Video players that support libVA should be compatible with the Cedrus libVA backend. However, details in implementations might result in incompatibility with certain players. See Player Support for the status of tested players.

Codec Support
Support status for specific codecs in the Sunxi-Cedrus libVA backend is presented in the following table:

SoC Support
Support for specific SoCs in the V4L2 M2M kernel driver is presented in the following table:

Player Support
The following players were tested with the Sunxi-Cedrus libVA backend:

Linux
The first step to installing Sunxi-Cedrus support is to build a Linux kernel with the latest patch series for the driver. A tree with all the required patches is available at:
 * Repository: https://github.com/free-electrons/linux-cedrus
 * Branch: cedrus/4.17/merge

The following kernel configuration options must be selected: CONFIG_MEDIA_SUPPORT CONFIG_MEDIA_CONTROLLER CONFIG_V4L_MEM2MEM_DRIVERS CONFIG_VIDEO_SUNXI_CEDRUS

Details about building a mainline Linux kernel are available on the Mainline Kernel Howto page.

Note that the associated kernel headers must be installed to the target device for the userspace components to build.

libva-cedrus
The main userspace component that supports the Sunxi-Cedrus VPU driver is the libva-cedrus VAAPI backend. It is available at:
 * Repository: https://github.com/free-electrons/libva-cedrus
 * Branch: master

The backend can be built and installed on the target device by following these steps: git clone https://github.com/free-electrons/libva-cedrus cd libva-cedrus ./autogen.sh && make && sudo make install

libdrm-sun4i
Building and installing 'libdrm-sun4i is only required when using the cedrus-frame-test tool. It imports the updated sun4i-drm Linux kernel driver definitions for installation in userspace and is available at:
 * Repository: https://github.com/free-electrons/libdrm-sun4i.git
 * Branch: master

It can be built and installed on the target device through the following steps: git clone https://github.com/free-electrons/libdrm-sun4i cd libdrm-sun4i ./autogen.sh --prefix=/usr && make && sudo make install

Alternatively to building libdrm-sun4i, the  file can be placed into   directly.

cedrus-frame-test
The cedrus-frame-test tool is available at:
 * Repository: https://github.com/free-electrons/cedrus-frame-test.git
 * Branch: master

It can be built on the target device through the following steps: git clone https://github.com/free-electrons/cedrus-frame-test.git cd cedrus-frame-test make

libva-dump
The libva-dump VAAPI backend is available at:
 * Repository: https://github.com/free-electrons/libva-dump.git
 * Branch: master

It can be built on the host or the target device through the following steps: git clone https://github.com/free-electrons/libva-dump.git cd libva-dump ./autogen.sh && make && sudo make install

VLC
In order to use VLC with Sunxi-Cedrus, the libva-cedrus VAAPI backend must be installed on the system. VLC must also be configured to use VAAPI for video decoding, through the following menus:

While OpenGL video output might be supported (with GPU support installed), it was not tested at this point.

Although VLC has been configured to use VAAPI for video decoding, libVA must be instructed to use libva-cedrus through the  environment variable : export LIBVA_DRIVER_NAME=sunxi_cedrus vlc path/to/video.mpeg

Examples
Here are some examples of VLC usage for specific use cases: export LIBVA_DRIVER_NAME=sunxi_cedrus export DISPLAY=:0 vlc --no-audio --no-autoscale --no-osd --play-and-exit path/to/video.mpeg
 * Remote testing without audio, scaling and OSD:

cedrus-frame-test
cedrus-frame-test is preferably used from its repository directory: the default path to the video slices is relative from the root of the repository. Since the tool uses the DRM KMS device directly, no graphical session (not even a login manager) must be running concurrently.

The tool currently contains a single preset with slices that allow decoding the first 25 frames of the sample MPEG2 Big Buck Bunny video. More presets can be added from the data dumped with libva-dump. cedrus-frame-test usage is displayed when the tool is called with the  argument.

Examples
Examples of cedrus-frame-test use include: ./cedrus-frame-test -f 25 -l ./cedrus-frame-test -ql ./cedrus-frame-test -v /dev/video1 -m /dev/media1
 * Decoding frames at 25 fps in a loop, with information:
 * Decoding frames as fast as possible in a loop:
 * Specifying the right nodes when another V4L2 driver is loaded:

Current Development
The development of both the Sunxi-Cedrus V4L2 kernel driver and the Cedrus libVA backend is currently work in progress by:
 * Paul Kocialkowski, intern at Bootlin
 * Maxime Ripard, engineer at Bootlin

This effort was funded by the related crowdfunding started by Bootlin in February 2018.

Weekly Reports
Reports on the advancement of the development of Sunxi-Cedrus are posted regularly on the Bootlin Blog, starting week 10 of 2018:
 * Week 16 Status Update
 * Week 15 Status Update
 * Week 14 Status Update
 * Week 13 Status Update
 * Week 12 Status Update
 * Week 11 Status Update
 * Week 10 Status Update

Past Development
Initial work on the Sunxi-Cedrus V4l2 kernel driver and libVA backend was carried out by Florent Revest (kido) during an internship at Bootlin (formerly Free Electrons), resulting in a proof-of-concept driver and associated backend with MPEG2 and partial MPEG4 support for the A13 SoC.

If you want to try it, you need a board with an A13 SoC, this page will try to guide you from your first steps with the driver to the details of its implementation. Make sure you've checked the known bugs and limitations before using it.

Installation
This procedure has been tested on the NextThingCo's CHIP board but should be adaptable to other A13 boards supported by the 4.8 mainline kernel. The first thing you need to do is to recompile a kernel with the sunxi-cedrus v4l driver and the corresponding device tree entry. You can follow the Mainline Kernel Howto but using the following repository:

https://github.com/FlorentRevest/linux-sunxi-cedrus

Use menuconfig to enable the VPU driver in Device Drivers -> Multimedia support -> Memory-to-memory multimedia devices -> Sunxi CEDRUS VPU driver. The driver should be compiled into the kernel and not as a module.

Your kernel will be located in arch/arm/boot/zImage and device tree in arch/arm/boot/dts/sun5i-r8-chip.dtb Don't forget to install kernel headers to your distribution if you want to be able to compile sunxi-cedrus-drv-video.

On a standard debian jessie system you will need to install the following build dependencies:

apt install git autoconf automake libtool pkg-config gcc libdrm-dev libva-dev libx11-dev make g++ vlc xorg

You will be able to compile and install a newer version of libVA supporting the LIBVA_DRIVER_NAME environment variable and then the sunxi-cedrus libVA backend:

git clone https://github.com/01org/libva cd libva git checkout 695f99ef0405cf4255e7767b44effb0da2fe706e ./autogen.sh --prefix=/usr --libdir=/usr/lib/arm-linux-gnueabihf/ make sudo make install

git clone https://github.com/FlorentRevest/sunxi-cedrus-drv-video cd sunxi-cedrus-drv-video ./autogen.sh make # DRM_CFLAGS=-I/path/to/your/linux/headers sudo make install

Once installed, make sure you've exported the environment variable telling VA to use the sunxi_cedrus backend, activate the VA X11 decoding in VLC settings (Tools -> Preferences -> Input / Codecs -> set 'Hardware-accelerated decoding' to 'VA-API video decoder via X11') and then run one of the sample media file you can find here and here

xinit& export DISPLAY=:0.0 export LIBVA_DRIVER_NAME=sunxi_cedrus vlc big_buck_bunny_480p_MPEG2_MP2_25fps_1800K.MPG vlc ducks_take_off_420_720p25.mp4

Supported features

 * MPEG2 Decoding
 * Partial MPEG4 Decoding

Known bugs and limitations

 * H264 and H265 are not supported (some of the underlying problems include: allocating 19 surfaces at once and queueing several slices by frames)
 * MPEG4 decoding has some glitches. When something moves in a video it usualy draws some kind of trace behind it, this behavior is believed to come from an inconsistency in movement prediction. (VLC also tries to SyncSurface more often than needed which results in error when dequeuing the capture buffers)
 * No encoding: this can be added later on but hasn't been tried yet
 * Direct rendering: currently, buffers coming out of the v4l driver in a tiled pixel format are converted to a standard YUV pixel format and then rendered on screen by ffmpeg/vlc. As soon as the support for YUV DRM planes will be added to the kernel, this behavior can be replaced and the performances will be much better.
 * Currently the video can only be played at a zoom of 1:1, otherwise the scaling is done by ffmpeg in full CPU and it is too slow. Having that in the DRM driver would also allow for hardware accelerated frames scaling.
 * We currently can't play a MPEG2 file and then a MPEG4 file (or vice versa) or the output will be full of garbage pixels. This is probably due to some registers of the MPEG engine being kept between the two decoding and "polluting" the MPEG4 decoding with older values from MPEG2 decoding. We should find a way to clear those registers when receiving a S_FMT ioctl in the kernel driver or when closing the video device.

Technical details and implementation
The Cedrus project has provided reverse engineering of the Allwinner's proprietary CedarX blob for a couple of years. This work has been done on the Allwinner's 3.4 kernel and led to the creation of a libVDPAU backend interfacing with the "cedar_dev" and "disp" kernel drivers available in the vendor's kernel. The "cedar_dev" kernel driver directly mapped registers and memory from the VE to the userspace and could potentially be a security risk. Those two drivers couldn't be upstreamed because they don't use any standard API or framework.

In order to use the VE on a mainline kernel, a new proper kernel driver had to be written from scratch with mainlinable methods in mind. The correct way to implement a codec device is to use the "video4linux" framework, referenced as v4l2. v4l2 handles many kind of video devices, some of them are cameras and sends data to a CAPTURE queue, others are screens and use data from an OUTPUT queue. Codec devices require a flow of data from OUTPUT to CAPTURE, they are "memory-to-memory" devices.

Until now, most of the codec devices were able to handle raw bitstream via a firmware, which means that the OUTPUT queue (containing the compressed input data) could directly contain a MPEG file and the CAPTURE queue was filled with video frames. But the inner working of Allwinner's VPU is different, indeed it requires prior bitstream parsing into smaller frames/slices alongside headers' data. This parsing can not be done in the kernel side since it would be a complex codebase to maintain so it requires a new user-space usecase, hence a new API.

The "Frame API" described here has been designed for the Rockchip's VP8 decoding support and is implemented here and here The "Frame API" aims to standardize the way VPU drivers should communicate frame by frame with the userspace. The advocated method is to bind "buffers" containing slice data from the OUTPUT queue to "extended controls" containing frame's header. The extended controls mechanism allows to send complex data structures to the kernel and program device's registers accordingly. However, the userspace might want to queue several frames in a row and set the corresponding extended controls at the same time. If the registers are programmed at the time an extended control is received, this means that at the time of processing a buffer, the registers might be programmed for another frame. This scenario is to be fixed by the "Request API".

The idea behind this API is to allow atomic operations like a QBUF and a S_EXT_CTRLS. As of August 2016, the "Request API" is still at the state of RFC, it has had quite a few proposals for the past few years but none of them got accepted into the kernel. The latest RFCs, related to the Media API are not able to handle controls so sunxi-cedrus had to use an older RFC.

The "sunxi-cedrus" kernel driver is hence made of a m2m v4l2 driver handling requests of MPEG2 or MPEG4 frames data with a standard header extended control. At the time of processing the m2m queue, it programs the VPU's registers depending on the used codec. Currently MPEG2 and MPEG4 are the only supported formats but H264 and H265 would be the next step.

A second limitation of the Allwinner's VPU is the need for buffers in the lower 256M of RAM. In order to allocate large sets of data in this area, "sunxi-cedrus" reserves a DMA pool that is then used by videobuf's dma-contig backend to allocate input and output buffers easily and integrate that with the v4l QBUF/DQBUF APIs.

From the userspace side of things, all the prior bitstream parsing is done by VA users(such as ffmpeg). Standard VA-API headers are given to the VA backend "sunxi-cedrus-drv-video" which is just in charge of ensuring a correspondence between v4l2 buffers and controls and VA structures. We can compare a VAPicture to a buffer plus an extended control in the OUTPUT queue and a VASurface to a multiplanar buffer in the CAPTURE queue. An Image is then "derived" from a Surface to produce a standard set of NV12 buffers that can be shown on screen by VLC for example. VA-API was an extremely appropriate choice compared to VDPAU since the data it provides are often very similar to the ones the VPU expects.

More info
You can HL Florent Revest (kido) on #cedrus on irc.freenode.net for in depth questions