FEL/USBBoot

On Allwinner SoCs it is possible to boot over USB OTG. This requires only minimal changes compared to the build steps in the manual build howto, and these changes are explained on this wiki page.

By booting over USB OTG, it is possible to forgo the SD-Card entirely, and it is especially useful for devices where no UART can be found, or where the UART is multiplexed with the SD-Card. You can then use a micro-SD breakout adapter to access the serial port.

= Install the tools =

There is a utility in the sunxi-tools repository called 'sunxi-fel'. This utility is used for booting the system over USB and it needs to be installed first.

The command line syntax of the 'sunxi-fel' utility: Usage: sunxi-fel [options] command arguments... [command...] -v, --verbose			Verbose logging

spl file			Load and execute U-Boot SPL If file additionally contains a main U-Boot binary (u-boot-sunxi-with-spl.bin), this command also transfers that to memory (default address from image), but won't execute it.

uboot file-with-spl		like "spl", but actually starts U-Boot U-Boot execution will take place when the fel utility exits. This allows combining "uboot" with further "write" commands (to transfer other files needed for the boot).

hex[dump] address length	Dumps memory region in hex dump address length		Binary memory dump exe[cute] address		Call function address read address length file	Write memory contents into file write address file		Store file contents into memory ver[sion]			Show BROM version clear address length		Clear memory fill address length value	Fill memory

= Switch your device into FEL mode =

Before the 'sunxi-fel' tool can actually talk to your device, the device needs to be connected to your PC using a "USB A to USB mini/micro B" cable.

And then the device needs to be switched into FEL mode. Please refer to the FEL howto for information on how to boot to FEL mode. Device pages should mention the button which triggers FEL mode as well.

If you run: sunxi-fel version

and it returns something like: AWUSBFEX soc=00162500(A13) 00000001 ver=0001 44 08 scratchpad=00007e00 00000000 00000000

Then you have successfully switched the device into FEL mode and it is ready to accept commands or load the system over USB.

= Boot the system over USB =

Getting the mainline U-Boot sources
To obtain the U-Boot sources, clone the current U-Boot master branch:

git clone git://git.denx.de/u-boot.git cd u-boot

Or alternatively the 'next' branch from the sunxi custodian tree:

git clone -b next git://git.denx.de/u-boot-sunxi.git cd u-boot-sunxi

Both of these branches are bleeding edge and may contain bugs from time to time. If you encounter troubles, try a tarball with the latest formal U-Boot release before giving up.

Booting U-Boot over USB
Then just do a regular u-boot build: make CROSS_COMPILE=arm-linux-gnueabihf- Cubietruck_defconfig make CROSS_COMPILE=arm-linux-gnueabihf- -j$(nproc)

And boot it over USB (the 'sunxi-fel uboot' command requires an up to date version of sunxi-tools): sunxi-fel uboot u-boot-sunxi-with-spl.bin

This boots U-Boot over USB. And after U-Boot takes control, it starts scanning various default locations for the boot.scr file in order to boot the rest of the system.

Booting the whole system over USB (U-Boot + kernel + initramfs)
This needs to be used with U-Boot v2015.10 or newer (which can automatically find the 'boot.scr' file in RAM after it had been uploaded there by 'sunxi-fel').

Assuming that you have a kernel image, a correct dtb blob for your hardware, a boot script and an initrd image, booting all of this can be done with just a single 'sunxi-fel' invocation: sunxi-fel -v uboot u-boot-sunxi-with-spl.bin \ write 0x42000000 uImage \ write 0x43000000 sun7i-a20-cubietruck.dtb \ write 0x43100000 boot.scr \ write 0x43300000 rootfs.cpio.lzma.uboot

If you wonder about all the magic addresses used in the command line above, the right values can be found in the U-Boot sources (kernel_addr_r=0x42000000, fdt_addr_r=0x43000000, scriptaddr=0x43100000, ramdisk_addr_r=0x43300000).

Early kernel development for new SoCs
Because FEL boot is supported by the BROM code, it is readily available out of the box. The 'sunxi-fel' tool only needs to be patched to add the SRAM layout information, but this is usually very simple and also extensively documented on this wiki page. A relatively challenging task is the U-Boot support, because it needs the DRAM initialization code for the SPL. But assuming that the U-Boot bootloader is already available, FEL boot can be used for kernel development even when there is no Ethernet or MMC support in the kernel yet. For the sake of convenience, it is best to write fel-sdboot.sunxi to the SD card (substitute /dev/sdX with the appropriate block device of your card reader): wget https://github.com/linux-sunxi/sunxi-tools/raw/master/bin/fel-sdboot.sunxi dd if=fel-sdboot.sunxi of=/dev/sdX bs=1024 seek=8

Then just use the instructions from the previous section about U-Boot + kernel + initramfs. And after the MMC support is implemented in the kernel, it becomes possible to move the rootfs to the SD card and get rid of the initrd image. But FEL USB boot is still useful until either Ethernet or USB becomes good enough for booting over network.

Legacy mainline U-Boot (v2015.04 and older versions)
Is here for historical reference only (click on the 'Expand' link to see it):

The U-Boot had to be configured using "*_felconfig" option before compiling (instead of "*_defconfig", as that builds a SPL only suitable for MMC boot). For example, in the case of a Cubietruck board, that would be: make CROSS_COMPILE=arm-linux-gnueabihf- Cubietruck_felconfig make CROSS_COMPILE=arm-linux-gnueabihf- -j$(nproc)

Then just use the 'sunxi-fel' tool to upload various pieces to certain magic addresses (the magic CONFIG_SPL_TEXT_BASE=0x2000 and CONFIG_SYS_TEXT_BASE=0x4a000000 values can be found in the U-Boot sources). And execute them in a certain order:

echo == upload the SPL to SRAM and execute it == sunxi-fel write 0x2000 spl/u-boot-spl.bin sunxi-fel exe  0x2000

sleep 1 # wait for DRAM initialization to complete

echo == upload the main u-boot binary to DRAM == sunxi-fel write 0x4a000000 u-boot.bin

echo == execute the main u-boot binary == sunxi-fel exe  0x4a000000

This boots U-Boot over USB. And after U-Boot takes control, it starts scanning various default locations for the boot.scr file in order to boot the rest of the system.

Legacy u-boot-sunxi
Is here for historical reference only (click on the 'Expand' link to see it):

Preparing U-Boot
U-boot needs to be built specifically for booting over USB. This is called FEL mode, and enabling this disables the full SD card (for u-boot, script.bin and the kernel might re-enable it later on).

Just follow the guide to compile u-boot, but  select a target whose name has _FEL.

Manual loading
While the script from the next section is perhaps better, you can now choose to load u-boot manually:

sunxi-fel write 0x2000 ../u-boot/spl/u-boot-spl.bin sunxi-fel exe 0x2000 sleep 1 # Wait for DRAM initialization to complete sunxi-fel write 0x4a000000 ../u-boot/u-boot.bin sunxi-fel exe 0x4a000000

usb-boot script
There was a script in the older releases of Sunxi-tools called usb-boot. You can download this script from github using the tag for an old version 1.2:

wget https://raw.githubusercontent.com/linux-sunxi/sunxi-tools/v1.2/usb-boot chmod +x usb-boot

And then use it as follows:

Usage: ./usb-boot u-boot-spl.bin u-boot.bin [boot.scr] [kernel script.bin [initramfs]]

u-boot-spl.bin and u-boot.bin are the u-boot binaries which are FEL enabled.

boot.scr is the u-boot script. If you do not specify a file ending with .scr the default is used.

uImage is your compiled kernel image.

script.bin is your hw configuration converted to binary from .fex.

initramfs is an optional initramfs/initrd image in u-boot mkimage format.

= Adding support for new SoC variants =

If everything is already working fine for you, then you can stop reading here :-) But sometimes you may get messages like "SPL: Unsupported SoC type" or "Warning: no 'soc_sram_info' data for your SoC" when trying to use the instructions from the previous sections. In this case, doing some work to add support for your SoC may be required. It is also a good idea to first check the SoC support status table below.

General description of the "sunxi-fel uboot" command implementation
Allwinner is treating booting over USB in a special way, and this behaviour is unfortunately hardcoded in the BROM. The main differences between booting from MMC/NAND and booting from USB are: This is at least inconsistent and definitely not good for us. U-Boot used to require a special size-optimized variant of the SPL specifically for booting via FEL, which also had the base address changed from 0x0 to 0x2000 as an additional inconvenience. Having a special variant of the SPL means an extra configuration to maintain. And the code size limitation is also a nasty problem because a certain set of features has to be disabled. But the "spl" and "uboot" commands, which are implemented by the "sunxi-fel" tool, can work-around this limitation by smuggling just the ordinary MMC or NAND variant of the U-Boot SPL into SRAM. More technical details are provided below.
 * For booting from the SD card or NAND, the BROM code is searching for a special eGON signature on a bootable media and loading up to 32K (in fact a bit less than that) of the initial code to the address 0x0 in SRAM (that's typically the SRAM section A1) and then executes it. This initial code (known as "SPL" in U-Boot or "boot0" in the Allwinner's bootloader) configures the DRAM to get access to more storage space, then loads the main part of the bootloader and the rest of the system there.
 * For booting via FEL, the Allwinner's idea is that we are supposed to upload only something like up to ~15K of code at the address 0x2000 in SRAM and execute it.

The reason why we have ~15K code size limitation when booting via FEL is illustrated on the picture below. When we are booting the device in FEL mode, a special code is activated in the BROM and starts communicating over USB using FEL protocol. The USB driver code from the BROM allocates two stacks at rather inconvenient locations inside of the first 32K of SRAM. The IRQ handler stack is set at the address 0x2000 and grows down. And the ordinary application stack is set at the address 0x7000 and also grows down. These stacks make the SRAM space fragmented, and the largest usable contiguous ~15K area is sandwiched between these two stacks. Overwriting either of these stacks via the "sunxi-fel write" crashes the device and it just stops responding to further FEL commands. So uploading a normal U-Boot SPL (which typically has size slightly larger than 20K) to the address 0x0 via "sunxi-fel write" command with the intention to execute it via "sunxi-fel exe" command does not work as expected.



So, how do we solve this problem? Allwinner devices typically have more than 32K of SRAM (the smallest total amount of SRAM among all devices is 48K in Allwinner A13). And we can use extra SRAM locations as a backup storage for the FEL stacks (shown as "backup area 1" and "backup area 2" on the picture above). We also upload a special thunk code, which is responsible for swapping the content of the FEL stacks with the content of these backup areas before jumping to the address 0x0. Now in order to execute a full-fledged SPL from U-Boot, we only need to split the SPL into chunks and upload it to SRAM, writing the parts which are supposed to overlap the FEL stacks to the backup areas. Executing the thunk code saves the FEL stacks to the backup areas, reassembles the SPL together and passes control to the SPL.

Why do we need to backup the original FEL stacks? The reason is that just uploading and executing the SPL alone is not enough to boot the system. The SPL code is very small and its primary task is to setup clocks and initialize the DRAM. After the DRAM is initialized, all the storage space problems are resolved and we want to load the main U-Boot code to the device. And for this we still need the BROM FEL code alive and getting control back, so that it can still talk with the 'sunxi-fel' tool over USB and execute FEL commands. Hence the SPL returns control back to the thunk code. The thunk code swaps FEL stacks with backup areas again and finally passes control back to the FEL code in the BROM, which is able to happily resume its work because it has all the original data back in its stacks.

The SoC-specific mandatory thing
The previous section describes how the "sunxi-fel spl" and "sunxi-fel uboot" commands work. But not everything is perfect. One inconvenient thing is that the SRAM address space layout (and the location of the backup areas) may be different for different SoC variants. Hence we need to provide the SRAM layout description information for each SoC in the source code of the 'sunxi-fel' program. The comments in the source code should provide reasonable explanations. And here is an example of such SRAM layout information for A31: /* * A31 is very similar to A10/A13/A20, except that it has no SRAM at 0x8000. * So we use the SRAM section at 0x44000 instead. This is the memory, which * is normally shared with the OpenRISC core (should we do an extra check to * ensure that this core is powered off and can't interfere?). */ sram_swap_buffers a31_sram_swap_buffers[] = { { .buf1 = 0x01800, .buf2 = 0x44000, .size = 0x800 }, { .buf1 = 0x05C00, .buf2 = 0x44800, .size = 0x8000 - 0x5C00 }, { 0 } /* End of the table */ };

soc_sram_info soc_sram_info_table[] = { {		.soc_id      = 0x1633, /* Allwinner A31 */ .thunk_addr  = 0x46E00, .thunk_size = 0x200, .swap_buffers = a31_sram_swap_buffers, },	{ 0 } /* End of the table */ }; Basically, the first backup area for A31 is set at 0x44000 and covers the IRQ stack (0x1800-0x2000). The second backup area is set at 0x44800 and covers the normal FEL stack plus also some extra area above it (0x5C00-0x8000). The thunk code is placed at 0x46E00.

The SoC-specific bonus features

 * While this is not strictly required, we can tweak the cacheability attributes of the memory sections by modifying the MMU translation table. This improves the performance of the "sunxi-fel write" command and helps to significantly reduce the time needed to upload large chunks of data to DRAM.


 * The Cortex-A8 based SoC variants need a tweak for the AUXCR L2EN bit. Without this, the L2 cache ends up disabled in Linux after booting over FEL.

Testing
After adding the support for a new SoC variant to the 'sunxi-fel' tool, it makes sense to actually test whether it works. In the case if U-Boot already has support for this particular SoC variant, the testing is simple and can be done by just running "sunxi-fel uboot u-boot-sunxi-with-spl.bin" command and confirming that U-Boot starts properly on the device.

In the case if U-Boot still does not have full support for this particular SoC variant yet, testing still can be done using the Allwinner's boot0 bootloader. If you have some boot0-based bootable SD card image (let's call it 'sdcard-image.bin'), then you can: If the system boots normally, in the same way as just doing a normal boot from the SD card, then the 'sunxi-fel' tool is working fine. Please note that you may get error messages from the 'sunxi-fel' tool while doing this. This is normal and expected (boot0 does not return control back to the FEL code in BROM but instead does its usual booting of the system from the SD card).
 * write this image to an SD card
 * extract the boot0 part from this image via running "dd if=sdcard-image.bin of=boot0.bin bs=1024 skip=8 count=32"
 * switch the device into FEL mode
 * insert the SD card
 * run "sunxi-fel spl boot0.bin"

SoC support status
= Potential future improvements =

There is quite a lot of room for improvement:
 * Allow the 'sunxi-fel spl' command to also use raw SD card images in addition to just recognizing 'u-boot-sunxi-with-spl.bin' format.
 * Store the magic addresses (CONFIG_SYS_TEXT_BASE, kernel_addr_r, fdt_addr_r, scriptaddr, ramdisk_addr_r) in the EGON header extension in order to avoid the need of passing them to the 'sunxi-fel' tool as command line parameters.
 * Single U-Boot binary for both the regular UART serial console and the UART over the MicroSD Breakout configurations. The configuration option can be provided to the 'sunxi-fel' tool in a command line argument and handed over to to U-Boot in the eGON header extension.

The exact eGON header extension format needs to be formalized. We should also check whether it can provide backwards/forward compatibility with the Allwinner's BOOT0 bootloader.

= Known issues =


 * Allwinner A80 is not supported yet

Please report problems to https://github.com/linux-sunxi/sunxi-tools/issues

= See also =


 * FEL
 * miniroot