CPUIDLE

CPUIDLE is a hardware block present on R40/H6 and newer SoCs that implements the CPU/cluster idle process in hardware. It can also be used to implement CPU hot removal in a clean and race-free manner.

CPU Power Off
At least the following actions are taken:
 * 1) DBGPWRDUP is cleared for the core.
 * 2) The C0_PWROFF_GATING_REG bit for the core is set.
 * 3) The CPU power switch is turned off, using the sequence and delays programmed below.
 * 4) The CPUn_OFF or CPUn_VOTE bit in CPUIDLE_STAT_REG, as appropriate, is set.

CPU Power On
At least the following actions are taken:
 * 1) The CPUn_OFF and CPUn_VOTE bits in CPUIDLE_STAT_REG are cleared.
 * 2) The CPU power switch is turned on, using the sequence and delays programmed below.
 * 3) The C0_PWROFF_GATING_REG</tt> bit for the core is cleared.
 * 4) The C0_PWRON_RESET_REG</tt> bit for the core is set (deasserted).
 * 5) The C0_RST_CTRL_REG</tt> bit for the core is set (deasserted).
 * 6) DBGPWRDUP</tt> is set for the core.

Cluster Power Off
At least the following actions are taken. There may be more actions (such as flushing L2 cache), but those happen too quickly to be observable.
 * 1) C0_PWROFF_GATING_REG</tt> is set to 0x1f (cluster and all CPUs gated).
 * 2) C0_PWRON_RESET_REG</tt> is set to 0x0 (all resets asserted).
 * 3) The CLUSTER_OFF</tt> bit in CPUIDLE_STAT_REG</tt> is set.

Notably, CPU_SYS_RESET</tt> is not asserted, so at least part of the GIC is still functional.

Cluster Power On
At least the following actions are taken:
 * 1) The CLUSTER_OFF</tt> bit in CPUIDLE_STAT_REG</tt> is cleared.
 * 2) The C0_PWROFF_GATING_REG</tt> bit for the cluster (bit 4) is cleared.
 * 3) The C0_PWRON_RESET_REG</tt> bit for the cluster (bit 16) is set (deasserted).
 * 4) The C0_RST_CTRL_REG</tt> bits for various cluster resets are set (deasserted).
 * 5) RVBARADDR</tt> for CPU 0 is set to the hardcoded value 0x20060 (in SRAM A1, after an assumed eGON header).
 * 6) The CPU power on sequence is performed for the lead core.

Note: AA64nAA32</tt> is not restored, making this functionality unsuitable for a 64-bit SoC.

Registers
The CPUIDLE hardware is controlled by several registers in the <tt>R_CPUCFG</tt> MMIO space. They are only partially documented. Many of the register and field names are guesses based on the observed functionality.

Register Layout (sun8i)
The register layout is partially documented in the A40i/T3/other manuals.

Register Layout (sun50i)
The register layout is not documented, but is very similar to the sun8i layout.

CLOSE_FLAG_REG
From the ARM CPUs, only the bits corresponding to the local CPU can be set. From the ARISC, any bit can be set.

No action will be taken until both:
 * 1) The corresponding <tt>CPUn_IDLE</tt> bit in <tt>CPUIDLE_STAT_REG</tt> is set (i.e. the core is in WFI), and
 * 2) The corresponding <tt>CPUn_VOTE</tt> or <tt>CPUn_OFF</tt> bit (as applicable) in <tt>CPUIDLE_STAT_REG</tt> is cleared

The bit will automatically clear when the the power off sequence finishes.

The bits cannot be cleared by writing to this register or by disabling the CPUIDLE hardware. If bits in this register are set while the CPU is already off, they can be cleared by clearing the <tt>CPUn_VOTE</tt> or <tt>CPUn_OFF</tt> bit in <tt>CPUIDLE_STAT_REG</tt>, thus re-triggering the power off sequence. However, if the CPU is on, there is no way to abort the sequence.

The power off sequence will proceed regardless of the value of <tt>CPUIDLE_PEND_REG</tt>.

CPUIDLE_PEND_REG
There are probably bits for FIQs, but currently Group 0 interrupts are unused.

CPUIDLE_WAKE_REG
There are probably bits for FIQs, but currently Group 0 interrupts are unused.

The Allwinner blob initializes this register to 0xffff at boot.

CPUIDLE_STAT_REG
The cluster is automatically powered off any time the following conditions are true:
 * 1) <tt>CLUSTER_IDLE</tt> is set,
 * 2) <tt>CLUSTER_OFF</tt> is clear, and
 * 3) all <tt>CPUn_VOTE</tt> bits are set.

At cluster idle time, as coordinated by PSCI or SCPI, software can set the <tt>CPUn_VOTE</tt> bits to trigger a cluster power off. (Since all CPUs are already off at that point, <tt>CLUSTER_IDLE</tt> should be set automatically.) Therefore, it is never necessary to use <tt>CLOSE_CPUn_AND_CLUSTER</tt> at CPU hotplug/idle time.

Entry and Exit Time Measurements
Here is some information explaining the idle state timings for A64, but the same general concepts apply to other SoCs:

Powering off idle CPUs saves about 30 mW compared to using WFI only. Additional power savings are possible by idling the L2 and downclocking the cluster when all CPUs are idle, but those were not explored.

Entry and exit latency were measured using a logic analyzer, with GPIO pins toggled in Linux after the calls to trace_cpu_idle in cpuidle_enter_state, and in the power management firmware after CPU power-off completes and immediately after detecting an interrupt.

800 us and 1500 us are worst-case values, largely driven by the fact that the power management firmware is single threaded. It can only handle commands to power off CPUs one at a time, and it cannot process any commands while powering on a CPU in response to an interrupt.

The cluster suspend process reliably takes 36 us; I rounded this up to 50 us. If all CPUs enter the cluster idle state at the same time, exit latency is actually reduced, because there is no contention in that case. However, if only some CPUs enter the cluster idle state, behavior is the same as for CPU idle.

Polling delay for the power management firmware to detect a pending interrupt is insignificant; it is less than 20 us.

min-residency was chosen as the point where enabling the idle state consumed no more average power than disabling the idle state at a variety of interrupt rates.