There is an excellent RISC-V board on my desk: a SpacemiT PicoITX/K3. And an excellent emulator installed on my workstation: QEMU with RVA23 support. So I thought: since this is the most advanced virtualization technology available, which hypervisor matching my idea of a "Resource Partitioner" already exists?
Bao, of course. There was a very interesting recording about it recently, from this year's RISC-V summit in Bologna.
Bao is a static partitioning hypervisor. No scheduler, no dynamic memory, no shared drivers: each guest gets its own cores, its own memory and its own devices, all decided at build time and never renegotiated. A few thousand lines of code you can actually read.
Given that I already have HFI BIOS and mr-bml, I thought: why not apply the same approach to improving bao that I already use in HFI BIOS? Trimming sources, patching, etc....
The name suggested itself: QBao 😃
Step 0: bao with no guest OSes at all
The first attempt was to run bao with no guest OSes whatsoever. It worked.
Trimming turned out to be simpler than with U-Boot: bao's build leaves a
.d file per object, listing exactly what the compiler actually opened.
So the list of needed files is not guessed but read back out of the build itself:
581 files at the top, 121 at the bottom. What survives is
src/arch/riscv with the AIA interrupt controller, src/core
and its MMU half, src/lib, the qemu-riscv64-virt platform
and the sbi_uart driver.
Three things worth noting straight away.
A commit, not a tag. I pinned to a commit on main rather than to the v2.0.0 tag: the tag predates the switch of the qemu-riscv64-virt platform from PLIC to AIA, and the vector, Zicbom/Zicboz and Sstateen work. All of that is RVA23 territory.
Debian ships only lp64d. Bao builds with -mabi=lp64 (soft
float, the right call for a hypervisor), so the very first
<limits.h> reaches through features.h →
gnu/stubs.h for gnu/stubs-lp64.h, which does not exist. A
freestanding build does not want the libc headers anyway, so the build drops them
(-nostdinc), hands back GCC's own, and adds an empty
limits.h for the #include_next that GCC's
limits.h ends on. Not a bao defect; a multilib gap.
There is no such thing as an empty configuration. Bao's configuration IS
its guest list. An empty one yields CONFIG_VM_NUM 0 and therefore a
zero-length array in vmm.c, which the build rejects. Upstream has
configs/null with one dummy VM, purely so the tree compiles; it does
not survive a run. And it does not survive it quietly: an ERROR()
inside mem_init() reaches console_write() before
console_init() has run, and spins there in
while (!console_ready). The machine simply says nothing, not even a
banner.
Hence my own configuration: one VM with zero cpus. Nothing is ever scheduled, and the hypervisor says so:
Bao Hypervisor v2.0.0-50-g53f2b4c-dirty (Sep 1 2026 - 15:45:29) BAO ERROR: cannot start guest OS: configuration declares no guest cpus
Step 1: the same thing, off a disk and through real firmware
Next, the long way round, the way it will be on the board. Nothing is injected
with -kernel:
QEMU reset -> HFI BIOS SPL (M-mode, -bios)
-> u-boot.itb off this disk's ESP, found by GPT type
-> OpenSBI (M) -> HFI/U-Boot (HS) -> the BIOS front end
-> mr-bml (ESP: EFI/boot/bootriscv64.efi)
-> QBao, in HS-mode
This is where it turned out that bao.elf cannot be loaded by any
loader. Bao sets its location counter to BAO_VAS_BASE and never
gives the loadable sections a load address, so every PT_LOAD carries
p_paddr == p_vaddr == 0xffffffc0.... QEMU's -kernel does
not care: it drops the flat bao.bin at 0x80200000 and enters there. A
real loader, however, copies each PT_LOAD to its p_paddr,
and no such physical address exists.
The fix is one objcopy --change-section-lma shifting every LMA down
by BAO_VAS_BASE - 0x80200000. The entry point stays high, so the
loader's own translation (e_entry - p_vaddr + p_paddr) lands exactly
where OpenSBI enters bao today. bao.bin comes out byte-identical, which
is the check that nothing else moved.
A second detail: the machine needs one hart more than bao uses.
platform.cpu_num is 4, and bao's master starts harts 0..3 itself over
SBI HSM, so all four must be free. The VideoBIOS, meanwhile, keeps a hart of its own
for its character generator and never gives it back. You can see it in the log:
VideoBIOS controller owns hart mask 0x10 — hart 4, just past bao.
The main thing: guests are linked into the hypervisor binary
And here the reason for all of this became visible. Bao's guest list is a C file,
and VM_IMAGE(name, path) is an .incbin. The guest images
live inside bao.elf, so changing the partitioning means
rebuilding the hypervisor.
I want the opposite: the guest list should arrive with the boot.
Why Multiboot3 rather than kernel + initrd
mr-bml has a kernel command which would load the hypervisor
perfectly well. But it can hand over exactly one blob in the role of an initrd,
because a device tree can carry linux,initrd-start only once.
That is not a convention but a structural limit: it is a property of the
/chosen node, and property names are unique within a node. Trivially
checked:
$ dtc -I dts -O dtb dup.dts
dup.dts:6.9-47: ERROR (duplicate_property_names):
/chosen:linux,initrd-start: Duplicate property name
module3, on the other hand, appends. Every call adds a module with
its own command line, so the guest list becomes a list:
menuentry 'QBao hypervisor + QSOE/N + Linux' {
multiboot3 /boot/qbao/bao-load.elf
module3 /boot/qbao/qbao.cfg qbao.cfg
module3 /boot/guests/skimmer.bin qsoe-n
module3 /boot/guests/modpkg.cpio qsoe-modpkg
}
The first word of a module's command line is its name, and that is what joins
this menu to the configuration. The Multiboot3 path in mr-bml had to be finished
first: the hand-off function for the case without EFI boot services was a stub
calling mrbml_fatal(), and by then the EFI console was already gone.
A silent death with no diagnostic. That went into mr-bml 0.9.
modrunner
QBao's own part is called modrunner/. It attaches without any patch
to the build system: bao already compiles an out-of-tree
CONFIG_REPO/CONFIG directory into the hypervisor and puts its
inc/ on the include path — exactly the hook a front end needs. The same
relationship BIOS/ has with U-Boot in HFI BIOS.
The Multiboot3 loader enters it with a0 holding the magic,
a1 the address of the boot information, and no stack at all. modrunner
then walks the tags, reads the partition map out of qbao.cfg, stages
the guest images into memory bao owns, fills in the same struct config
bao was going to read, and enters bao.
The configuration is mandatory — inventing one would be a worse answer than stopping:
Guest QSOEN {
module = qsoe-n
initrd = qsoe-modpkg
cpus = 2
base = 0x80200000
size = 0x10000000
entry = 0x80200000
}
Five traps, all of them silent
Each of these produces a machine that says nothing at all, for the reason
described above: an ERROR() raised in early memory has nowhere to go.
Recorded here because each one cost hours.
Nothing that must survive may live in .bss. Bao clears .bss
after modrunner has handed over. Anything the hypervisor is meant to read
later goes in .datanocopy, which is loaded from the image and never
cleared. A stack in .bss is perfectly fine: it is dead by then.
No pointer may be baked into an initializer. modrunner runs with the MMU
off, so &x under -mcmodel=medany is PC-relative and
comes out physical, while anything the linker resolved is virtual.
Writing through such a pointer is a fault at an address starting
0xffffffc0. This needed an explicit conversion, anchored on a word in
assembly holding the link-time address of _image_start — in assembly
precisely so the compiler cannot fold it back into a PC-relative computation of the
same symbol.
The entry stub must let secondary harts through. Bao brings its other
harts up with sbi_hart_start(hartid, load_addr, 0), and
load_addr is the start of the image — which is now my stub. They arrive
with a0 = hartid rather than the magic, and must fall through into bao.
Turn them away and the master waits at its boot barrier forever. The magic is
exactly what distinguishes the two entries.
image.separately_loaded must be true. Otherwise
config_init() rewrites load_addr = load_addr - BAO_VAS_BASE +
img_addr. Correct for an .incbin'd image carrying a link-time
address, nonsense for one the loader placed.
Guests are staged downward from the top of bao's region. They have to be
staged at all because the loader takes memory from the bottom (around 0x80003000),
while bao owns only what its platform description claims, from 0x80200000 up. And
they go at the top because immediately above the hypervisor image sits the page
pool's bitmap (root_pool_set_up_bitmap(), 128 KiB for a 4 GiB pool).
The console
The guest then ran, but had nowhere to put a word:
BAO WARNING: guest issued unsupport sbi extension call (1145193294)
1145193294 is 0x4442434E, "DBCN": the SBI debug console. Bao does not forward it.
I started with console_write_byte (FID 2), where the byte arrives in a
register and there is nothing to translate, and added a warning for the remaining
functions. The very first run said function 0: QSOE/N writes buffers,
not bytes.
And console_write hands over a buffer by guest-physical
address, which the hypervisor's address space does not map. So each page of the
buffer is walked through the guest's second stage and mapped in turn. A page at a
time deliberately: guest-physical contiguity says nothing about host contiguity when
the memory came from a page pool.
console_read I do not forward. One console and several guests is an
input policy: only one of them can be listening, and deciding which belongs to the
configuration, not to this file. For the same reason DBCN is deliberately not added
to the extension table: a guest that asks whether the extension is available is told
no, which is true of the extension as a whole.
Since the wire is shared for now, every guest line is labelled with the name its configuration gave it:
QSOEN-> ================================================
QSOEN-> QSOE/N microkernel ("Skimmer") v0.25
QSOEN-> ================================================
QSOEN-> *** PANIC at kernel/arch/riscv/fdt.c:1099:
fdt_init: NULL FDT pointer (bootloader broke contract) ***
A device tree for each partition
The guest is right: bao enters it with a1 = 0, and
vcpu_arch_reset() even carries a comment saying that ought to be the
DTB address.
The host's tree will not do. It describes the whole machine: every hart, all of
memory, every device. A partition owns a slice, and handing it the whole picture
would be handing it a map of memory it must not touch and cpus that are not its own.
So the tree is built from the partition's own description — the same
qbao.cfg.
Two details I would not have guessed, and learned by asking the guest.
First, QSOE/N does not merely prefer an initrd: without
/chosen/linux,initrd-* its fdt_init panics. Which closes
the circle rather neatly, since that same property — the one a tree can carry only
once — is precisely why the guest list needed module3.
Second, the APLIC node needs a phandle of its own. The guest collects
APLIC candidates and then picks the one whose msi-parent points at the
S-mode IMSIC, and a node without its own phandle never makes it into the candidate
list at all. So: "no interrupt controller", with the controller described right
there. One property.
The AIA addresses also have to go not only into the tree but into
vm_config.platform.arch.irqc: bao emulates the APLIC at
irqc.aia.aplic.base and places the guest's IMSIC files at
irqc.aia.imsic.base + PAGE_SIZE * vcpu_id. A tree that disagrees points
the guest at memory nothing answers.
The result: the guest reads the tree, believes it, and fences off exactly the regions modrunner placed:
QSOEN-> fdt: no PCI host bridge -- PCI surface disabled QSOEN-> timer: timebase 10000000 Hz, tick 10000 cycles, scan hart 0 QSOEN-> FDT: aplic@0xd000000 srcs=96 imsic@0x28000000 ids=255 gidxbits=0 stride=0x1000 QSOEN-> ram@0x80200000 size=0x10000000 QSOEN-> initrd@0x900cf000 size=0x120e00 QSOEN-> physmem: 1 bank(s), 3 exclusion(s) QSOEN-> ram [0x80200000 .. 0x90200000) fdt memory@ node QSOEN-> excl [0x901f0000 .. 0x901f04ad) fdt blob QSOEN-> excl [0x900cf000 .. 0x901efe00) initrd (modpkg cpio) QSOEN-> excl [0x80200000 .. 0x81181000) firmware + kernel QSOEN-> intc: selected aia backend BAO ERROR: unknown synchronous exception (22)
Every number there came out of qbao.cfg. Nothing was recompiled to
put a guest there.
Where it stands now
scause 22 is virtual instruction: an instruction the hardware
refused to execute in VS-mode and handed to the hypervisor, which has no handler for
it, so it dies as "unknown". QSOE/N drives
siselect/sireg/stopei directly, and that is
the likeliest culprit. Naming the instruction precisely would take
htinst, which that handler already has to hand and simply discards.
Console input is open too: output works (that whole log is DBCN), but nothing can be typed at a guest, because no UART is assigned to one. Which partition owns which console is a configuration question.
Conclusion
The evening came to four patches against bao. Two of them are the price of grafting one's own front end onto any hypervisor: a diagnostic, and some room in the linker script. The other two are of a different kind: bao gives a guest neither a console nor a device tree.
And those are not oversights. They follow from where bao stands: the integrator bakes everything at build time, and the guest is a bare-metal RTOS compiled for one board, which needs no tree, gets its UART passed through, and asks almost nothing of SBI. QSOE/N and Linux are the opposite kind of guest: general operating systems that expect the SBI/Linux boot contract — hart id, device tree, console, HSM, timer. Every gap I hit today is that one mismatch, showing up in a different place.
Bao is well made. It is simply made for a different guest.
What does come out of this evening for certain: Multiboot3 works as a hand-off
mechanism. It carried a hypervisor, a configuration file and two guest images off a
GPT disk through real firmware, and the contract held. Along the way it found two
things in mr-bml that would otherwise still be sitting there: an unimplemented
hand-off function, and a trap in the width of the type field in tags
(most tags begin with a 32-bit type, but MODULE and MEMMAP with a 64-bit one, so
size sits at offset 8 rather than 4; a walker assuming 4 reads the zero
upper half of the type as the size and loops on the spot forever).
No comments:
Post a Comment