thank you for reading and appreciated for reply!
I followed this doc and successfully ran xen 4.12.0 on qemu 3.1.0 with Linux 4.20.11 as Dom0.
Then I tried to use higher version of qemu(qemu 5.1.0 && 6.0.0) to do the same thing, but the process got stuck in the step of running xen with linux.
Here is how I download and build the Qemu 6.0.0:
wget -nd https://download.qemu.org/qemu-6.0.0.tar.xz
tar xvf qemu-6.0.0.tar.xz
cd qemu-6.0.0
mkdir build
cd build
../configure --target-list=aarch64-softmmu
make -j8
./aarch64-softmmu/qemu-system-aarch64 --version
# QEMU emulator version 6.0.0
# Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
Then I used this newly built qemu to run with XEN, but got Linux kernel panic error on booting. Here is the log output:
$ uname -a
Linux Cooper-VirtualBox 5.8.0-59-generic #66~20.04.1-Ubuntu SMP Thu Jun 17 11:14:10 UTC 2021 x86_64 x86_64 x86_64 GNU_Linux
$ ls
Image Image.gz rootfs.cpio.gz u-boot.bin virt-gicv3.dtb xen
$ qemu-system-aarch64-600 \
-machine virt,gic-version=3 -machine virtualization=true \
-cpu cortex-a57 -machine type=virt -m 4096 -smp 4 -bios u-boot.bin \
-device loader,file=xen,force-raw=on,addr=0x49000000 \
-device loader,file=Image.gz,addr=0x47000000 \
-device loader,file=virt-gicv3.dtb,addr=0x44000000 \
-nographic -no-reboot -chardev socket,id=qemu-monitor,host=localhost,port=7777,server,nowait,telnet -mon qemu-monitor,mode=readline
U-Boot 2019.01 (Jul 14 2021 - 11:02:43 +0800)
DRAM: 4 GiB
Flash: 128 MiB
*** Warning - bad CRC, using default environment
In: pl011@9000000
Out: pl011@9000000
Err: pl011@9000000
Net: No ethernet found.
Hit any key to stop autoboot: 0
scanning bus for devices...
Device 0: unknown device
Device 0: unknown device
starting USB...
No controllers found
No ethernet found.
No ethernet found.
=> fdt addr 0x44000000
=> fdt set /chosen \#address-cells <1>
=> fdt set /chosen \#size-cells <1>
=> fdt mknod /chosen module@0
=> fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
=> fdt set /chosen/module@0 reg <0x47000000
Unexpected character '
'
=> fdt set /chosen/module@0 reg <0x47000000 0x7ccfb8>
=> fdt set /chosen/module@0 bootargs "earlyprintk=serial,ttyAMA0
> console=ttyAMA0,115200n8 earlycon=xenboot"
=> fdt resize
=> booti 0x49000000 - 0x44000000
## Flattened Device Tree blob at 44000000
Booting using the fdt blob at 0x44000000
reserving fdt memory region: addr=44000000 size=2000
Loading Device Tree to 00000000ffffa000, end 00000000ffffefff ... OK
Starting kernel ...
Xen 4.12.0
(XEN) Xen version 4.12.0 (Cooper@) (aarch64-linux-gnu-gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0) debug=n Wed Jul 14 11:05:52 CST 2021
(XEN) Latest ChangeSet:
(XEN) Processor: 411fd070: "ARM Limited", variant: 0x1, part 0xd07, rev 0x0
(XEN) 64-bit Execution:
(XEN) Processor Features: 0000000001000222 0000000000000000
(XEN) Exception Levels: EL3:No EL2:64+32 EL1:64+32 EL0:64+32
(XEN) Extensions: FloatingPoint AdvancedSIMD GICv3-SysReg
(XEN) Debug Features: 0000000010305106 0000000000000000
(XEN) Auxiliary Features: 0000000000000000 0000000000000000
(XEN) Memory Model Features: 0000000000001124 0000000000000000
(XEN) ISA Features: 0000000000011120 0000000000000000
(XEN) 32-bit Execution:
(XEN) Processor Features: 00000131:10011001
(XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
(XEN) Extensions: GenericTimer
(XEN) Debug Features: 03010066
(XEN) Auxiliary Features: 00000000
(XEN) Memory Model Features: 10101105 40000000 01260000 02102211
(XEN) ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 62500 KHz
(XEN) GICv3 initialization:
(XEN) gic_dist_addr=0x00000008000000
(XEN) gic_maintenance_irq=25
(XEN) gic_rdist_stride=0
(XEN) gic_rdist_regions=1
(XEN) redistributor regions:
(XEN) - region 0: 0x000000080a0000 - 0x00000009000000
(XEN) GICv3: 256 lines, (IID 0000043b).
(XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN) Allocated console ring of 16 KiB.
(XEN) Bringing up CPU1
(XEN) GICv3: CPU1: Found redistributor in region 0 @000000004003c000
(XEN) Bringing up CPU2
(XEN) GICv3: CPU2: Found redistributor in region 0 @000000004005c000
(XEN) Bringing up CPU3
(XEN) GICv3: CPU3: Found redistributor in region 0 @000000004007c000
(XEN) Adding cpu 3 to runqueue 0
(XEN) CPU 3 booted.
(XEN) Brought up 4 CPUs
(XEN) P2M: 44-bit IPA with 44-bit PA and 8-bit VMID
(XEN) P2M: 4 levels with order-0 root, VTCR 0x80043594
(XEN) I/O virtualisation disabled
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading Domd0 kernel from boot module @ 0000000047000000
(XEN) Allocating 1:1 mappings totalling 512MB for dom0:
(XEN) BANK[0] 0x00000060000000-0x00000080000000 (512MB)
(XEN) Grant table range: 0x00000049000000-0x00000049040000
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading zImage from 0000000138000000 to 0000000060080000-00000000612c1200
(XEN) Loading dom0 DTB to 0x0000000068000000-0x0000000068001d0c
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) ***************************************************
(XEN) No support for ARM_SMCCC_ARCH_WORKAROUND_1.
(XEN) Please update your firmware.
(XEN) ***************************************************
(XEN) No support for ARM_SMCCC_ARCH_WORKAROUND_1.
(XEN) Please update your firmware.
(XEN) ***************************************************
(XEN) No support for ARM_SMCCC_ARCH_WORKAROUND_1.
(XEN) Please update your firmware.
(XEN) ***************************************************
(XEN) PLEASE SPECIFY dom0_mem PARAMETER - USING 512M FOR NOW
(XEN) ***************************************************
(XEN) 3... 2... 1...
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 312kB init memory.
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]
[ 0.000000] Linux version 4.20.11 (Cooper@Cooper-VirtualBox) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #1 SMP PREEMPT Wed Jul 14 10:53:34 CST 2021
[ 0.000000] Machine model: linux,dummy-virt
[ 0.000000] earlycon: xenboot0 at I/O port 0x0 (options '')
[ 0.000000] printk: bootconsole [xenboot0] enabled
[ 0.000000] Xen 4.12 support found
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: UEFI not found.
[ 0.000000] cma: Reserved 32 MiB at 0x000000007e000000
[ 0.000000] NUMA: No NUMA configuration found
[ 0.000000] NUMA: Faking a node at [mem 0x0000000060000000-0x000000007fffffff]
[ 0.000000] NUMA: NODE_DATA [mem 0x7dfea840-0x7dfebfff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA32 [mem 0x0000000060000000-0x000000007fffffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000060000000-0x000000007fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000060000000-0x000000007fffffff]
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv1.1 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: Trusted OS migration not required
[ 0.000000] psci: SMC Calling Convention v1.1
[ 0.000000] random: get_random_bytes called from start_kernel+0x94/0x40c with crng_init=0
[ 0.000000] percpu: Embedded 23 pages/cpu @(____ptrval____) s55704 r8192 d30312 u94208
[ 0.000000] Detected PIPT I-cache on CPU0
[ 0.000000] CPU features: enabling workaround for ARM erratum 832075
[ 0.000000] CPU features: enabling workaround for ARM erratum 834220
[ 0.000000] CPU features: enabling workaround for EL2 vector hardening
[ 0.000000] CPU features: enabling workaround for Speculative Store Bypass Disable
[ 0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 129024
[ 0.000000] Policy zone: DMA32
[ 0.000000] Kernel command line: earlyprintk=serial,ttyAMA0
[ 0.000000] console=ttyAMA0,115200n8 earlycon=xenboot
[ 0.000000] Memory: 455476K/524288K available (10748K kernel code, 1412K rwdata, 5112K rodata, 1344K init, 380K bss, 36044K reserved, 32768K cma-reserved)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[ 0.000000] rcu: Preemptible hierarchical RCU implementation.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4.
[ 0.000000] Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[ 0.000000] GICv3: Distributor has no Range Selector support
[ 0.000000] GICv3: no VLPI support, no direct LPI support
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER4
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER8
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER12
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER16
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER20
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER24
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER28
[ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000
(XEN) d0v0: vGICR: SGI: unhandled word write 0xffffffff to ICACTIVER0
[ 0.000000] arch_timer: cp15 timer(s) running at 62.50MHz (virt).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x1cd42e208c, max_idle_ns: 881590405314 ns
[ 0.000108] sched_clock: 56 bits at 62MHz, resolution 16ns, wraps every 4398046511096ns
[ 0.017243] Console: colour dummy device 80x25
[ 0.020478] Calibrating delay loop (skipped), value calculated using timer frequency.. 125.00 BogoMIPS (lpj=250000)
[ 0.021062] pid_max: default: 32768 minimum: 301
[ 0.031104] LSM: Security Framework initializing
[ 0.032772] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.033612] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[ 0.034046] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)
[ 0.034320] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes)
[ 0.121936] ASID allocator initialised with 32768 entries
[ 0.130646] xen:grant_table: Grant tables using version 1 layout
[ 0.131832] Grant table initialized
[ 0.145062] xen:events: Using FIFO-based ABI
[ 0.147228] Xen: initializing cpu0
[ 0.174809] rcu: Hierarchical SRCU implementation.
[ 0.194598] EFI services will not be available.
[ 0.215474] smp: Bringing up secondary CPUs ...
(XEN) d0v1: vGICR: SGI: unhandled word write 0xffffffff to ICACTIVER0
(XEN) d0v2: vGICR: SGI: unhandled word write 0xffffffff to ICACTIVER0
[ 0.273655] Detected PIPT I-cache on CPU1
[ 0.274338] GICv3: CPU1: found redistributor 1 region 0:0x00000000080c0000
[ 0.275368] Xen: initializing cpu1
[ 0.275561] CPU1: Booted secondary processor 0x0000000001 [0x411fd070]
[ 0.339645] Detected PIPT I-cache on CPU2
[ 0.339831] GICv3: CPU2: found redistributor 2 region 0:0x00000000080e0000
[ 0.340694] Xen: initializing cpu2
[ 0.340845] CPU2: Booted secondary processor 0x0000000002 [0x411fd070]
[ 0.400734] Detected PIPT I-cache on CPU3
[ 0.400925] GICv3: CPU3: found redistributor 3 region 0:0x0000000008100000
[ 0.401355] Xen: initializing cpu3
[ 0.401498] CPU3: Booted secondary processor 0x0000000003 [0x411fd070]
[ 0.574444] smp: Brought up 1 node, 4 CPUs
[ 0.703442] SMP: Total of 4 processors activated.
[ 0.703908] CPU features: detected: GIC system register CPU interface
[ 0.704393] CPU features: detected: 32-bit EL0 Support
[ 0.704796] CPU features: detected: CRC32 instructions
[ 1.039745] CPU: All CPU(s) started at EL1
[ 1.046203] alternatives: patching kernel code
[ 1.380096] devtmpfs: initialized
[ 1.662983] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 1.667044] futex hash table entries: 1024 (order: 4, 65536 bytes)
[ 1.709743] pinctrl core: initialized pinctrl subsystem
[ 1.904352] DMI not present or invalid.
[ 1.934684] NET: Registered protocol family 16
[ 2.083444] audit: initializing netlink subsys (disabled)
[ 2.327780] audit: type=2000 audit(0.476:1): state=initialized audit_enabled=0 res=1
[ 2.434832] vdso: 2 pages (1 code @ (____ptrval____), 1 data @ (____ptrval____))
[ 2.437256] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[ 2.625009] DMA: preallocated 256 KiB pool for atomic allocations
[ 2.628185] xen:swiotlb_xen: Warning: only able to allocate 4 MB for software IO TLB
[ 2.643167] software IO TLB: mapped [mem 0x7bc00000-0x7c000000] (4MB)
[ 2.666593] Serial: AMBA PL011 UART driver
[ 2.781142] Internal error: synchronous external abort: 96000010 [#1] PREEMPT SMP
[ 2.784420] Modules linked in:
[ 2.785879] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.20.11 #1
[ 2.787890] Hardware name: linux,dummy-virt (DT)
[ 2.789880] pstate: 40000005 (nZcv daif -PAN -UAO)
[ 2.796098] pc : amba_device_try_add+0x150/0x2a8
[ 2.797158] lr : amba_device_try_add+0x13c/0x2a8
[ 2.798292] sp : ffff00000807bc30
[ 2.799045] x29: ffff00000807bc30 x28: 0000000000000000
[ 2.800351] x27: 0000000000000001 x26: ffff000009272740
[ 2.801523] x25: ffff80003dff3238 x24: ffff00000800d000
[ 2.802827] x23: 0000000000000000 x22: 0000000000001000
[ 2.818140] x21: ffff80003c1256f0 x20: 0000000000000000
[ 2.821090] x19: ffff80003c125400 x18: ffff0000091696c8
[ 2.822412] x17: 0000000000000000 x16: 0000000000000000
[ 2.823657] x15: 0000000000000001 x14: ffffffffffffffff
[ 2.824826] x13: 0000000000000000 x12: 0101010101010101
[ 2.826219] x11: 0000000000000020 x10: 0101010101010101
[ 2.828012] x9 : 0000000000000000 x8 : ffff80003c108c80
[ 2.829810] x7 : 0000000000000000 x6 : 0000000000000004
[ 2.849524] x5 : 0000000000000001 x4 : ffff80003c920000
[ 2.850898] x3 : 0000000000000000 x2 : ffff00000800dfe0
[ 2.854982] x1 : ffff80003c920000 x0 : 0000000000000000
[ 2.856240] Process swapper/0 (pid: 1, stack limit = 0x(____ptrval____))
[ 2.857992] Call trace:
[ 2.858729] amba_device_try_add+0x150/0x2a8
[ 2.859697] amba_device_add+0x18/0xd0
[ 2.860546] of_platform_bus_create+0x2e8/0x370
[ 2.861588] of_platform_populate+0x50/0xc8
[ 2.862639] of_platform_default_populate_init+0xac/0xc8
[ 2.864555] do_one_initcall+0x54/0x158
[ 2.865771] kernel_init_freeable+0x188/0x224
[ 2.881225] kernel_init+0x10/0x100
[ 2.882137] ret_from_fork+0x10/0x18
[ 2.883266] Code: 35000940 d10082c2 52800000 8b020302 (88dffc41)
[ 2.885516] ---[ end trace fccca99c319bfe50 ]---
[ 2.889573] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[ 2.895687] SMP: stopping secondary CPUs
[ 2.944175] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---
It seems that something happens in the procedure of loading AMBA. The kernel booted smoothly when I simply run Qemu 6.0.0 and busybox:
qemu-system-aarch64 -machine virt,gic-version=3 -machine virtualization=true -cpu cortex-a57 -machine type=virt -m 4096 -smp 4 -kernel Image.gz -nographic -no-reboot -initrd rootfs.cpio.gz -append "rw root=/dev/ram rdinit=/sbin/init earlyprintk=serial,ttyAMA0 console=ttyAMA0"
So I think the kernel and rootfs is OK. And with U-boot, it could also boot until loading VFS (AMBA loaded). Then I tried to regenerate the device tree blob file:
qemu-system-aarch64-600 -machine virt,gic-version=3 -machine virtualization=true -cpu cortex-a57 -machine type=virt -m 4096 -smp 4 -display none -machine dumpdtb=virt-gicv3.dtb
# qemu-system-aarch64: info: dtb dumped to virt-gicv3.dtb. Exiting.
But the same problem happens. Could someone help me with this problem?
You have to disable ARM PL061. PL061 is an Advanced Microcontroller Bus Architecture (AMBA) compliant controller.
At least that's what Xen tests do.