接前一篇文章: QEMU源码全解析 —— 内存虚拟化(1)
本文内容参考:
《 QEMU /KVM》源码解析与应用 —— 李强,机械工业出版社
特此致谢!
QEMU 内存初始化
1. 基本结构
在开始介绍内存初始化的时候,首先需要对QEMU中几个与内存相关的数据结构进行介绍。
(1)AddressSpace
首先是AddressSpace结构。 AddressSpace结构用来表示一个虚拟机或者虚拟CPU能够访问的所有物理地址 。注意,这里的“访问”和“能够访问”并不是一回事。与进程的地址空间一样,一个进程的虚拟地址空间为4GB(32位),但这并不是说操作系统需要为进程分配这么大的空间。同样,QEMU中的AddressSpace表示的是一段地址空间,整个系统可以有一个全局的地址空间,CPU可以有自己的地址空间视角,设备也可以有自己的地址空间视角。
AddressSpace的意义从名字就可以看出,是针对整个地址空间,最顶层的内存管理结构。对于x86架构来说,其实是有两种寻址方式的,一种是memory、一种是IO。相应的QEMU会维护两个AddressSpace结构,address_space_memory和address_space_io。
AddressSpace的定义在include/qemu/typedefs.h中,如下:
typedef struct AddressSpace AddressSpace;
而struct AddressSpace的定义在include/exec/memory.h中,代码如下:
- /**
- * struct AddressSpace: describes a mapping of addresses to #MemoryRegion objects
- */
- struct AddressSpace {
- /* private: */
- struct rcu_head rcu;
- char *name;
- MemoryRegion *root;
-
- /* Accessed via RCU. */
- struct FlatView *current_map;
-
- int ioeventfd_nb;
- struct MemoryRegionIoeventfd *ioeventfds;
- QTAILQ_HEAD(, MemoryListener) listeners;
- QTAILQ_ENTRY(AddressSpace) address_spaces_link;
- };
根据struct AddressSpace的注释,此结构描述了一个到MemoryRegion对象的地址映射。
其中:
- MemoryRegion *root成员表示AddressSpace对应的一个根MemoryRegion。
- struct FlatView *current_map成员表示该地址空间是一个平坦模式下的视图。
- QEMU的其它子系统可以注册地址变更的事件,所有注册的信息都通过QTAILQ_HEAD(, MemoryListener) listeners成员连接起来。
- 所有的AddressSpace通过address_spaces_link这个node连接起来(QTAILQ_ENTRY(AddressSpace) address_spaces_link),链表头是address_spaces。
AddressSpace的定义其实比较简单,有两个QEMU内存管理框架的核心结构,就是MemoryRegion和FlatView,两者配合使用,MemoryRegion是基础(MemoryRegion),用于正向管理;Flatview用于反向查找。
对于一个地址空间(AddressSpace),会有多个内存区域(MemoryRegion)组成树型结构。这其中,root是这棵树的根。另外,还有一个MemoryListener链表,当内存区域发生变化的时候,需要做一些动作,使得用户态和内核态能够协同,就是由这些MemoryListener来完成的。
在QEMU monitor中输入“info qtree”,可以看到所有的AddressSpace,如下所示:
- $ qemu-system-x86_64 -vnc :1 -monitor stdio
- QEMU 6.2.0 monitor - type 'help' for more information
- (qemu) info mtree
- address-space: memory
- 0000000000000000-ffffffffffffffff (prio 0, i/o): system
- 0000000000000000-0000000007ffffff (prio 0, ram): alias ram-below-4g @pc.ram 0000000000000000-0000000007ffffff
- 0000000000000000-ffffffffffffffff (prio -1, i/o): pci
- 00000000000a0000-00000000000bffff (prio 1, i/o): vga-lowmem
- 00000000000c0000-00000000000dffff (prio 1, rom): pc.rom
- 00000000000e0000-00000000000fffff (prio 1, rom): alias isa-bios @pc.bios 0000000000020000-000000000003ffff
- 00000000fd000000-00000000fdffffff (prio 1, ram): vga.vram
- 00000000feb80000-00000000feb9ffff (prio 1, i/o): e1000-mmio
- 00000000febb0000-00000000febb0fff (prio 1, i/o): vga.mmio
- 00000000febb0000-00000000febb017f (prio 0, i/o): edid
- 00000000febb0400-00000000febb041f (prio 0, i/o): vga ioports remapped
- 00000000febb0500-00000000febb0515 (prio 0, i/o): bochs dispi interface
- 00000000febb0600-00000000febb0607 (prio 0, i/o): qemu extended regs
- 00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios
- 00000000000a0000-00000000000bffff (prio 1, i/o): alias smram-region @pci 00000000000a0000-00000000000bffff
- 00000000000c0000-00000000000c3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000c0000-00000000000c3fff
- 00000000000c4000-00000000000c7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000c4000-00000000000c7fff
- 00000000000c8000-00000000000cbfff (prio 1, ram): alias pam-rom @pc.ram 00000000000c8000-00000000000cbfff
- 00000000000cb000-00000000000cdfff (prio 1000, ram): alias kvmvapic-rom @pc.ram 00000000000cb000-00000000000cdfff
- 00000000000cc000-00000000000cffff (prio 1, ram): alias pam-rom @pc.ram 00000000000cc000-00000000000cffff
- 00000000000d0000-00000000000d3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000d0000-00000000000d3fff
- 00000000000d4000-00000000000d7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000d4000-00000000000d7fff
- 00000000000d8000-00000000000dbfff (prio 1, ram): alias pam-rom @pc.ram 00000000000d8000-00000000000dbfff
- 00000000000dc000-00000000000dffff (prio 1, ram): alias pam-rom @pc.ram 00000000000dc000-00000000000dffff
- 00000000000e0000-00000000000e3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000e0000-00000000000e3fff
- 00000000000e4000-00000000000e7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000e4000-00000000000e7fff
- 00000000000e8000-00000000000ebfff (prio 1, ram): alias pam-ram @pc.ram 00000000000e8000-00000000000ebfff
- 00000000000ec000-00000000000effff (prio 1, ram): alias pam-ram @pc.ram 00000000000ec000-00000000000effff
- 00000000000f0000-00000000000fffff (prio 1, ram): alias pam-rom @pc.ram 00000000000f0000-00000000000fffff
- 00000000fec00000-00000000fec00fff (prio 0, i/o): ioapic
- 00000000fed00000-00000000fed003ff (prio 0, i/o): hpet
- 00000000fee00000-00000000feefffff (prio 4096, i/o): apic-msi
-
- address-space: I/O
- 0000000000000000-000000000000ffff (prio 0, i/o): io
- 0000000000000000-0000000000000007 (prio 0, i/o): dma-chan
- 0000000000000008-000000000000000f (prio 0, i/o): dma-cont
- 0000000000000020-0000000000000021 (prio 0, i/o): pic
- 0000000000000040-0000000000000043 (prio 0, i/o): pit
- 0000000000000060-0000000000000060 (prio 0, i/o): i8042-data
- 0000000000000061-0000000000000061 (prio 0, i/o): pcspk
- 0000000000000064-0000000000000064 (prio 0, i/o): i8042-cmd
- 0000000000000070-0000000000000071 (prio 0, i/o): rtc
- 0000000000000070-0000000000000070 (prio 0, i/o): rtc-index
- 000000000000007e-000000000000007f (prio 0, i/o): kvmvapic
- 0000000000000080-0000000000000080 (prio 0, i/o): ioport80
- 0000000000000081-0000000000000083 (prio 0, i/o): dma-page
- 0000000000000087-0000000000000087 (prio 0, i/o): dma-page
- 0000000000000089-000000000000008b (prio 0, i/o): dma-page
- 000000000000008f-000000000000008f (prio 0, i/o): dma-page
- 0000000000000092-0000000000000092 (prio 0, i/o): port92
- 00000000000000a0-00000000000000a1 (prio 0, i/o): pic
- 00000000000000b2-00000000000000b3 (prio 0, i/o): apm-io
- 00000000000000c0-00000000000000cf (prio 0, i/o): dma-chan
- 00000000000000d0-00000000000000df (prio 0, i/o): dma-cont
- 00000000000000f0-00000000000000f0 (prio 0, i/o): ioportF0
- 0000000000000170-0000000000000177 (prio 0, i/o): ide
- 00000000000001ce-00000000000001d1 (prio 0, i/o): vbe
- 00000000000001f0-00000000000001f7 (prio 0, i/o): ide
- 0000000000000376-0000000000000376 (prio 0, i/o): ide
- 0000000000000378-000000000000037f (prio 0, i/o): parallel
- 00000000000003b4-00000000000003b5 (prio 0, i/o): vga
- 00000000000003ba-00000000000003ba (prio 0, i/o): vga
- 00000000000003c0-00000000000003cf (prio 0, i/o): vga
- 00000000000003d4-00000000000003d5 (prio 0, i/o): vga
- 00000000000003da-00000000000003da (prio 0, i/o): vga
- 00000000000003f1-00000000000003f5 (prio 0, i/o): fdc
- 00000000000003f6-00000000000003f6 (prio 0, i/o): ide
- 00000000000003f7-00000000000003f7 (prio 0, i/o): fdc
- 00000000000003f8-00000000000003ff (prio 0, i/o): serial
- 00000000000004d0-00000000000004d0 (prio 0, i/o): elcr
- 00000000000004d1-00000000000004d1 (prio 0, i/o): elcr
- 0000000000000510-0000000000000511 (prio 0, i/o): fwcfg
- 0000000000000514-000000000000051b (prio 0, i/o): fwcfg.dma
- 0000000000000600-000000000000063f (prio 0, i/o): piix4-pm
- 0000000000000600-0000000000000603 (prio 0, i/o): acpi-evt
- 0000000000000604-0000000000000605 (prio 0, i/o): acpi-cnt
- 0000000000000608-000000000000060b (prio 0, i/o): acpi-tmr
- 0000000000000700-000000000000073f (prio 0, i/o): pm-smbus
- 0000000000000cf8-0000000000000cfb (prio 0, i/o): pci-conf-idx
- 0000000000000cf9-0000000000000cf9 (prio 1, i/o): piix3-reset-control
- 0000000000000cfc-0000000000000cff (prio 0, i/o): pci-conf-data
- 0000000000005658-0000000000005658 (prio 0, i/o): vmport
- 000000000000ae00-000000000000ae17 (prio 0, i/o): acpi-pci-hotplug
- 000000000000af00-000000000000af1f (prio 0, i/o): acpi-cpu-hotplug
- 000000000000afe0-000000000000afe3 (prio 0, i/o): acpi-gpe0
- 000000000000c000-000000000000c03f (prio 1, i/o): e1000-io
- 000000000000c040-000000000000c04f (prio 1, i/o): piix-bmdma-container
- 000000000000c040-000000000000c043 (prio 0, i/o): piix-bmdma
- 000000000000c044-000000000000c047 (prio 0, i/o): bmdma
- 000000000000c048-000000000000c04b (prio 0, i/o): piix-bmdma
- 000000000000c04c-000000000000c04f (prio 0, i/o): bmdma
-
- address-space: cpu-memory-0
- 0000000000000000-ffffffffffffffff (prio 0, i/o): system
- 0000000000000000-0000000007ffffff (prio 0, ram): alias ram-below-4g @pc.ram 0000000000000000-0000000007ffffff
- 0000000000000000-ffffffffffffffff (prio -1, i/o): pci
- 00000000000a0000-00000000000bffff (prio 1, i/o): vga-lowmem
- 00000000000c0000-00000000000dffff (prio 1, rom): pc.rom
- 00000000000e0000-00000000000fffff (prio 1, rom): alias isa-bios @pc.bios 0000000000020000-000000000003ffff
- 00000000fd000000-00000000fdffffff (prio 1, ram): vga.vram
- 00000000feb80000-00000000feb9ffff (prio 1, i/o): e1000-mmio
- 00000000febb0000-00000000febb0fff (prio 1, i/o): vga.mmio
- 00000000febb0000-00000000febb017f (prio 0, i/o): edid
- 00000000febb0400-00000000febb041f (prio 0, i/o): vga ioports remapped
- 00000000febb0500-00000000febb0515 (prio 0, i/o): bochs dispi interface
- 00000000febb0600-00000000febb0607 (prio 0, i/o): qemu extended regs
- 00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios
- 00000000000a0000-00000000000bffff (prio 1, i/o): alias smram-region @pci 00000000000a0000-00000000000bffff
- 00000000000c0000-00000000000c3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000c0000-00000000000c3fff
- 00000000000c4000-00000000000c7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000c4000-00000000000c7fff
- 00000000000c8000-00000000000cbfff (prio 1, ram): alias pam-rom @pc.ram 00000000000c8000-00000000000cbfff
- 00000000000cb000-00000000000cdfff (prio 1000, ram): alias kvmvapic-rom @pc.ram 00000000000cb000-00000000000cdfff
- 00000000000cc000-00000000000cffff (prio 1, ram): alias pam-rom @pc.ram 00000000000cc000-00000000000cffff
- 00000000000d0000-00000000000d3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000d0000-00000000000d3fff
- 00000000000d4000-00000000000d7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000d4000-00000000000d7fff
- 00000000000d8000-00000000000dbfff (prio 1, ram): alias pam-rom @pc.ram 00000000000d8000-00000000000dbfff
- 00000000000dc000-00000000000dffff (prio 1, ram): alias pam-rom @pc.ram 00000000000dc000-00000000000dffff
- 00000000000e0000-00000000000e3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000e0000-00000000000e3fff
- 00000000000e4000-00000000000e7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000e4000-00000000000e7fff
- 00000000000e8000-00000000000ebfff (prio 1, ram): alias pam-ram @pc.ram 00000000000e8000-00000000000ebfff
- 00000000000ec000-00000000000effff (prio 1, ram): alias pam-ram @pc.ram 00000000000ec000-00000000000effff
- 00000000000f0000-00000000000fffff (prio 1, ram): alias pam-rom @pc.ram 00000000000f0000-00000000000fffff
- 00000000fec00000-00000000fec00fff (prio 0, i/o): ioapic
- 00000000fed00000-00000000fed003ff (prio 0, i/o): hpet
- 00000000fee00000-00000000feefffff (prio 4096, i/o): apic-msi
-
- address-space: cpu-smm-0
- 0000000000000000-ffffffffffffffff (prio 0, i/o): memory
- 0000000000000000-00000000ffffffff (prio 1, i/o): alias smram @smram 0000000000000000-00000000ffffffff
- 0000000000000000-ffffffffffffffff (prio 0, i/o): alias memory @system 0000000000000000-ffffffffffffffff
-
- address-space: i440FX
- 0000000000000000-ffffffffffffffff (prio 0, i/o): bus master container
-
- address-space: PIIX3
- 0000000000000000-ffffffffffffffff (prio 0, i/o): bus master container
-
- address-space: VGA
- 0000000000000000-ffffffffffffffff (prio 0, i/o): bus master container
-
- address-space: e1000
- 0000000000000000-ffffffffffffffff (prio 0, i/o): bus master container
- 0000000000000000-ffffffffffffffff (prio 0, i/o): alias bus master @system 0000000000000000-ffffffffffffffff
-
- address-space: piix3-ide
- 0000000000000000-ffffffffffffffff (prio 0, i/o): bus master container
-
- address-space: PIIX4_PM
- 0000000000000000-ffffffffffffffff (prio 0, i/o): bus master container
-
- memory-region: pc.ram
- 0000000000000000-0000000007ffffff (prio 0, ram): pc.ram
-
- memory-region: pc.bios
- 00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios
-
- memory-region: pci
- 0000000000000000-ffffffffffffffff (prio -1, i/o): pci
- 00000000000a0000-00000000000bffff (prio 1, i/o): vga-lowmem
- 00000000000c0000-00000000000dffff (prio 1, rom): pc.rom
- 00000000000e0000-00000000000fffff (prio 1, rom): alias isa-bios @pc.bios 0000000000020000-000000000003ffff
- 00000000fd000000-00000000fdffffff (prio 1, ram): vga.vram
- 00000000feb80000-00000000feb9ffff (prio 1, i/o): e1000-mmio
- 00000000febb0000-00000000febb0fff (prio 1, i/o): vga.mmio
- 00000000febb0000-00000000febb017f (prio 0, i/o): edid
- 00000000febb0400-00000000febb041f (prio 0, i/o): vga ioports remapped
- 00000000febb0500-00000000febb0515 (prio 0, i/o): bochs dispi interface
- 00000000febb0600-00000000febb0607 (prio 0, i/o): qemu extended regs
- 00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios
-
- memory-region: smram
- 0000000000000000-00000000ffffffff (prio 0, i/o): smram
- 00000000000a0000-00000000000bffff (prio 0, ram): alias smram-low @pc.ram 00000000000a0000-00000000000bffff
-
- memory-region: system
- 0000000000000000-ffffffffffffffff (prio 0, i/o): system
- 0000000000000000-0000000007ffffff (prio 0, ram): alias ram-below-4g @pc.ram 0000000000000000-0000000007ffffff
- 0000000000000000-ffffffffffffffff (prio -1, i/o): pci
- 00000000000a0000-00000000000bffff (prio 1, i/o): vga-lowmem
- 00000000000c0000-00000000000dffff (prio 1, rom): pc.rom
- 00000000000e0000-00000000000fffff (prio 1, rom): alias isa-bios @pc.bios 0000000000020000-000000000003ffff
- 00000000fd000000-00000000fdffffff (prio 1, ram): vga.vram
- 00000000feb80000-00000000feb9ffff (prio 1, i/o): e1000-mmio
- 00000000febb0000-00000000febb0fff (prio 1, i/o): vga.mmio
- 00000000febb0000-00000000febb017f (prio 0, i/o): edid
- 00000000febb0400-00000000febb041f (prio 0, i/o): vga ioports remapped
- 00000000febb0500-00000000febb0515 (prio 0, i/o): bochs dispi interface
- 00000000febb0600-00000000febb0607 (prio 0, i/o): qemu extended regs
- 00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios
- 00000000000a0000-00000000000bffff (prio 1, i/o): alias smram-region @pci 00000000000a0000-00000000000bffff
- 00000000000c0000-00000000000c3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000c0000-00000000000c3fff
- 00000000000c4000-00000000000c7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000c4000-00000000000c7fff
- 00000000000c8000-00000000000cbfff (prio 1, ram): alias pam-rom @pc.ram 00000000000c8000-00000000000cbfff
- 00000000000cb000-00000000000cdfff (prio 1000, ram): alias kvmvapic-rom @pc.ram 00000000000cb000-00000000000cdfff
- 00000000000cc000-00000000000cffff (prio 1, ram): alias pam-rom @pc.ram 00000000000cc000-00000000000cffff
- 00000000000d0000-00000000000d3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000d0000-00000000000d3fff
- 00000000000d4000-00000000000d7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000d4000-00000000000d7fff
- 00000000000d8000-00000000000dbfff (prio 1, ram): alias pam-rom @pc.ram 00000000000d8000-00000000000dbfff
- 00000000000dc000-00000000000dffff (prio 1, ram): alias pam-rom @pc.ram 00000000000dc000-00000000000dffff
- 00000000000e0000-00000000000e3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000e0000-00000000000e3fff
- 00000000000e4000-00000000000e7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000e4000-00000000000e7fff
- 00000000000e8000-00000000000ebfff (prio 1, ram): alias pam-ram @pc.ram 00000000000e8000-00000000000ebfff
- 00000000000ec000-00000000000effff (prio 1, ram): alias pam-ram @pc.ram 00000000000ec000-00000000000effff
- 00000000000f0000-00000000000fffff (prio 1, ram): alias pam-rom @pc.ram 00000000000f0000-00000000000fffff
- 00000000fec00000-00000000fec00fff (prio 0, i/o): ioapic
- 00000000fed00000-00000000fed003ff (prio 0, i/o): hpet
- 00000000fee00000-00000000feefffff (prio 4096, i/o): apic-msi
-
- address-space: memory
第一个是系统全局的AddressSpace,表示 虚拟机 能够访问的所有地址。
- address-space: I/O
I/O表示x86系统下I/O端口的地址空间。
- address-space: cpu-memory-0
cpu-memory-x表示CPU视角下的地址空间。
- address-space: i440FX和address-space: PIIX3
i440FX和PIIX3是设备视角下的地址空间,虽然大部分情况下很多都是相同的,但是逻辑意义并不一样。
至此,QEMU中与内存相关的第一个数据结构AddressSpace就讲解完了。
(2)MemoryRegion
第二个是基本数据结构是MemoryRegion(就是上边AddressSpace中提到的)。MemoryRegion结构的定义在include/exec/memory.h中,如下:
- #define TYPE_MEMORY_REGION "memory-region"
- DECLARE_INSTANCE_CHECKER(MemoryRegion, MEMORY_REGION,
- TYPE_MEMORY_REGION)
在此不详细展开宏,而直接给出struct MemoryRegion定义,也在include/exec/memory.h中,代码如下:
- /** MemoryRegion:
- *
- * A struct representing a memory region.
- */
- struct MemoryRegion {
- Object parent_obj;
-
- /* private: */
-
- /* The following fields should fit in a cache line */
- bool romd_mode;
- bool ram;
- bool subpage;
- bool readonly; /* For RAM regions */
- bool nonvolatile;
- bool rom_device;
- bool flush_coalesced_mmio;
- uint8_t dirty_log_mask;
- bool is_iommu;
- RAMBlock *ram_block;
- Object *owner;
- /* owner as TYPE_DEVICE. Used for re-entrancy checks in MR access hotpath */
- DeviceState *dev;
-
- const MemoryRegionOps *ops;
- void *opaque;
- MemoryRegion *container;
- int mapped_via_alias; /* Mapped via an alias, container might be NULL */
- Int128 size;
- hwaddr addr;
- void (*destructor)(MemoryRegion *mr);
- uint64_t align;
- bool terminates;
- bool ram_device;
- bool enabled;
- bool warning_printed; /* For reservations */
- uint8_t vga_logging_count;
- MemoryRegion *alias;
- hwaddr alias_offset;
- int32_t priority;
- QTAILQ_HEAD(, MemoryRegion) subregions;
- QTAILQ_ENTRY(MemoryRegion) subregions_link;
- QTAILQ_HEAD(, CoalescedMemoryRange) coalesced;
- const char *name;
- unsigned ioeventfd_nb;
- MemoryRegionIoeventfd *ioeventfds;
- RamDiscardManager *rdm; /* Only for RAM */
-
- /* For devices designed to perform re-entrant IO into their own IO MRs */
- bool disable_reentrancy_guard;
- };
MemoryRegion表示的是虚拟机中的一段内存区域 (对比:AddressSpace结构用来表示一个虚拟机或者虚拟CPU能够访问的所有物理地址,其描述了一个到MemoryRegion对象的地址映射)。 MemoryRegion是内存模拟中的核心结构 , 整个内存的模拟都是通过MemoryRegion构成的无环图完成的 。图的 叶子节点是实际分配给虚拟机的物理内存或者MMIO , 中间节点则表示内存总线 ,内存控制器是其它MemoryRegion的别名。AddressSpace与MemoryRegion的关系如下图所示:
对于MemoryRegion各成员的详细讲解,请看下回。