XEN Faq General ------- Q: What is Xen? A: Xen is a platform that allows you to run multiple operating systems running in virtual machines on one physical box of hardware. Q: How fast is Xen? A: Xen seems to be one of the top performing virtual servers. Benchmarks against older versions of Xen can be found at http://www.cl.cam.ac.uk/netos/xen/performance.html Q: What's the difference between Xen and other virtualization products? A: The main difference is that operating systems running on Xen know they're running in a VM, and not on the raw iron. The OS has to be ported to the Xeno "architecture" as if it was another type chipset. The big win is that the guest OS' can be optimized for the VM, which greatly improves the speed. Here's a longer list of pros and cons of different VM products: xen -> pros: - near to "mainframe" architecture - very high speed - high scalability with very good resource isolation - fine grained resource allocation and QOS guarantees for guest OS' - high stability - application compatibility very high - open source - helpful and responsive developers xen -> cons: - guess OS' have to be ported to the Xen architecture - relatively large amount of codelines to patch linux kernels, but mostly in the architecture dependent code - currently has less resource sharing/saver features like UML CoW sparse virtual disks, but these are under development - currently only has support for maximum 4GB memory, but the x86_64 port will fix this virtuosso -> pros: - very high speed - high stability - manageability through webapplications - SMP for virtual servers - up to 64 GB RAM virtuosso -> cons: - not open source - customized linux kernels - fault tolerance between virtual servers - security between virtual servers user-mode-linux -> pros: - open source - production quality - a relative large user community - a "official" linux architecture - good manageability (cow-files, sparse-files, flexible device-model) - security between virtual servers (but shared host OS) - fault tolerance between virtual machines (but shared host OS) user-mode-linux -> cons: - a very high context switching rate - relative low performance block devices (no "raw" access) - resource consumption up to 80 percent and more - no SMP for virtual servers - performance bottlenecks with high RAM servers - small host OS patching required - no resource QOS, especially noticeable under heavy disk I/O vmware -> pros: - very stable system - high quality software for datacenter users - SMP (for ESX Server) - installation of different native OS - security between virtual machines (but shared host OS) - fault tolerance between virtual servers (but shared host OS) vmware -> cons: - resource consumption up to 50 percent and more - special drivers for high performance required - very high price (GSX and ESX Server???) - closed source vserver -> pros: - production quality - a relative large user community - very high performance - good manageability, many tools for production use - open source vserver -> cons: - fault tolerance between virtual servers (shared host OS) - security between virtual servers (shared host OS) - relative high amount of codelines to patch linux kernels Q: Do you have a mailing list? A: Yes, see http://lists.sourceforge.net/lists/listinfo/xen-devel for subscription and archive information. Q: How do you want outside patches? A: Patchsets to xen-devel is the best way to send updates. See http://lists.sourceforge.net/lists/listinfo/xen-devel Q: Where can I find the Xen web site? A: http://www.cl.cam.ac.uk/netos/xen/ Q: Where can I find more documentation on Xen? A: A very detailed overview of Xen's design is here :http://www.cl.cam.ac.uk/netos/papers/2003-xensosp.pdf A couple of READMEs are available here: http://www.cl.cam.ac.uk/netos/xen/readmes/ A nice HOWTO is at: http://www.cl.cam.ac.uk/users/br260/doc/xeno-1.1-howto.txt Q: What is Xen's relationship with 'Xen' the programming language that unites C#, XML and SQL programming languages. (http://www.microsoft-watch.com/article2/0,4248,1441416,00.asp) A: None. They've got nothing to do with each other. Q: What is a domain? A: In our terminology, Virtual Machines run in domains, and this VM could end up being assigned any domain id -- if the VM reboots, it'll certainly get a different domain id. We rather regret that we didn't just assign domain ids pseudo randomly, just so people didn't write scripts that rely on guessing the id. Q: How isolated/secure is Xen virtual domains? A: Xen has the explicit goal of providing absolute resource isolation between domains. You won't, for example, be able to tcpdump on a virtual hosts and see traffic intended for other virtual hosts. Q: What operating systems are supported by Xen A: Theoretically any OS can be ported to run on Xen. At the moment only Linux and NetBSD has been ported. A port of Windows XP is in the works, but not ready yet, and it's unknown if it will be available to the public. Installing Xen -------------- Q: What version of GCC should I use to compile Xen? A: We have reason to doubt the integrity of 3.0.4 (definitely) and also 3.1.1 (quite possibly). We've had nothing but success with 3.2.2 and 3.3.2. Q: How do I install Xen? A: Howto are available at http://www.cl.cam.ac.uk/users/br260/doc/xeno-1.1-howto.txt and http://www.csc.liv.ac.uk/~u2ycc/xen/xen-install-domain-fs-howto.txt. In short however, you just have to install a normal Red Hat/Debian/Suse distribution. After using the CDs to install a file system for domain0, just drop a suitable xen.gz, xenolinux.gz and /boot/grub/menu.lst and you're ready to go. Installation of other domains can be achieved just by doing an appropriate mkfs, mount, 'cp -a' etc. Disks and file systems ---------------------- Q: I need more file system related documentation. A: Wish gratnted by Yan-Ching: http://www.csc.liv.ac.uk/~u2ycc/xen/xen-install-domain-fs-howto.txt Q: What is a VBD? A: Virtual Block Devices (VBDs) are an abstraction provided by Xen and domains that use them talk to Xen directly, without going through Dom0. VBDs are a way of securely sharing portions of the physical disk space in the system between domains - Dom0 creates VBDs to tell Xen what domains can access what parts of the disk. Those domains then talk to Xen directly to access those areas. A domain could be given read only access to the VBD with a "standard image" for all domains on it and read-write access to another VBD for it's own personal use. Q: Does Xen work with iSCSI? A: Yes, it should work. Q: Does Xen work with HyperSCSI? A: Since HyperSCSI uses raw Ethernet frames, this originally had issues for the 'Virtual Firewall Router' inside Xen, but its now believed to work. Q: Can I use as a VBD? A: Mostly no. Xen currently assumes that all vbds are backed by local disk. Q: Can I use a file and make it look like a block device to the domains? A: Currently, it's not possible to use a file as a block device for a guest domain. Hoewver, the new virtual disk management in 1.2 allows file-based disk images to be imported to Xen virtual disks and vice-versa. Virtual disks are allocated within a pool of free space made up of devices and partitions on your system that you have set aside for the purpose. You can extend this free pool when you want. Virtual disks can also be enlarged by allocating more space from the free pool. (Currently, they can't be enlarged whilst mounted by a domain and have that domain see the changes. ) Q: Can I use the same / file system for domain0 and its subdomains? A: Mostly no, but read on. Mounting the same partition read-write in two domains is guaranteed disastrous. Granting the new domain read-only access to daomin0's root file system should be safe, but the new domain is likely to get somewhat upset with having a read-only root. Furthermore if you happen to make any changes to the file system from domain0, the new domain is likely to come to the conclusion that the file system is corrupt, as it will see inconsistencies between its buffer cache and what its reading off disk. If you don't want to create a new partition for your new domain, the best way is to run an NFS server in domain0 and export a copy of the root directory to domain1. There's a rather terse example of how to do this in README.CD, drawing on the example /etc/exports file in /etc/xen-exports Q: Can I use the same /usr, /var, /homes etc file systems for domain0 and its subdomains? A: Read-only direct mount of /usr should work fine. For things like /homes, having shared read-write access via NFS should work just fine. If you use knfsd as opposed to unfsd you'll need to export each partition individually, as exports don't cross mount points (they do on unfsd with the -r option). Having per-domain copies of /var seems sensible. /tmp could be shared, but you might have filename collisions, so make them per-domain. Start by copying your entire root, /var and /tmp partitions to some NFS exportable location e.g. /exports/roots/root1 Edit the etc/fstab file to reflect the NFS / mount and read-only disk partition /usr mount. It should boot and work just fine with the appropriate kernel command line arguments. (remember to restart your nfsd after editing /etc/exports If you're feeling a bit more adventurous you can reduce the amount of disk space you need for the root1 by arranging that the /bin, /sbin and /lib directories are actually symlinks to read-only copies in /usr/root/. README.CD gives some advice on how to do this. In the mid-term, we hope to release a user-space NFS server that implements a proper stackable copy-on-write file system, but its still undergoing development. Q: Do I really need a separate / file system for all sub-domains? Almost all my / partitions look the same, with only some minor differences in some /etc and other configuration files. I don't want to have, say 5, separate disk images that are 99% the same. A: One independent root filesystem for each domain requires too much space, this is a major problem of today's Xen that we are trying to tackle. The method is to use Copy on Write (CoW), e.g. different domains share as many files as possible and retains modified files separately. CoW can be done both in user space and in kernel space. Russ has implemented a CoW NFS, and Bin Ren is at the last stages of writing a CoW multi-disk driver for linux kernel 2.4 series Before CoW features become stable and robust, you can: 1. duplicate root filesystems for each domain (Yes, I can hear you screaming...) 2. share /usr among all the domains to save a lot of space 3. get a new hard disk and go to either (1) or (2). Booting up, and shutting down ----------------------------- Q: shutdown -h now in dom0 reboots the machine instead of halting it. A: Specify 'noreboot' on the Xen command line then it will halt instead of rebooting. Q: When booting XenoLinux directly on my home desktop I get a crash at: DOM0: Guest OS virtual load address is c000000 A: Add 'console=tty0' to xenolinux command line. Otherwise console output by default is directed via Xen and thru to the serial line (COM1). If you want output to go both to screen and serial, use 'console=tty0 console=xencons0'. Creating and shutting down domains ----------------------------------- Q: I get the following error when trying to create a domain: ERROR: Error when executing privileged control ioctl (1 = Operation not permitted) A: Make sure you're running need to run xenctl as root. Also try lowering the amount of memory allocated to the domain. Q: xenctl works, and I can create a domain, but when running the newdomain script, it would fail when mapping the cdrom_link. Apparently a control file in /var/lib/xen dealing with virtual devices is missing. A: Ignore the error about the missing file in /var/lib/xen/vdstat.xml -- you only have this file if you've created virtual disks (rather than physical partitions), which I'm guessing you haven't. You should be able to touch vdstat.xml to make the warning go away, as xenctl's parser won't actually care if the file is empty. Q: I'm using physical partitions, instead of virtual disks, but I can't create sub-domains. What is wrong? A: The default /etc/xen-mynewdom script is trying to grant the new domain access to the physical partition it needs to be able to read and write. The default is 'no access'. For the CD, domains need to be able to mount the CD read-only as their /usr partition. The link called /dev/cdrom_link is created on boot and points to whatever device is the CDROM (/dev/hdc,/dev/sda etc). Once you've installed on your hard disk, you'll need to update /etc/xen-mynewdom to grant physical access as appropriate for your root partition (plus swap and anything else you want it to be able to access). e.g. to grant read/write access to /dev/sda2, do: xenctl physical grant -psda2 -w Q: Is it possible for a domain to request its destruction? A: Yes. that's what the HYPERVISOR_exit() hypercall is for. It's the preferred way to terminate a domain because it allows the guest OS to clean up first. Q: How can domain0 tell a virtual domain to gracefully shut down? A: There's a python script, xc_dom_control, that allow you to do this: xc_dom_control.py shutdown Q: Is the suspend/restore function is ready? I have tried Create->Stop->Save a domain and Restore->Start it again. The network interface and sshd seem totally down after restore, if the sshd is down there is no way to rescue it. A: The VIF state is not restored. Unfortunately, this is not currently part of the suspend record, so needs to be done manually. Networking Related ------------------ Q: After starting new domain I can only ssh to it by 'ssh root@169.254.1.x' but not as what in the README.CD of Xen 1.2 (and earlier) says to ssh with port number 2200 + domain_id, if so I would get connection refused. I have started 'xen_nat_enable' am I setting anything wrong or missing anything? A: That's a documentation bug coupled with a slight 'democd' config file bug. There's nothing special about 169.254.x.x addresses other than they never go on the LAN. You can assign a new VM any arbitrary IP address (though it should be unique!) and then be able to ssh into it from domain 0. If you run the xen_nat_enable script, some port forwarding rules are inserted to enable you to access the new domain from other machines on your LAN (assuming domain 0 has a LAN address as well as 169.254.1.0, which it gets by convention). The port forwarding rules arrange to route port 2200+x on domain0's IP to 169.254.1.x port 22 (for 1 < x < 20). If you've assigned a domain a link local IP that's in this range it should get the packets. Xen time -------- Q: My xen machines doesn't accept setting its time. Basically, it's stuck to RTC time. ntpdate, date, hwclock all "seem" to work, but they don't actually change the system time. The only way I have to change it right now is to change it in the BIOS. A: Only affects 1.0. Fixed in newer versions. Q: Where does a domain get its time from? A: Briefly, Xen reads the RTC at start of day and by default will track that with the precision of the periodic timer crystal. Xen's estimate of the wall-clock time can only be updated by domain 0. If domain 0 runs ntpdate, ntpd, etc. then the synchronised time will automatically be pushed down to Xen every minute (and written to the RTC every 11 minutes, just as normal x86 Linux does). All other domains always track Xen's wall-clock time: setting the date, or running ntpd, on these domains will not affect their wall-clock time. Note that the wall-clock time exported by Xen is UTC --- all domains must have appropriate timezone handling (i.e. a correct /etc/localtime file). Q: Is there is some cross-domain time synchronization : are they always in perfect sync, or should I run some kind of ntp in each subdomain ? Or only domain 0 would be enough ? A:If you want each domain to keep its own time, there are two ways to cause a domain to run its wallclock independently from Xen: 1. Specify 'independent_wallclock' on the command line. 2. 'echo 1 >/proc/sys/xeno/independent_wallclock' To reenable tracking of Xen wallclock: 1. 'echo 0 >/proc/sys/xeno/independent_wallclock' Programming related ------------------- Q: I'm looking for a way to intercept system calls within the guest O/S using the Xen VMM (i.e. when a user-mode application makes a system call,I want to intercept it and do something with it before it gets to the kernel of the guest O/S). A: Take a look at xen/arxh/i386/traps.c Credits ------- This FAQ is based on discussions on the Xen mailing lists. Maintained by Jan van Rensburg: xendoc@NOSPAM.epiuse.com