Release the brakes on your virtual servers

Step up the pace

One of the dirty little secrets of virtualisation is the performance cost: operating systems running inside a virtual machine are slower than those running natively on the same hardware, sometimes by quite some margin.

This is termed virtualisation overhead, and with current whole-system virtualisation, it's a given. It always happens. The question is, how much.

Back in the days of ESX Server 3, VMware itself admitted that integer performance suffered an overhead of up to six percent and more complex CPU operations up to 18 per cent. It claimed that Xen 3 was about twice as bad.

These days, things are not so serious, and the performance differential between the main hypervisor vendors has mostly evened out.

Stiff competition

Even so, bear in mind that this was on a single host with a single virtual machine. Contention for shared resources is also a significant issue, especially when it comes to disk storage where virtual machines often share a single drive or array.

Most current virtualisation on x86 is whole-system virtualisation: each virtual machine is a complete emulated PC containing a complete PC operating system. The virtual machine's "disks" are actually files in a file system managed by a different operating system.

All the componentry of that nice uniform hardware platform that means you can move virtual machines from host to host – network cards, motherboard chipset, graphics adaptor and so on – is not nice fast hardware, it is software emulations running as part of the hypervisor.

Hardware extensions

Gradually, x86 chips are acquiring hardware extensions to assist in this emulation. The first generation of hardware virtualisation assist extensions was Intel's VT, introduced in some of the last models of Pentium 4, the 662 and 672, in 2005. AMD-V followed with the Socket AM2 Athlons in 2006.

This hardware virtualisation merely allowed hypervisors to create a "Ring minus-one" – essentially, trapping Ring 0 (kernel-mode) code and running it though a software emulator. The CPU-intensive process of mapping virtual machines' memory to the host's physical memory still had to be done in software.

This changed in 2007 with the arrival of AMD's second wave of hardware virtualisation, Rapid Virtualization Indexing (RVI), in the Barcelona generation of Athlons and Opterons. RVI provides the hypervisor with shadow page tables in hardware.

Page tables hold the map that translates addresses in an operating system’s memory layout to physical RAM addresses. But from inside a virtual machine, these emulated physical addresses are actually blocks of the host's RAM. RVI's second level of indirection accelerates the translation of memory addresses inside virtual machines to real physical memory addresses.

This makes little difference to a virtual machine’s pure CPU performance, but significantly enhances memory-intensive workloads, to the tune of 42 to 48 per cent.

Intel's equivalent is called Extended Page Table and appeared with the Nehalem-family Core i3, i5 and i7 processors in late 2008.

For the moment, this is all hardware can do to help. The remaining techniques are a matter of software and system configuration.

In praise of paravirtualisation

Emulation is expensive so another good way to boost performance is to avoid it. From the virtual machine perspective, one way to do this is to modify the guest operating system or its drivers to be aware that they are running in a hypervisor.

For example, a guest operating system can be provided with special drivers that talk directly to the virtual network connecting the virtual machines to the host machine, rather than emulating a physical network card.

Similar methods can be applied to storage (such as SCSI and iSCSI), graphics, input devices and even memory management.

Microsoft calls this Enlightened I/O for its Hyper-V Server; support is built into Vista SP1, Windows Server 2008 and later, and drivers are available for Windows Server 2003, SUSE Linux Enterprise Server 10 SP3 and Red Hat Enterprise Linux 5.2 to 5.5.

VMware had a similar approach, the Virtual Machine Interface, which allowed Linux guests to communicate with the hypervisor, but this has now been outpaced by hardware virtualisation.

VMware also offers the vmxnet virtual NIC, as well as enhanced vmxnet2 and vmxnet3 drivers that offer TCP Offload Engine acceleration to virtual machines on suitably equipped hosts.

For Xen, there are PV drivers and for KVM, Virtio, which both offer analagous functionality.

Just passing through

The final, and in some ways most drastic, step is to avoid the emulation overhead by directly connecting virtual machines to physical hardware.

The simplest and theoretically cleanest way of doing this is by offloading storage, for instance, to a SAN; a virtual machine accessing a SAN in principle suffers no more slowdown than a physical server would.

As in the case of a physical server accessing storage over the network, though, this ideally means dedicating network interfaces to storage – which may mean adding multiple network interface cards to the host and configuring dedicated routes between virtual machines and networked storage devices.

Hyper-V also supports pass-through disks, where a virtual machine can directly control a dedicated LUN of a storage device on the host machine.

Windows Server 2008 R2 adds a new feature, Cluster Shared Volumes, which allows multiple hosts to share access to a single storage LUN, adding a degree of scalability to pass-through disks.

VMware currently takes the prize in this department, though, with its ability to directly dedicate not only SCSI controllers, but as of ESX 4, entire physical PCI and PCIe devices to a specific virtual machine.

The VMDirectPath feature allows one or two PCI cards in the host machine to be connected to the operating sysem running in a specific virtual machine rather than being managed by the hypervisor itself – from a simple USB controller to a dedicated physical NIC or storage device.

A slightly more modest optimisation is to place the swapfiles of Windows virtual machines directly onto the host's vmfs storage.

There is always a price to pay, though, and this one is a biggie. There is a significant drawback in attaching dedicated devices to virtual machines, whether they are just disk partitions on the host server or physical interface cards and any attached devices.

Although such techniques can deliver pretty much full native performance, they hinder some of the key advantages of virtualisation: the ability to snapshot virtual machines for backup purposes, duplicate them and migrate them from one host to another.

At best, virtual machines accessing external, non-virtual resources often need to be shut down before migration, and snapshots must also duplicate any external resources – thus removing the scalability and fault-tolerance of virtualisation.

Differences of opinion

Virtualisation is now a key part of the x86 platform and it is not going to go away again. Further hardware advances will continue to improve speed and reduce overhead but there's a long way to go before x86 servers can match the performance and scalability features of systems that have been doing virtualisation for decades, such as IBM's System z mainframes.

On the other hand, paravirtualisation is also important. There are significant gains from having guest operating systems that know they are guests and can request services from the host server or its assistance in performing demanding operations.

Some of the possible improvements will probably remain limited by competitive demands, such as the different virtual machine formats of all the main hypervisor vendors and their totally different driver architectures.

Historically, such issues have receded either when everyone becomes compatible with Microsoft's formats or the functionality moves into hardware.

That is still some way off for x86 virtualisation but it is a rapidly developing area, so watch this space. ®

Other stories you might like

  • Prisons transcribe private phone calls with inmates using speech-to-text AI

    Plus: A drug designed by machine learning algorithms to treat liver disease reaches human clinical trials and more

    In brief Prisons around the US are installing AI speech-to-text models to automatically transcribe conversations with inmates during their phone calls.

    A series of contracts and emails from eight different states revealed how Verus, an AI application developed by LEO Technologies and based on a speech-to-text system offered by Amazon, was used to eavesdrop on prisoners’ phone calls.

    In a sales pitch, LEO’s CEO James Sexton told officials working for a jail in Cook County, Illinois, that one of its customers in Calhoun County, Alabama, uses the software to protect prisons from getting sued, according to an investigation by the Thomson Reuters Foundation.

    Continue reading
  • Battlefield 2042: Please don't be the death knell of the franchise, please don't be the death knell of the franchise

    Another terrible launch, but DICE is already working on improvements

    The RPG Greetings, traveller, and welcome back to The Register Plays Games, our monthly gaming column. Since the last edition on New World, we hit level cap and the "endgame". Around this time, item duping exploits became rife and every attempt Amazon Games made to fix it just broke something else. The post-level 60 "watermark" system for gear drops is also infuriating and tedious, but not something we were able to address in the column. So bear these things in mind if you were ever tempted. On that note, it's time to look at another newly released shit show – Battlefield 2042.

    I wanted to love Battlefield 2042, I really did. After the bum note of the first-person shooter (FPS) franchise's return to Second World War theatres with Battlefield V (2018), I stupidly assumed the next entry from EA-owned Swedish developer DICE would be a return to form. I was wrong.

    The multiplayer military FPS market is dominated by two forces: Activision's Call of Duty (COD) series and EA's Battlefield. Fans of each franchise are loyal to the point of zealotry with little crossover between player bases. Here's where I stand: COD jumped the shark with Modern Warfare 2 in 2009. It's flip-flopped from WW2 to present-day combat and back again, tried sci-fi, and even the Battle Royale trend with the free-to-play Call of Duty: Warzone (2020), which has been thoroughly ruined by hackers and developer inaction.

    Continue reading
  • American diplomats' iPhones reportedly compromised by NSO Group intrusion software

    Reuters claims nine State Department employees outside the US had their devices hacked

    The Apple iPhones of at least nine US State Department officials were compromised by an unidentified entity using NSO Group's Pegasus spyware, according to a report published Friday by Reuters.

    NSO Group in an email to The Register said it has blocked an unnamed customers' access to its system upon receiving an inquiry about the incident but has yet to confirm whether its software was involved.

    "Once the inquiry was received, and before any investigation under our compliance policy, we have decided to immediately terminate relevant customers’ access to the system, due to the severity of the allegations," an NSO spokesperson told The Register in an email. "To this point, we haven’t received any information nor the phone numbers, nor any indication that NSO’s tools were used in this case."

    Continue reading

Biting the hand that feeds IT © 1998–2021