Virtualization over Directed I/O support

[vuldin]

I recently purchased the ASRock Z97 Extreme6 motherboard and it should be arriving around Friday. This motherboard will be replacing an Asus Z97-WS that I had purchased recently, and I had mistakenly believed that the Asus board would support Virtualization over Directed I/O (VT-d). As it turns out, Asus claims that both the Z87 and Z97 chipsets don't support VT-d! I know that ASRock is one of the few manufacturers that have consistently supported VT-d on their motherboards, and I'm glad to be getting one of these boards for this reason (I wish I would have done so from the beginning).

I'd like to ask about VT-d support, and specifically what tests are done to ensure VT-d is supported. Does Intel require certain tests to pass in order for motherboard manufacturers to claim VT-d support? The Asus rep I talked to about this said there were 50 VT-d tests, and that none of their boards pass all of them (which is why they are not claiming VT-d support). But instead of taking blame and stopping there, he said the Z87 and Z97 chipsets don't support VT-d, and that any manufacturer claiming differently had more lax testing standards than Asus. He seemed to forget that his company also claimed support on some of their own Z87 boards. All of this conversation is in the following thread if anyone is interested:

ASUS Z97 Motherboards - Official Support Thread - Page 14

I know VT-d works on ASRock motherboards, as there are numerous threads on various forums where people have posted successful VT-d tests on Z87-based systems. I'm sure the same will be true for Z97 boards. It makes no sense to me that Intel would produce multiple chips like the i7-4790, which has VT-d support and is Z97-compatible, if VT-d wasn't possible with a Z97 chipset.

If there are 50 VT-d tests from Intel, then does anyone have a link to details on these tests? Also, how do the ASRock motherboards do on these tests? I'm wary of how this VT-d support has been handled by all parties involved at this point. My goal is to build a computer that will be used for virtualization, gaming, work, multiple users. A critical aspect of this is VT-d and VGA passthrough. ASRock seems to be the best source for motherboards that adequately handle this technology, so hopefully they can provide details on this

[- wardog -]

Re: Virtualization over Directed I/O support



or pdf of same:

http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf

[vuldin]

Re: Virtualization over Directed I/O support

or pdf of same:

http://www.intel.com/content/dam/www...ed-io-spec.pdf

Awesome thanks for the link. I was looking through this earlier and trying to find any mention of specific VT-d tests that manufacturers must pass and didn't see it. From looking at this file it's apparent there are many ways to test the described functionality, but I'm not in the position to know how to come up with those tests based on reading that file. I would expect that either Intel has another document they send out to manufacturers (most likely) or the manufacturers themselves come up with tests (which would make the VT-d test meaningless).

[- wardog -]

Re: Virtualization over Directed I/O support

As far as I understand it, the BIOS is like the marriage papers between a chipset and processor as far as virtualization. A formality, but a necessary one none the less.

[pixo]

Re: Virtualization over Directed I/O support

The information provided by Asus is actually true from my experience. The Z97 chipset does not fully support vt-d.
But I still have working vga-passthrough because my CPU supports it and my BIOS enables the vt-d support even if it is limited for chipset.
That means i have fully working PCI and VGA passtrough for the PCIe lanes coming from CPU and can attach only one or both devices (needed a ACL override patch to my linux kernel).
But for the PCIe lanes coming from the chipset i need to pass them all as vfio will not allow me to pass just one, this include ASMedia SCSI and USB controllers my MB has.
So in the end you have to differentiate between chipset support and CPU support and find out if you need support only in the CPU or both and make sure you have BIOS support.
I do not know what hypervisor solution you are using so cant help you much but hope this info helps you.

My solution:
linux 3.14 + KVM (ACL override patch + patch for igp)
qemu 2.0.0
Asus Maximus VII Gene
i7-4770
Intel HD4600 - main
AMD Radeon HD 6450 - main
Z97 usb and scsi - main
AMD Radeon R9 290 - virtual
ASMedia usb and scsi - virtual

[vuldin]

Re: Virtualization over Directed I/O support

The information provided by Asus is actually true from my experience. The Z97 chipset does not fully support vt-d.
But I still have working vga-passthrough because my CPU supports it and my BIOS enables the vt-d support even if it is limited for chipset.
That means i have fully working PCI and VGA passtrough for the PCIe lanes coming from CPU and can attach only one or both devices (needed a ACL override patch to my linux kernel).
But for the PCIe lanes coming from the chipset i need to pass them all as vfio will not allow me to pass just one, this include ASMedia SCSI and USB controllers my MB has.
So in the end you have to differentiate between chipset support and CPU support and find out if you need support only in the CPU or both and make sure you have BIOS support.
I do not know what hypervisor solution you are using so cant help you much but hope this info helps you.

My solution:
linux 3.14 + KVM (ACL override patch + patch for igp)
qemu 2.0.0
Asus Maximus VII Gene
i7-4770
Intel HD4600 - main
AMD Radeon HD 6450 - main
Z97 usb and scsi - main
AMD Radeon R9 290 - virtual
ASMedia usb and scsi - virtual

This is very helpful! I plan to use the hardware in my signature (with dom0 using the integrated Intel graphics and Win8.1 using the 290X). As for software, I plan on using archlinux and the latest Xen (either unstable or 4.4). Do you have a link to notes or a guide you followed? Also which Linux distro are you using?

I also expect that what the Asus rep has said is true for the most part. At the same time, I think it's reasonable for him to provide more details on what tests he is referring to so that we can better understand VT-d technology and to what extent it is supported. I may try to follow up with him regarding this, but unfortunately I don't think he will add anything more to the discussion (he said he was finished discussing VT-d after his last post mentioned he is tired of going in the same circle with users regarding VT-d support after each new chipset generation). If we can get details on these tests then that would help determine where each motherboard and each manufacturer stands on VT-d support.

Regardless of the status of these tests on Asus, ASRock, or other manufacturer boards, the fact remains that people like you have successfully used the technology in certain cases. I'd definitely like to know if this successful use is being accomplished through only partial VT-d support by the manufacturers, but my first concern is getting it to work using whatever support available (incomplete or otherwise).

[pixo]

Re: Virtualization over Directed I/O support

My distro is Gentoo but i have used Arch thread at https://bbs.archlinux.org/viewtopic.php?id=162768&p=1
It uses KVM not Xen so don`t know how helpful it will be for you.

Also if you wish to passtrough one of the chipset USB controllers you should disable USB3(xHCI) support in bios and find out which ports belong to which controller.
If you have USB3 activated all devices are under one tree according to lsusb. Then you have to experiment if you can pass one with USB3 activated or not.
I had to leave USB3 deactivated on Z87 to be able to pass whole controller. Z97 has the same agregating behavior with USB3 activated.
You may need this if you have wifi or other cards on the chipset PCIe, or use USB passtrough but that may have some drawbacks depending on device connected.

But this can differ on the Xen and ASRock bios, don`t have experience with them.

[vuldin]

Re: Virtualization over Directed I/O support

Thanks for the link to archlinux config of a KVM VT-d solution! I've only focused on Xen because I assumed support for gfx_passthrough was better with that project. It looks like that may not necessarily be the case. I'll read up on KVM features around this and may go that route, I'm much more familiar with virtualization using KVM than I am with Xen (I've used KVM for a while where I've only read about Xen).

[parsec]

Re: Virtualization over Directed I/O support

Intel's VT-d support is based upon the CPU supporting it. Your i7-4790 does support VT-d. But the chipset is involved in VT-d support too, according to Intel.

Intel lists six eight series chipsets that are compatible with your processor, but only the Q87 is specifically listed as supporting VT-d. The H81, B85, Q85, and Z87 specifically say they do not support VT-d. The H87 information does not indicate either way. The Z87 being the over clocking chipset platform, simply states the vPro technology is not supported, which is most if not all virtualization.

Eight series chipsets info links:

The Z97 chipset is also the over clocking chipset platform, and like the Z87 states that vPro technology is not supported.

Z97 info page:

But it seems the workaround posted here will work for you, good luck with that.

[Nebel]

Re: Virtualization over Directed I/O support

I was curious if you received your z97 Exterme 6 board and were able to successfully pass thru PCI and other I/Os. I am debating between the z87 Extreme4 and the new z97 board but the whole point of the build is to leverage Vt-d to pass thru video cards and such using Esxi. The z87 Asrock seem to have a good track record of doing this, but wasn't sure if things might change with the z97 boards.

[vuldin]

Re: Virtualization over Directed I/O support

I'm sure my issues with VT-d passthrough are related to the device I'm trying to pass rather than the board. I'm trying to pass a 290x through via vga passthrough and running into issues that I think are related to xen not knowing exactly where the GPU BIOS is in memory. I haven't done much testing over the last week since I've been too busy with other things, but this is something that I'll come back to and I'll post more details then. My next steps are to ensure pass through works for other components without issue, and then I'll start trying to use different xen config, apply patches to xen, and/or try a different version of xen.

[Nebel]

Re: Virtualization over Directed I/O support

Not sure if this will help you at all, but the research I've done on passing thru graphic cards indicates that you have to dedicate RAM to the VM you are trying to assign the card to. That was based on an ESXi install though so not sure if the same rule applies to Xen. I'm still in the process of putting my rig together so no hands on experience yet. Good luck with your build.

[guidugli]

Re: Virtualization over Directed I/O support

If there are 50 VT-d tests from Intel, then does anyone have a link to details on these tests? Also, how do the ASRock motherboards do on these tests? I'm wary of how this VT-d support has been handled by all parties involved at this point. My goal is to build a computer that will be used for virtualization, gaming, work, multiple users. A critical aspect of this is VT-d and VGA passthrough. ASRock seems to be the best source for motherboards that adequately handle this technology, so hopefully they can provide details on this

When I was building my rig I did not find much information about compatibility or tests. What I found on my research is that ASRock has history enabling VT-d support, no big issues, while other brands do not enable VT-d, perform a buggy implementation, or break it on BIOS update. I was considering ASUS but give up after reading about several people with problem after Bios upgrade and such.
I am very happy with ASRock so far.

I'm sure my issues with VT-d passthrough are related to the device I'm trying to pass rather than the board. I'm trying to pass a 290x through via vga passthrough and running into issues that I think are related to xen not knowing exactly where the GPU BIOS is in memory. I haven't done much testing over the last week since I've been too busy with other things, but this is something that I'll come back to and I'll post more details then. My next steps are to ensure pass through works for other components without issue, and then I'll start trying to use different xen config, apply patches to xen, and/or try a different version of xen.

With my Z87 Fatality Pro with Intel i-4770, I had successfully passed through my Radeon R9 290 GPU, the Fatal1ty USB controller and Creative Core3D soundcard. The video performance is astonishing. Did a benchmark inside the VM and outside and the numbers were pratically the same.
Also, on another virtual machine, I was able to passthrough an Asmedia controller with my CDROM, so I could use Alcohol 120% and Winamp. Also did some tests passing through network cards and it works as well.

I am using Fedora 20 with latest kernel (3.14.8) and KVM (qemu-kvm-1.6.2-6.fc20.x86_64). Didn't need to recompile kernel or apply additional patches. Just needed to change kernel boot parameters on Grub (add devices to PCI-Stub, enable VT-d, etc), and add the devices using virt-manager. I also tested with qemu-2.0 and it works fine too. You may consider doing a test with Fedora 20.