Please report all spam threads, posts and suspicious members. We receive spam notifications and will take immediate action!
Results 1 to 4 of 4

Thread: Xen and GPU passthrough

  1. #1
    vuldin is offline Junior Member
    Join Date
    May 2014

    Default Xen and GPU passthrough

    I've made alot of progress on getting a Windows VM to have control over a 290x using Xen, but it's not yet working properly. I have a ton of details I can share on what I've done, but for now I have some specific questions for those who have gone through these steps successfuly.

    My 290x doesn't seem to support FLR (function level reset). I'm not sure if this is a requirement of all GPUs that will be passed to a VM or just Nvidia cards, but regardless I found this out by running sudo lspci -vv looking for my 290x and then finding the section starting with 'DevCap' (FLReset- is present). For what it's worth, all my devices that report on this functionality are listed as no support (FLReset-). First I need to ensure this is critical to successful PCI passthrouth. If so, then this makes me think I need to either 1) enable something in the BIOS or 2) change a kernel or module parameter.

    Here is the output of of xl dmesg:
     Xen 4.4.0
    (XEN) Xen version 4.4.0 ([email protected]) (gcc (GCC) 4.9.0 20140521 (prerelease)) debug=n Sat May 31 15:52:14 PDT 2014
    (XEN) Latest ChangeSet: 
    (XEN) Bootloader: GRUB 2.02~beta2
    (XEN) Command line: /xen-4.4.0.gz iommu=1 iommu_inclusive_mapping=1 dom0_mem=16G xsave=1
    (XEN) Video information:
    (XEN)  VGA is text mode 80x25, font 8x16
    (XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
    (XEN) Disc information:
    (XEN)  Found 0 MBR signatures
    (XEN)  Found 0 EDD information structures
    (XEN) Multiboot-e820 RAM map:
    (XEN)  0000000000000000 - 0000000000058000 (usable)
    (XEN)  0000000000058000 - 0000000000059000 (reserved)
    (XEN)  0000000000059000 - 000000000009f000 (usable)
    (XEN)  000000000009f000 - 00000000000a0000 (reserved)
    (XEN)  0000000000100000 - 0000000069e4c000 (usable)
    (XEN)  0000000069e4c000 - 0000000069e53000 (ACPI NVS)
    (XEN)  0000000069e53000 - 000000006ab54000 (usable)
    (XEN)  000000006ab54000 - 000000006b085000 (reserved)
    (XEN)  000000006b085000 - 000000007d185000 (usable)
    (XEN)  000000007d185000 - 000000007d216000 (reserved)
    (XEN)  000000007d216000 - 000000007d25d000 (usable)
    (XEN)  000000007d25d000 - 000000007d398000 (ACPI NVS)
    (XEN)  000000007d398000 - 000000007df7d000 (reserved)
    (XEN)  000000007df7d000 - 000000007dfff000 type 20
    (XEN)  000000007dfff000 - 000000007e000000 (usable)
    (XEN)  000000007f000000 - 00000000bf200000 (reserved)
    (XEN)  00000000f0000000 - 00000000f8000000 (reserved)
    (XEN)  00000000fec00000 - 00000000fec01000 (reserved)
    (XEN)  00000000fed00000 - 00000000fed04000 (reserved)
    (XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
    (XEN)  00000000fee00000 - 00000000fee01000 (reserved)
    (XEN)  00000000ff000000 - 0000000100000000 (reserved)
    (XEN)  0000000100000000 - 000000083fe00000 (usable)
    (XEN) ACPI: RSDP 000F0010, 0024 (r2 ALASKA)
    (XEN) ACPI: XSDT 7D365088, 0094 (r1 ALASKA    A M I  1072009 AMI     10013)
    (XEN) ACPI: FACP 7D376090, 010C (r5 ALASKA    A M I  1072009 AMI     10013)
    (XEN) ACPI: DSDT 7D3651B8, 10ED6 (r2 ALASKA    A M I       35 INTL 20120711)
    (XEN) ACPI: FACS 7D397F80, 0040
    (XEN) ACPI: APIC 7D3761A0, 0092 (r3 ALASKA    A M I  1072009 AMI     10013)
    (XEN) ACPI: FPDT 7D376238, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
    (XEN) ACPI: SSDT 7D376280, 0539 (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
    (XEN) ACPI: SSDT 7D3767C0, 0B74 (r1 CpuRef  CpuSsdt     3000 INTL 20051117)
    (XEN) ACPI: SSDT 7D377338, 01C7 (r1  PmRef LakeTiny     3000 INTL 20051117)
    (XEN) ACPI: MCFG 7D377500, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
    (XEN) ACPI: HPET 7D377540, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
    (XEN) ACPI: SSDT 7D377578, 036D (r1 SataRe SataTabl     1000 INTL 20120711)
    (XEN) ACPI: SSDT 7D3778E8, 5B5E (r1 SaSsdt  SaSsdt      3000 INTL 20120711)
    (XEN) ACPI: AAFT 7D37D448, 049F (r1 ALASKA OEMAAFT   1072009 MSFT       97)
    (XEN) ACPI: BGRT 7D37D8E8, 0038 (r0 ALASKA    A M I  1072009 AMI     10013)
    (XEN) ACPI: DMAR 7D37D920, 00B8 (r1 INTEL      BDW         1 INTL        1)
    (XEN) ACPI: SSDT 7D37D9D8, 0579 (r1 Intel_ IsctTabl     1000 INTL 20120711)
    (XEN) System RAM: 31690MB (32450764kB)
    (XEN) Domain heap initialised
    (XEN) ACPI: 32/64X FACS address mismatch in FADT - 7d397f80/0000000000000000, using 32
    (XEN) Processor #0 7:12 APIC version 21
    (XEN) Processor #2 7:12 APIC version 21
    (XEN) Processor #4 7:12 APIC version 21
    (XEN) Processor #6 7:12 APIC version 21
    (XEN) Processor #1 7:12 APIC version 21
    (XEN) Processor #3 7:12 APIC version 21
    (XEN) Processor #5 7:12 APIC version 21
    (XEN) Processor #7 7:12 APIC version 21
    (XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
    (XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
    (XEN) Failed to enable Interrupt Remapping: Will not enable x2APIC.
    (XEN) Using scheduler: SMP Credit Scheduler (credit)
    (XEN) Detected 3598.990 MHz processor.
    (XEN) Initing memory sharing.
    (XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
    (XEN) spurious 8259A interrupt: IRQ7.
    (XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
    (XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
    (XEN) Intel VT-d Snoop Control not enabled.
    (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
    (XEN) Intel VT-d Queued Invalidation enabled.
    (XEN) Intel VT-d Interrupt Remapping enabled.
    (XEN) Intel VT-d Shared EPT tables not enabled.
    (XEN) I/O virtualisation enabled
    (XEN)  - Dom0 mode: Relaxed
    (XEN) Interrupt remapping enabled
    (XEN) Enabled directed EOI with ioapic_ack_old on!
    (XEN)  -> Using old ACK method
    (XEN) Platform timer is 14.318MHz HPET
    (XEN) Allocated console ring of 16 KiB.
    (XEN) VMX: Supported advanced features:
    (XEN)  - APIC MMIO access virtualisation
    (XEN)  - APIC TPR shadow
    (XEN)  - Extended Page Tables (EPT)
    (XEN)  - Virtual-Processor Identifiers (VPID)
    (XEN)  - Virtual NMI
    (XEN)  - MSR direct-access bitmap
    (XEN)  - Unrestricted Guest
    (XEN)  - VMCS shadowing
    (XEN) HVM: ASIDs enabled.
    (XEN) HVM: VMX enabled
    (XEN) HVM: Hardware Assisted Paging (HAP) detected
    (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
    (XEN) Brought up 8 CPUs
    (XEN) *** LOADING DOMAIN 0 ***
    (XEN)  Xen  kernel: 64-bit, lsb, compat32
    (XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1f5b000
    (XEN)  Dom0 alloc.:   0000000820000000->0000000828000000 (4158762 pages to be allocated)
    (XEN)  Init. ramdisk: 000000083f32a000->000000083fdff200
    (XEN)  Loaded kernel: ffffffff81000000->ffffffff81f5b000
    (XEN)  Init. ramdisk: ffffffff81f5b000->ffffffff82a30200
    (XEN)  Phys-Mach map: ffffffff82a31000->ffffffff84a31000
    (XEN)  Start info:    ffffffff84a31000->ffffffff84a314b4
    (XEN)  Page tables:   ffffffff84a32000->ffffffff84a5b000
    (XEN)  Boot stack:    ffffffff84a5b000->ffffffff84a5c000
    (XEN)  TOTAL:         ffffffff80000000->ffffffff84c00000
    (XEN)  ENTRY ADDRESS: ffffffff818ee1f0
    (XEN) Dom0 has maximum 8 VCPUs
    (XEN) Scrubbing Free RAM: .....................................................................................................................................................done.
    (XEN) Initial low memory virq threshold set at 0x4000 pages.
    (XEN) Std. Loglevel: Errors and warnings
    (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
    (XEN) Xen is relinquishing VGA console.
    (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
    (XEN) Freed 260kB init memory.
    Here is my /etc/xen/vwin.hvm (vm config):
    name = 'vwin'
    builder = 'hvm'
    memory = 8192
    vcpus = 2
    disk = [ '/home/joshua/vms/vwin.img,raw,xvda,w', 'file:/home/joshua/vms/win81.iso,hdc:cdrom,r' ]
    vif = [ 'bridge=xenbr0,ip=' ]
    vnclisten = ''
    vnc = 1
    pci = [ '01:00.0', '01:00.1' ]
    pci_permissive = 1
    When I successfully start the VM, here is the output:
    Parsing config from /etc/xen/vwin.hvm
    libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.0
    libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.1
    The above at first made me think there is an issue, but I not sure that it is at this point. When Windows boots I see the 290x, and when I run the AMD test to see what hardware I have it finds it as being a 290x. I can install the drivers for the card and no errors are reported. I reboot and look at Device Manager and I see the following message associated with the problem 290x device:
    This device cannot find enough free resources that it can use. (Code 12)
    Here's the relevant portion of my grub config:
    multiboot /xen-4.4.0.gz /xen-4.4.0.gz iommu=1 iommu_inclusive_mapping=1 dom0_mem=16G xsave=1
    module /vmlinuz-linux /vmlinuz-linux root=UUID=917153ed-f386-4987-8fd0-9e293f8d969a rw  modprobe.blacklist=radeon i915 quiet rootflags=subvol=rootvol console=tty0 xen-pciback.passthrough=1 xen-pciback.hide=(01:00.0)(01:00.1)
    I think 'iommu=1' is related to previous versions of Xen, and I'm not sure what 'iommu_inclusive_mapping=1' even does. I haven't verified that I get the same current functionality without these lines yet.
    Last edited by vuldin; 06-02-2014 at 05:59 AM.

  2. #2
    vuldin is offline Junior Member
    Join Date
    May 2014

    Default Re: Xen and GPU passthrough

    The more I'm reading about FLR the more I'm worried that this is a feature which needs to be enabled from the BIOS on motherboards. For instance, the Intel DH55HC has an option in the chipset configuration section of its BIOS to enable FLR. I see no such option in the BIOS v1.0 for the Z97 Extreme6. Here is a definition of FLR:

    Function Level Reset Support (FLR)
    The integrated GbE controller supports FLR capability. FLR capability can be used in
    conjunction with Intel® Virtualization Technology. FLR allows an operating system in a
    Virtual Machine to have complete control over a device, including its initialization,
    without interfering with the rest of the platform. The device provides a software
    interface that enables the operating system to reset the entire device as if a PCI reset
    was asserted.

    Is this something that could be enabled as an update to the BIOS?

  3. #3
    vuldin is offline Junior Member
    Join Date
    May 2014

    Default Re: Xen and GPU passthrough

    I've posted details of the above issue to xen-users mailing list:

    Xen-users Info Page

    Hopefully someone there will be able to shed some light on where my issue is. I've also posted to an thread that is focused on GPU passthrough:

    {Guide} Create a Gaming Virtual Machine - Page 77

    One way or another I'm going to get this working. I'll post how it's done when I make some progress, which will likely have to wait until later this week (possibly this weekend).

  4. #4
    vuldin is offline Junior Member
    Join Date
    May 2014

    Default Re: Xen and GPU passthrough

    I believe I found my issue, although I haven't tested it out yet. I need to add the following line to my /etc/xen/vwin.hvm:
    device_model_version = 'qemu-xen-traditional'
    As it turns out, xen builds its own version of qemu. This new version has alot of cool features, but one of the few things it doesn't yet have (that the old version does) is VGA passthrough. For more information, see the following links:

    QEMU vs qemu-traditionnal Feature Comparison - Xen
    QEMU Upstream - Xen

    I said I wouldn't test all this out until this weekend but I'm not sure I'll be able to wait that long :D . It will be awesome if the fix is this simple.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts