From the ASRock X79 Extreme9 specifications web page:
- - 5 x PCI Express 3.0 x16 slots (PCIE1/PCIE2/PCIE4/PCIE5: x8/8/8/8 mode or x16/0/16/0 mode; PCIE6: x8 mode)
There might be more detailed information in your motherboard user manual.
It looks like pci-e1 and pci-e4 can run at x16 when pci-e2 and pci-e5 are empty.
Your video card might run at x16 in pci-e4 when pci-e2 and pci-e5 are empty.
It looks like pci-e6 will always run in x8 mode.
With your video card in the pci-e4 slot, leave pci-e2 and pci-e5 empty.
Run GPU-z and see what it reports for the Bus Interface value.
Another (not cheap) option is to buy an aftermarket cooler for your video card that only takes up one additional slot.
With your video card in 3.0 mode, you will probably find that running with x8 provides no noticeable performance difference.