I have an Asus X99-E-10G WS board with an Intel Xeon E5 2679 V4. I know the CPU has 40 PCIe lanes and has support for IOMMU so passing through GPUs in proxmox is trivial.

This board has PLX chips to split 32x PCIe 3.0 lanes into its 7x PCIe slots that operate at all 8x or 4 of them at 16x. I have passed through multiple GPUs to VMs on this board without issues before, but I just got a Mellanox ConnectX3 FCBT to connect to my NAS and it seems like this is causing issues with passing through a GPU that’s also on the same PLX chip as the Mellanox card.

I have a Tesla P100 that I am trying to pass through that’s plugged into a PCIe slot coming from the second PLX chip that also has the Mellanox card plugged into another port from the same PLX chip. This causes a code 10 error in windows device manager that said there is not enough resources to start the API and the GPU won’t start and can’t be used by the driver.

I have 4G Decoding, Virtualization, VT-D and ACS enabled in bios as well as CSM disabled and it still does not work. It will only work if I plug my Tesla P100 into another slot that is connected to the first PLX chip while the mellanox card is in the second PLX chip. This is an issue because then I would effectively reduce the number of PCIe slots available for use for GPUs on the board.

Is this fixable or just an inherent behaviour of Mellanox 40G cards? Thanks for any help.