Skip to content
NewA calmer, faster ZeroHour
Otherinfo

Linux PCI sizing bug blocks 32 GB GPU BARs behind a switch

Original title: Fitting Four 32 GB Resizable Bars Behind a PCIe Switch on Linux

Hacker News · AIby dougcalobrisi ()5 min read1
AI summary · x-ai/grok-4.7Summaries by Vani AI

An out-of-tree initramfs module sets BAR sizes and pads alignment so four R9700s map before amdgpu loads.

A builder using a Broadcom PEX88096 switch with four AMD Radeon AI PRO R9700s and two RTX 3090s could not get 32 GB Resizable BARs from the BIOS or from Linux 7.0's bridge window allocator. The post says pbus_size_mem() omits nearly 32 GB gaps between each GPU BAR and its doorbell, a regression already reported on linux-pci, with a 14-patch fix at v5 as of November 6 that is not merged. An out-of-tree module, gpu-rebar-layout, resizes and rescans from the initramfs; after that, amdgpu reported BAR=32768M and R9700 peer-to-peer copies reached 14.0 GB/s per pair.

Unconfirmed: The kernel fix is reported as tested by the author but not yet merged.

Key facts
  • The failure was seen on Linux 7.0.6 with a roughly 128.6 GB switch window.
  • Ilpo Järvinen's unmerged v5 series placed all four 32 GB R9700 BARs in testing.
  • The module is on GitHub and depends on kernel internals that may change.
  • With ACS redirect disabled, four R9700s copying in a ring moved 54.7 GB/s.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

5 min read at doug.sh

My PCIe switch is a Broadcom PEX88096 in the board’s one x16 slot, with four AMD Radeon AI PRO R9700s and the two RTX 3090s The switch gives each card eight lanes and lets the cards copy to each other without going through the CPU. AMD’s ROCm calls that peer-to-peer (P2P), and it only turns it on when each card’s whole

Hacker News · AI

By dougcalobrisi · doug.sh

Read the full article at Hacker News · AI

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.