POWER9, ARM64 and 64k page sizes

I've recently had discussions with other developers in the Fedora world about the default 64k page size on POWER9. The vast majority of GNU/Linux users have a 4k page size. There is now a change proposal for the ppc64le page size on Fedora 35 and a related discussion on the devel mailing list.

Why and when the non-x86 architectures are relevant

With each new generation of x86 processors from Intel and AMD there is a larger quantity of opaque microcode that independent developers are unable to audit or fix.

When a vendor has such an incredible market share, it is inevitable that problems like this will arise.

Investing some time and effort on alternative architectures is good insurance: when the day comes that this microcode is compromised, some percentage of users will jump ship.

Getting started on POWER9 and ARM64 is easier than ever

For POWER9, please see my recent blog about the Talos II Quickstart.

Similar blogs have recently appeared about ARM64, for example, this Fedora Magazine article about SolidRun HoneyComb LX2K.

Choosing between POWER9 and ARM64

ARM64 may be a better choice if you are very sensitive to the heat emissions and energy consumption costs.

POWER9 may be a better choice if you need a lot of compute power.

While neither ARM64 nor POWER9 have microcode comparable to Intel or AMD products, it is still important to look at the overall system. For example, Raptor's Talos II motherboard has the FSF Respects Your Freedom (RYF) certification but if you order the optional SATA controller, you end up with some proprietary firmware.

Why 64k page sizes are an issue

The GNU/Linux kernel for these platforms can be compiled with either 4k or 64k page size. The distribution chooses which of these options to select. The kernel created by the distribution is included in the installation disk for the distribution.

One acute consequence of this is the relationship between Btrfs sectorsize and kernel page size. Btrfs filesystems can only be used on systems with the same page size. The Btrfs driver is being improved to remove this restriction but for users of Fedora 34 and older systems, this is a very inconvenient issue. If you need to move Btrfs filesystems between systems with different page sizes then they simply won't work.

It appears that nobody tests the kernel and amdgpu drivers on these non-standard page sizes before each official release. Consequently, if there is a problem, it is only discovered by users after the upstream release. This means that users on these platforms are always a step behind users on other platforms.

Improving support for 64k page sizes

Personally, I'm quite keen to see the 64k page size succeed.

I believe that is only possible when these platforms have critical mass and when some of the upstream developers use these machines on a daily basis.

Until we get to that point, I feel it is a chicken-and-egg problem: things don't work, so people buy in more slowly, so there are less people to report bugs and/or fix things.

Automated CI testing of the kernel and amdgpu code may also help to catch some issues before official releases.

To summarize, I have an open mind about how to go about this. Please feel free to share your ideas and experiences in the discussion or through your blog.

Workarounds

Anybody who buys one of these machines today can still use it almost immediately.

One option is to use a distribution with a 4k page size or compile your own kernel with a 4k page size.

Another idea is to simply avoid using Btrfs for another six months: if you use the installation system of your preferred distribution to create filesystems, check that it isn't using Btrfs. You may need to manually override it to use ext4 for the moment.

If you don't need a modern GPU, some of the previous generation, such as the Radeon RX 580, seem to be working fine on any page size. These cards are available with up to 8GB VRAM. The performance of those cards is more than adequate for many workstation users.

The page size issue should not deter anybody from buying the hardware today. The community is always here to help.