Linux Support for NUMA Hardware

Large count multiprocessors are being built with non-uniform memory access (NUMA) times - access times that are dependent upon where within the machine a piece of memory physically resides. For optimal performance, the kernel needs to be aware of where memory is located, and keep memory used as close as possible to the user of the memory. Examples of NUMA machines include the NEC Azusa, the IBM x440 and the IBM NUMAQ.

The 2.5 Linux kernel includes many enhancements in support of NUMA machines. Data structures and macros are provided within the kernel for determining the layout of the memory and processors on the system. These enable the VM subsystem to make decisions on the optimal placement of memory for processes. This topology information is also exported to user-space via sysfs.

In addition to items that have been incorported into the 2.5 Linux kernel, there are NUMA features that have been developed that continue to be supported as patchsets. These include NUMA enhancements to the scheduler, multipath I/O and a user-level API that provides user control over the allocation of resources in respect to NUMA nodes.

This page provides links to information about the various Linux on NUMA projects. Discussions related to Linux on NUMA take place on the lse-tech mailing list and on the linux kernel mailing list.

Introduction to NUMA:

Demystify NUMA via the FAQ
Current NUMA status of Linux

Linux-on-NUMA Projects:

NUMA-aware scheduler:

On NUMA systems, optimal performance is obtained by locating processes as close to the memory they access as possible. For most processes, optimal performance is obtained by allocating all memory for the process from the same node, and dispatching the process on processors on that node. NUMA awareness within the scheduler is necessary in order to support locality of processes to memory - primarily by dispatching a process on the same node through the duration of the procesess' life.

Discontigmem:

short description - [to do]

NUMA-aware locking primitives

short description - [to do]

NUMA related APIs:

To facilitate user space programs to optimize for NUMA configurations, APIs are necessary to export topology information, and to provide user specification of processor and memory resources to be used. There are also internal kernel APIs to provide NUMA topology information for use by kernel subsystems. The following links provide information on various NUMA APIs, some of which are accepted in the mainline kernel, others which exist as independent patchsets, and others which are in the planning/proposal stage.
- In-Kernel Topology API - Included: 2.5.40
- Sysfs Topology API - Included: 2.5.46
- Memory Binding API - Posted to LKML for discussion
- setaffinity - Included: early 2.5 - developed for general SMP, but necessary for NUMA resource binding
- launch policies
- Process Scheduling and Memory Placement Design Notes a.k.a. CpuMemSets

Multi-path I/O:

short description - [to do]

Linux-on-NUMA Related Resources:

lse-tech@lists.sourceforge.net is a great place for all kinds of discussion regarding the Linux Scalability Effort as a whole.
NUMA-related email archive
Linux-kernel mailing list and its archives also contain some valuable information. Check out the lkmlFAQ.

Linux-on-NUMA Meeting Links:

NUMA Links by Corporation:

Last updated: 11/20/02 Any problems, additions, etc., please send email to this page's maintainer.