| |
cpuset(7) - phpMan
CPUSET(7) Linux Programmer's Manual CPUSET(7)
NAME
cpuset - confine processes to processor and memory node subsets
DESCRIPTION
The cpuset filesystem is a pseudo-filesystem interface to the kernel cpuset mechanism,
which is used to control the processor placement and memory placement of processes. It is
commonly mounted at /dev/cpuset.
On systems with kernels compiled with built in support for cpusets, all processes are
attached to a cpuset, and cpusets are always present. If a system supports cpusets, then
it will have the entry nodev cpuset in the file /proc/filesystems. By mounting the cpuset
filesystem (see the EXAMPLE section below), the administrator can configure the cpusets on
a system to control the processor and memory placement of processes on that system. By
default, if the cpuset configuration on a system is not modified or if the cpuset filesys‐
tem is not even mounted, then the cpuset mechanism, though present, has no affect on the
system's behavior.
A cpuset defines a list of CPUs and memory nodes.
The CPUs of a system include all the logical processing units on which a process can exe‐
cute, including, if present, multiple processor cores within a package and Hyper-Threads
within a processor core. Memory nodes include all distinct banks of main memory; small
and SMP systems typically have just one memory node that contains all the system's main
memory, while NUMA (non-uniform memory access) systems have multiple memory nodes.
Cpusets are represented as directories in a hierarchical pseudo-filesystem, where the top
directory in the hierarchy (/dev/cpuset) represents the entire system (all online CPUs and
memory nodes) and any cpuset that is the child (descendant) of another parent cpuset con‐
tains a subset of that parent's CPUs and memory nodes. The directories and files repre‐
senting cpusets have normal filesystem permissions.
Every process in the system belongs to exactly one cpuset. A process is confined to run
only on the CPUs in the cpuset it belongs to, and to allocate memory only on the memory
nodes in that cpuset. When a process fork(2)s, the child process is placed in the same
cpuset as its parent. With sufficient privilege, a process may be moved from one cpuset
to another and the allowed CPUs and memory nodes of an existing cpuset may be changed.
When the system begins booting, a single cpuset is defined that includes all CPUs and mem‐
ory nodes on the system, and all processes are in that cpuset. During the boot process,
or later during normal system operation, other cpusets may be created, as subdirectories
of this top cpuset, under the control of the system administrator, and processes may be
placed in these other cpusets.
Cpusets are integrated with the sched_setaffinity(2) scheduling affinity mechanism and the
mbind(2) and set_mempolicy(2) memory-placement mechanisms in the kernel. Neither of these
mechanisms let a process make use of a CPU or memory node that is not allowed by that
process's cpuset. If changes to a process's cpuset placement conflict with these other
mechanisms, then cpuset placement is enforced even if it means overriding these other
mechanisms. The kernel accomplishes this overriding by silently restricting the CPUs and
memory nodes requested by these other mechanisms to those allowed by the invoking
process's cpuset. This can result in these other calls returning an error, if for exam‐
ple, such a call ends up requesting an empty set of CPUs or memory nodes, after that
request is restricted to the invoking process's cpuset.
Typically, a cpuset is used to manage the CPU and memory-node confinement for a set of
cooperating processes such as a batch scheduler job, and these other mechanisms are used
to manage the placement of individual processes or memory regions within that set or job.
FILES
Each directory below /dev/cpuset represents a cpuset and contains a fixed set of pseudo-
files describing the state of that cpuset.
New cpusets are created using the mkdir(2) system call or the mkdir(1) command. The prop‐
erties of a cpuset, such as its flags, allowed CPUs and memory nodes, and attached pro‐
cesses, are queried and modified by reading or writing to the appropriate file in that
cpuset's directory, as listed below.
The pseudo-files in each cpuset directory are automatically created when the cpuset is
created, as a result of the mkdir(2) invocation. It is not possible to directly add or
remove these pseudo-files.
A cpuset directory that contains no child cpuset directories, and has no attached pro‐
cesses, can be removed using rmdir(2) or rmdir(1). It is not necessary, or possible, to
remove the pseudo-files inside the directory before removing it.
The pseudo-files in each cpuset directory are small text files that may be read and writ‐
ten using traditional shell utilities such as cat(1), and echo(1), or from a program by
using file I/O library functions or system calls, such as open(2), read(2), write(2), and
close(2).
The pseudo-files in a cpuset directory represent internal kernel state and do not have any
persistent image on disk. Each of these per-cpuset files is listed and described below.
tasks List of the process IDs (PIDs) of the processes in that cpuset. The list is for‐
matted as a series of ASCII decimal numbers, each followed by a newline. A process
may be added to a cpuset (automatically removing it from the cpuset that previously
contained it) by writing its PID to that cpuset's tasks file (with or without a
trailing newline).
Warning: only one PID may be written to the tasks file at a time. If a string is
written that contains more than one PID, only the first one will be used.
notify_on_release
Flag (0 or 1). If set (1), that cpuset will receive special handling after it is
released, that is, after all processes cease using it (i.e., terminate or are moved
to a different cpuset) and all child cpuset directories have been removed. See the
Notify On Release section, below.
cpuset.cpus
List of the physical numbers of the CPUs on which processes in that cpuset are
allowed to execute. See List Format below for a description of the format of cpus.
The CPUs allowed to a cpuset may be changed by writing a new list to its cpus file.
cpuset.cpu_exclusive
Flag (0 or 1). If set (1), the cpuset has exclusive use of its CPUs (no sibling or
cousin cpuset may overlap CPUs). By default this is off (0). Newly created
cpusets also initially default this to off (0).
Two cpusets are sibling cpusets if they share the same parent cpuset in the
/dev/cpuset hierarchy. Two cpusets are cousin cpusets if neither is the ancestor
of the other. Regardless of the cpu_exclusive setting, if one cpuset is the ances‐
tor of another, and if both of these cpusets have nonempty cpus, then their cpus
must overlap, because the cpus of any cpuset are always a subset of the cpus of its
parent cpuset.
cpuset.mems
List of memory nodes on which processes in this cpuset are allowed to allocate mem‐
ory. See List Format below for a description of the format of mems.
cpuset.mem_exclusive
Flag (0 or 1). If set (1), the cpuset has exclusive use of its memory nodes (no
sibling or cousin may overlap). Also if set (1), the cpuset is a Hardwall cpuset
(see below). By default this is off (0). Newly created cpusets also initially
default this to off (0).
Regardless of the mem_exclusive setting, if one cpuset is the ancestor of another,
then their memory nodes must overlap, because the memory nodes of any cpuset are
always a subset of the memory nodes of that cpuset's parent cpuset.
cpuset.mem_hardwall (since Linux 2.6.26)
Flag (0 or 1). If set (1), the cpuset is a Hardwall cpuset (see below). Unlike
mem_exclusive, there is no constraint on whether cpusets marked mem_hardwall may
have overlapping memory nodes with sibling or cousin cpusets. By default this is
off (0). Newly created cpusets also initially default this to off (0).
cpuset.memory_migrate (since Linux 2.6.16)
Flag (0 or 1). If set (1), then memory migration is enabled. By default this is
off (0). See the Memory Migration section, below.
cpuset.memory_pressure (since Linux 2.6.16)
A measure of how much memory pressure the processes in this cpuset are causing.
See the Memory Pressure section, below. Unless memory_pressure_enabled is enabled,
always has value zero (0). This file is read-only. See the WARNINGS section,
below.
cpuset.memory_pressure_enabled (since Linux 2.6.16)
Flag (0 or 1). This file is present only in the root cpuset, normally /dev/cpuset.
If set (1), the memory_pressure calculations are enabled for all cpusets in the
system. By default this is off (0). See the Memory Pressure section, below.
cpuset.memory_spread_page (since Linux 2.6.17)
Flag (0 or 1). If set (1), pages in the kernel page cache (filesystem buffers) are
uniformly spread across the cpuset. By default this is off (0) in the top cpuset,
and inherited from the parent cpuset in newly created cpusets. See the Memory
Spread section, below.
cpuset.memory_spread_slab (since Linux 2.6.17)
Flag (0 or 1). If set (1), the kernel slab caches for file I/O (directory and
inode structures) are uniformly spread across the cpuset. By default this is off
(0) in the top cpuset, and inherited from the parent cpuset in newly created
cpusets. See the Memory Spread section, below.
cpuset.sched_load_balance (since Linux 2.6.24)
Flag (0 or 1). If set (1, the default) the kernel will automatically load balance
processes in that cpuset over the allowed CPUs in that cpuset. If cleared (0) the
kernel will avoid load balancing processes in this cpuset, unless some other cpuset
with overlapping CPUs has its sched_load_balance flag set. See Scheduler Load Bal‐
ancing, below, for further details.
cpuset.sched_relax_domain_level (since Linux 2.6.26)
Integer, between -1 and a small positive value. The sched_relax_domain_level con‐
trols the width of the range of CPUs over which the kernel scheduler performs imme‐
diate rebalancing of runnable tasks across CPUs. If sched_load_balance is dis‐
abled, then the setting of sched_relax_domain_level does not matter, as no such
load balancing is done. If sched_load_balance is enabled, then the higher the
value of the sched_relax_domain_level, the wider the range of CPUs over which imme‐
diate load balancing is attempted. See Scheduler Relax Domain Level, below, for
further details.
In addition to the above pseudo-files in each directory below /dev/cpuset, each process
has a pseudo-file, /proc/<pid>/cpuset, that displays the path of the process's cpuset
directory relative to the root of the cpuset filesystem.
Also the /proc/<pid>/status file for each process has four added lines, displaying the
process's Cpus_allowed (on which CPUs it may be scheduled) and Mems_allowed (on which mem‐
ory nodes it may obtain memory), in the two formats Mask Format and List Format (see
below) as shown in the following example:
Cpus_allowed: ffffffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list: 0-127
Mems_allowed: ffffffff,ffffffff
Mems_allowed_list: 0-63
The "allowed" fields were added in Linux 2.6.24; the "allowed_list" fields were added in
Linux 2.6.26.
EXTENDED CAPABILITIES
In addition to controlling which cpus and mems a process is allowed to use, cpusets pro‐
vide the following extended capabilities.
Exclusive cpusets
If a cpuset is marked cpu_exclusive or mem_exclusive, no other cpuset, other than a direct
ancestor or descendant, may share any of the same CPUs or memory nodes.
A cpuset that is mem_exclusive restricts kernel allocations for buffer cache pages and
other internal kernel data pages commonly shared by the kernel across multiple users. All
cpusets, whether mem_exclusive or not, restrict allocations of memory for user space.
This enables configuring a system so that several independent jobs can share common kernel
data, while isolating each job's user allocation in its own cpuset. To do this, construct
a large mem_exclusive cpuset to hold all the jobs, and construct child, non-mem_exclusive
cpusets for each individual job. Only a small amount of kernel memory, such as requests
from interrupt handlers, is allowed to be placed on memory nodes outside even a mem_exclu‐
sive cpuset.
Hardwall
A cpuset that has mem_exclusive or mem_hardwall set is a hardwall cpuset. A hardwall
cpuset restricts kernel allocations for page, buffer, and other data commonly shared by
the kernel across multiple users. All cpusets, whether hardwall or not, restrict alloca‐
tions of memory for user space.
This enables configuring a system so that several independent jobs can share common kernel
data, such as filesystem pages, while isolating each job's user allocation in its own
cpuset. To do this, construct a large hardwall cpuset to hold all the jobs, and construct
child cpusets for each individual job which are not hardwall cpusets.
Only a small amount of kernel memory, such as requests from interrupt handlers, is allowed
to be taken outside even a hardwall cpuset.
Notify on release
If the notify_on_release flag is enabled (1) in a cpuset, then whenever the last process
in the cpuset leaves (exits or attaches to some other cpuset) and the last child cpuset of
that cpuset is removed, the kernel will run the command /sbin/cpuset_release_agent, sup‐
plying the pathname (relative to the mount point of the cpuset filesystem) of the aban‐
doned cpuset. This enables automatic removal of abandoned cpusets.
The default value of notify_on_release in the root cpuset at system boot is disabled (0).
The default value of other cpusets at creation is the current value of their parent's
notify_on_release setting.
The command /sbin/cpuset_release_agent is invoked, with the name (/dev/cpuset relative
path) of the to-be-released cpuset in argv[1].
The usual contents of the command /sbin/cpuset_release_agent is simply the shell script:
#!/bin/sh
rmdir /dev/cpuset/$1
As with other flag values below, this flag can be changed by writing an ASCII number 0 or
1 (with optional trailing newline) into the file, to clear or set the flag, respectively.
Memory pressure
The memory_pressure of a cpuset provides a simple per-cpuset running average of the rate
that the processes in a cpuset are attempting to free up in-use memory on the nodes of the
cpuset to satisfy additional memory requests.
This enables batch managers that are monitoring jobs running in dedicated cpusets to effi‐
ciently detect what level of memory pressure that job is causing.
This is useful both on tightly managed systems running a wide mix of submitted jobs, which
may choose to terminate or reprioritize jobs that are trying to use more memory than
allowed on the nodes assigned them, and with tightly coupled, long-running, massively par‐
allel scientific computing jobs that will dramatically fail to meet required performance
goals if they start to use more memory than allowed to them.
This mechanism provides a very economical way for the batch manager to monitor a cpuset
for signs of memory pressure. It's up to the batch manager or other user code to decide
what action to take if it detects signs of memory pressure.
Unless memory pressure calculation is enabled by setting the pseudo-file
/dev/cpuset/cpuset.memory_pressure_enabled, it is not computed for any cpuset, and reads
from any memory_pressure always return zero, as represented by the ASCII string "0\n".
See the WARNINGS section, below.
A per-cpuset, running average is employed for the following reasons:
* Because this meter is per-cpuset rather than per-process or per virtual memory region,
the system load imposed by a batch scheduler monitoring this metric is sharply reduced
on large systems, because a scan of the tasklist can be avoided on each set of queries.
* Because this meter is a running average rather than an accumulating counter, a batch
scheduler can detect memory pressure with a single read, instead of having to read and
accumulate results for a period of time.
* Because this meter is per-cpuset rather than per-process, the batch scheduler can
obtain the key information—memory pressure in a cpuset—with a single read, rather than
having to query and accumulate results over all the (dynamically changing) set of pro‐
cesses in the cpuset.
The memory_pressure of a cpuset is calculated using a per-cpuset simple digital filter
that is kept within the kernel. For each cpuset, this filter tracks the recent rate at
which processes attached to that cpuset enter the kernel direct reclaim code.
The kernel direct reclaim code is entered whenever a process has to satisfy a memory page
request by first finding some other page to repurpose, due to lack of any readily avail‐
able already free pages. Dirty filesystem pages are repurposed by first writing them to
disk. Unmodified filesystem buffer pages are repurposed by simply dropping them, though
if that page is needed again, it will have to be reread from disk.
The cpuset.memory_pressure file provides an integer number representing the recent (half-
life of 10 seconds) rate of entries to the direct reclaim code caused by any process in
the cpuset, in units of reclaims attempted per second, times 1000.
Memory spread
There are two Boolean flag files per cpuset that control where the kernel allocates pages
for the filesystem buffers and related in-kernel data structures. They are called
cpuset.memory_spread_page and cpuset.memory_spread_slab.
If the per-cpuset Boolean flag file cpuset.memory_spread_page is set, then the kernel will
spread the filesystem buffers (page cache) evenly over all the nodes that the faulting
process is allowed to use, instead of preferring to put those pages on the node where the
process is running.
If the per-cpuset Boolean flag file cpuset.memory_spread_slab is set, then the kernel will
spread some filesystem-related slab caches, such as those for inodes and directory
entries, evenly over all the nodes that the faulting process is allowed to use, instead of
preferring to put those pages on the node where the process is running.
The setting of these flags does not affect the data segment (see brk(2)) or stack segment
pages of a process.
By default, both kinds of memory spreading are off and the kernel prefers to allocate mem‐
ory pages on the node local to where the requesting process is running. If that node is
not allowed by the process's NUMA memory policy or cpuset configuration or if there are
insufficient free memory pages on that node, then the kernel looks for the nearest node
that is allowed and has sufficient free memory.
When new cpusets are created, they inherit the memory spread settings of their parent.
Setting memory spreading causes allocations for the affected page or slab caches to ignore
the process's NUMA memory policy and be spread instead. However, the effect of these
changes in memory placement caused by cpuset-specified memory spreading is hidden from the
mbind(2) or set_mempolicy(2) calls. These two NUMA memory policy calls always appear to
behave as if no cpuset-specified memory spreading is in effect, even if it is. If cpuset
memory spreading is subsequently turned off, the NUMA memory policy most recently speci‐
fied by these calls is automatically reapplied.
Both cpuset.memory_spread_page and cpuset.memory_spread_slab are Boolean flag files. By
default they contain "0", meaning that the feature is off for that cpuset. If a "1" is
written to that file, that turns the named feature on.
Cpuset-specified memory spreading behaves similarly to what is known (in other contexts)
as round-robin or interleave memory placement.
Cpuset-specified memory spreading can provide substantial performance improvements for
jobs that:
a) need to place thread-local data on memory nodes close to the CPUs which are running the
threads that most frequently access that data; but also
b) need to access large filesystem data sets that must to be spread across the several
nodes in the job's cpuset in order to fit.
Without this policy, the memory allocation across the nodes in the job's cpuset can become
very uneven, especially for jobs that might have just a single thread initializing or
reading in the data set.
Memory migration
Normally, under the default setting (disabled) of cpuset.memory_migrate, once a page is
allocated (given a physical page of main memory), then that page stays on whatever node it
was allocated, so long as it remains allocated, even if the cpuset's memory-placement pol‐
icy mems subsequently changes.
When memory migration is enabled in a cpuset, if the mems setting of the cpuset is
changed, then any memory page in use by any process in the cpuset that is on a memory node
that is no longer allowed will be migrated to a memory node that is allowed.
Furthermore, if a process is moved into a cpuset with memory_migrate enabled, any memory
pages it uses that were on memory nodes allowed in its previous cpuset, but which are not
allowed in its new cpuset, will be migrated to a memory node allowed in the new cpuset.
The relative placement of a migrated page within the cpuset is preserved during these
migration operations if possible. For example, if the page was on the second valid node
of the prior cpuset, then the page will be placed on the second valid node of the new
cpuset, if possible.
Scheduler load balancing
The kernel scheduler automatically load balances processes. If one CPU is underutilized,
the kernel will look for processes on other more overloaded CPUs and move those processes
to the underutilized CPU, within the constraints of such placement mechanisms as cpusets
and sched_setaffinity(2).
The algorithmic cost of load balancing and its impact on key shared kernel data structures
such as the process list increases more than linearly with the number of CPUs being bal‐
anced. For example, it costs more to load balance across one large set of CPUs than it
does to balance across two smaller sets of CPUs, each of half the size of the larger set.
(The precise relationship between the number of CPUs being balanced and the cost of load
balancing depends on implementation details of the kernel process scheduler, which is sub‐
ject to change over time, as improved kernel scheduler algorithms are implemented.)
The per-cpuset flag sched_load_balance provides a mechanism to suppress this automatic
scheduler load balancing in cases where it is not needed and suppressing it would have
worthwhile performance benefits.
By default, load balancing is done across all CPUs, except those marked isolated using the
kernel boot time "isolcpus=" argument. (See Scheduler Relax Domain Level, below, to
change this default.)
This default load balancing across all CPUs is not well suited to the following two situa‐
tions:
* On large systems, load balancing across many CPUs is expensive. If the system is man‐
aged using cpusets to place independent jobs on separate sets of CPUs, full load bal‐
ancing is unnecessary.
* Systems supporting real-time on some CPUs need to minimize system overhead on those
CPUs, including avoiding process load balancing if that is not needed.
When the per-cpuset flag sched_load_balance is enabled (the default setting), it requests
load balancing across all the CPUs in that cpuset's allowed CPUs, ensuring that load bal‐
ancing can move a process (not otherwise pinned, as by sched_setaffinity(2)) from any CPU
in that cpuset to any other.
When the per-cpuset flag sched_load_balance is disabled, then the scheduler will avoid
load balancing across the CPUs in that cpuset, except in so far as is necessary because
some overlapping cpuset has sched_load_balance enabled.
So, for example, if the top cpuset has the flag sched_load_balance enabled, then the
scheduler will load balance across all CPUs, and the setting of the sched_load_balance
flag in other cpusets has no effect, as we're already fully load balancing.
Therefore in the above two situations, the flag sched_load_balance should be disabled in
the top cpuset, and only some of the smaller, child cpusets would have this flag enabled.
When doing this, you don't usually want to leave any unpinned processes in the top cpuset
that might use nontrivial amounts of CPU, as such processes may be artificially con‐
strained to some subset of CPUs, depending on the particulars of this flag setting in
descendant cpusets. Even if such a process could use spare CPU cycles in some other CPUs,
the kernel scheduler might not consider the possibility of load balancing that process to
the underused CPU.
Of course, processes pinned to a particular CPU can be left in a cpuset that disables
sched_load_balance as those processes aren't going anywhere else anyway.
Scheduler relax domain level
The kernel scheduler performs immediate load balancing whenever a CPU becomes free or
another task becomes runnable. This load balancing works to ensure that as many CPUs as
possible are usefully employed running tasks. The kernel also performs periodic load bal‐
ancing off the software clock described in time(7). The setting of
sched_relax_domain_level applies only to immediate load balancing. Regardless of the
sched_relax_domain_level setting, periodic load balancing is attempted over all CPUs
(unless disabled by turning off sched_load_balance.) In any case, of course, tasks will
be scheduled to run only on CPUs allowed by their cpuset, as modified by sched_setaffin‐
ity(2) system calls.
On small systems, such as those with just a few CPUs, immediate load balancing is useful
to improve system interactivity and to minimize wasteful idle CPU cycles. But on large
systems, attempting immediate load balancing across a large number of CPUs can be more
costly than it is worth, depending on the particular performance characteristics of the
job mix and the hardware.
The exact meaning of the small integer values of sched_relax_domain_level will depend on
internal implementation details of the kernel scheduler code and on the non-uniform archi‐
tecture of the hardware. Both of these will evolve over time and vary by system architec‐
ture and kernel version.
As of this writing, when this capability was introduced in Linux 2.6.26, on certain popu‐
lar architectures, the positive values of sched_relax_domain_level have the following
meanings.
(1) Perform immediate load balancing across Hyper-Thread siblings on the same core.
(2) Perform immediate load balancing across other cores in the same package.
(3) Perform immediate load balancing across other CPUs on the same node or blade.
(4) Perform immediate load balancing across over several (implementation detail) nodes [On
NUMA systems].
(5) Perform immediate load balancing across over all CPUs in system [On NUMA systems].
The sched_relax_domain_level value of zero (0) always means don't perform immediate load
balancing, hence that load balancing is done only periodically, not immediately when a CPU
becomes available or another task becomes runnable.
The sched_relax_domain_level value of minus one (-1) always means use the system default
value. The system default value can vary by architecture and kernel version. This system
default value can be changed by kernel boot-time "relax_domain_level=" argument.
In the case of multiple overlapping cpusets which have conflicting
sched_relax_domain_level values, then the highest such value applies to all CPUs in any of
the overlapping cpusets. In such cases, the value minus one (-1) is the lowest value,
overridden by any other value, and the value zero (0) is the next lowest value.
FORMATS
The following formats are used to represent sets of CPUs and memory nodes.
Mask format
The Mask Format is used to represent CPU and memory-node bit masks in the /proc/<pid>/sta‐
tus file.
This format displays each 32-bit word in hexadecimal (using ASCII characters "0" - "9" and
"a" - "f"); words are filled with leading zeros, if required. For masks longer than one
word, a comma separator is used between words. Words are displayed in big-endian order,
which has the most significant bit first. The hex digits within a word are also in big-
endian order.
The number of 32-bit words displayed is the minimum number needed to display all bits of
the bit mask, based on the size of the bit mask.
Examples of the Mask Format:
00000001 # just bit 0 set
40000000,00000000,00000000 # just bit 94 set
00000001,00000000,00000000 # just bit 64 set
000000ff,00000000 # bits 32-39 set
00000000,000e3862 # 1,5,6,11-13,17-19 set
A mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:
00000001,00000001,00010117
The first "1" is for bit 64, the second for bit 32, the third for bit 16, the fourth for
bit 8, the fifth for bit 4, and the "7" is for bits 2, 1, and 0.
List format
The List Format for cpus and mems is a comma-separated list of CPU or memory-node numbers
and ranges of numbers, in ASCII decimal.
Examples of the List Format:
0-4,9 # bits 0, 1, 2, 3, 4, and 9 set
0-2,7,12-14 # bits 0, 1, 2, 7, 12, 13, and 14 set
RULES
The following rules apply to each cpuset:
* Its CPUs and memory nodes must be a (possibly equal) subset of its parent's.
* It can be marked cpu_exclusive only if its parent is.
* It can be marked mem_exclusive only if its parent is.
* If it is cpu_exclusive, its CPUs may not overlap any sibling.
* If it is memory_exclusive, its memory nodes may not overlap any sibling.
PERMISSIONS
The permissions of a cpuset are determined by the permissions of the directories and
pseudo-files in the cpuset filesystem, normally mounted at /dev/cpuset.
For instance, a process can put itself in some other cpuset (than its current one) if it
can write the tasks file for that cpuset. This requires execute permission on the encom‐
passing directories and write permission on the tasks file.
An additional constraint is applied to requests to place some other process in a cpuset.
One process may not attach another to a cpuset unless it would have permission to send
that process a signal (see kill(2)).
A process may create a child cpuset if it can access and write the parent cpuset direc‐
tory. It can modify the CPUs or memory nodes in a cpuset if it can access that cpuset's
directory (execute permissions on the each of the parent directories) and write the corre‐
sponding cpus or mems file.
There is one minor difference between the manner in which these permissions are evaluated
and the manner in which normal filesystem operation permissions are evaluated. The kernel
interprets relative pathnames starting at a process's current working directory. Even if
one is operating on a cpuset file, relative pathnames are interpreted relative to the
process's current working directory, not relative to the process's current cpuset. The
only ways that cpuset paths relative to a process's current cpuset can be used are if
either the process's current working directory is its cpuset (it first did a cd or
chdir(2) to its cpuset directory beneath /dev/cpuset, which is a bit unusual) or if some
user code converts the relative cpuset path to a full filesystem path.
In theory, this means that user code should specify cpusets using absolute pathnames,
which requires knowing the mount point of the cpuset filesystem (usually, but not neces‐
sarily, /dev/cpuset). In practice, all user level code that this author is aware of sim‐
ply assumes that if the cpuset filesystem is mounted, then it is mounted at /dev/cpuset.
Furthermore, it is common practice for carefully written user code to verify the presence
of the pseudo-file /dev/cpuset/tasks in order to verify that the cpuset pseudo-filesystem
is currently mounted.
WARNINGS
Enabling memory_pressure
By default, the per-cpuset file cpuset.memory_pressure always contains zero (0). Unless
this feature is enabled by writing "1" to the pseudo-file /dev/cpuset/cpuset.memory_pres‐
sure_enabled, the kernel does not compute per-cpuset memory_pressure.
Using the echo command
When using the echo command at the shell prompt to change the values of cpuset files,
beware that the built-in echo command in some shells does not display an error message if
the write(2) system call fails. For example, if the command:
echo 19 > cpuset.mems
failed because memory node 19 was not allowed (perhaps the current system does not have a
memory node 19), then the echo command might not display any error. It is better to use
the /bin/echo external command to change cpuset file settings, as this command will dis‐
play write(2) errors, as in the example:
/bin/echo 19 > cpuset.mems
/bin/echo: write error: Invalid argument
EXCEPTIONS
Memory placement
Not all allocations of system memory are constrained by cpusets, for the following rea‐
sons.
If hot-plug functionality is used to remove all the CPUs that are currently assigned to a
cpuset, then the kernel will automatically update the cpus_allowed of all processes
attached to CPUs in that cpuset to allow all CPUs. When memory hot-plug functionality for
removing memory nodes is available, a similar exception is expected to apply there as
well. In general, the kernel prefers to violate cpuset placement, rather than starving a
process that has had all its allowed CPUs or memory nodes taken offline. User code should
reconfigure cpusets to refer only to online CPUs and memory nodes when using hot-plug to
add or remove such resources.
A few kernel-critical, internal memory-allocation requests, marked GFP_ATOMIC, must be
satisfied immediately. The kernel may drop some request or malfunction if one of these
allocations fail. If such a request cannot be satisfied within the current process's
cpuset, then we relax the cpuset, and look for memory anywhere we can find it. It's bet‐
ter to violate the cpuset than stress the kernel.
Allocations of memory requested by kernel drivers while processing an interrupt lack any
relevant process context, and are not confined by cpusets.
Renaming cpusets
You can use the rename(2) system call to rename cpusets. Only simple renaming is sup‐
ported; that is, changing the name of a cpuset directory is permitted, but moving a direc‐
tory into a different directory is not permitted.
ERRORS
The Linux kernel implementation of cpusets sets errno to specify the reason for a failed
system call affecting cpusets.
The possible errno settings and their meaning when set on a failed cpuset call are as
listed below.
E2BIG Attempted a write(2) on a special cpuset file with a length larger than some ker‐
nel-determined upper limit on the length of such writes.
EACCES Attempted to write(2) the process ID (PID) of a process to a cpuset tasks file when
one lacks permission to move that process.
EACCES Attempted to add, using write(2), a CPU or memory node to a cpuset, when that CPU
or memory node was not already in its parent.
EACCES Attempted to set, using write(2), cpuset.cpu_exclusive or cpuset.mem_exclusive on a
cpuset whose parent lacks the same setting.
EACCES Attempted to write(2) a cpuset.memory_pressure file.
EACCES Attempted to create a file in a cpuset directory.
EBUSY Attempted to remove, using rmdir(2), a cpuset with attached processes.
EBUSY Attempted to remove, using rmdir(2), a cpuset with child cpusets.
EBUSY Attempted to remove a CPU or memory node from a cpuset that is also in a child of
that cpuset.
EEXIST Attempted to create, using mkdir(2), a cpuset that already exists.
EEXIST Attempted to rename(2) a cpuset to a name that already exists.
EFAULT Attempted to read(2) or write(2) a cpuset file using a buffer that is outside the
writing processes accessible address space.
EINVAL Attempted to change a cpuset, using write(2), in a way that would violate a
cpu_exclusive or mem_exclusive attribute of that cpuset or any of its siblings.
EINVAL Attempted to write(2) an empty cpuset.cpus or cpuset.mems list to a cpuset which
has attached processes or child cpusets.
EINVAL Attempted to write(2) a cpuset.cpus or cpuset.mems list which included a range with
the second number smaller than the first number.
EINVAL Attempted to write(2) a cpuset.cpus or cpuset.mems list which included an invalid
character in the string.
EINVAL Attempted to write(2) a list to a cpuset.cpus file that did not include any online
CPUs.
EINVAL Attempted to write(2) a list to a cpuset.mems file that did not include any online
memory nodes.
EINVAL Attempted to write(2) a list to a cpuset.mems file that included a node that held
no memory.
EIO Attempted to write(2) a string to a cpuset tasks file that does not begin with an
ASCII decimal integer.
EIO Attempted to rename(2) a cpuset into a different directory.
ENAMETOOLONG
Attempted to read(2) a /proc/<pid>/cpuset file for a cpuset path that is longer
than the kernel page size.
ENAMETOOLONG
Attempted to create, using mkdir(2), a cpuset whose base directory name is longer
than 255 characters.
ENAMETOOLONG
Attempted to create, using mkdir(2), a cpuset whose full pathname, including the
mount point (typically "/dev/cpuset/") prefix, is longer than 4095 characters.
ENODEV The cpuset was removed by another process at the same time as a write(2) was
attempted on one of the pseudo-files in the cpuset directory.
ENOENT Attempted to create, using mkdir(2), a cpuset in a parent cpuset that doesn't
exist.
ENOENT Attempted to access(2) or open(2) a nonexistent file in a cpuset directory.
ENOMEM Insufficient memory is available within the kernel; can occur on a variety of sys‐
tem calls affecting cpusets, but only if the system is extremely short of memory.
ENOSPC Attempted to write(2) the process ID (PID) of a process to a cpuset tasks file when
the cpuset had an empty cpuset.cpus or empty cpuset.mems setting.
ENOSPC Attempted to write(2) an empty cpuset.cpus or cpuset.mems setting to a cpuset that
has tasks attached.
ENOTDIR
Attempted to rename(2) a nonexistent cpuset.
EPERM Attempted to remove a file from a cpuset directory.
ERANGE Specified a cpuset.cpus or cpuset.mems list to the kernel which included a number
too large for the kernel to set in its bit masks.
ESRCH Attempted to write(2) the process ID (PID) of a nonexistent process to a cpuset
tasks file.
VERSIONS
Cpusets appeared in version 2.6.12 of the Linux kernel.
NOTES
Despite its name, the pid parameter is actually a thread ID, and each thread in a threaded
group can be attached to a different cpuset. The value returned from a call to gettid(2)
can be passed in the argument pid.
BUGS
cpuset.memory_pressure cpuset files can be opened for writing, creation, or truncation,
but then the write(2) fails with errno set to EACCES, and the creation and truncation
options on open(2) have no effect.
EXAMPLE
The following examples demonstrate querying and setting cpuset options using shell com‐
mands.
Creating and attaching to a cpuset.
To create a new cpuset and attach the current command shell to it, the steps are:
1) mkdir /dev/cpuset (if not already done)
2) mount -t cpuset none /dev/cpuset (if not already done)
3) Create the new cpuset using mkdir(1).
4) Assign CPUs and memory nodes to the new cpuset.
5) Attach the shell to the new cpuset.
For example, the following sequence of commands will set up a cpuset named "Charlie", con‐
taining just CPUs 2 and 3, and memory node 1, and then attach the current shell to that
cpuset.
$ mkdir /dev/cpuset
$ mount -t cpuset cpuset /dev/cpuset
$ cd /dev/cpuset
$ mkdir Charlie
$ cd Charlie
$ /bin/echo 2-3 > cpuset.cpus
$ /bin/echo 1 > cpuset.mems
$ /bin/echo $$ > tasks
# The current shell is now running in cpuset Charlie
# The next line should display '/Charlie'
$ cat /proc/self/cpuset
Migrating a job to different memory nodes.
To migrate a job (the set of processes attached to a cpuset) to different CPUs and memory
nodes in the system, including moving the memory pages currently allocated to that job,
perform the following steps.
1) Let's say we want to move the job in cpuset alpha (CPUs 4-7 and memory nodes 2-3) to a
new cpuset beta (CPUs 16-19 and memory nodes 8-9).
2) First create the new cpuset beta.
3) Then allow CPUs 16-19 and memory nodes 8-9 in beta.
4) Then enable memory_migration in beta.
5) Then move each process from alpha to beta.
The following sequence of commands accomplishes this.
$ cd /dev/cpuset
$ mkdir beta
$ cd beta
$ /bin/echo 16-19 > cpuset.cpus
$ /bin/echo 8-9 > cpuset.mems
$ /bin/echo 1 > cpuset.memory_migrate
$ while read i; do /bin/echo $i; done < ../alpha/tasks > tasks
The above should move any processes in alpha to beta, and any memory held by these pro‐
cesses on memory nodes 2-3 to memory nodes 8-9, respectively.
Notice that the last step of the above sequence did not do:
$ cp ../alpha/tasks tasks
The while loop, rather than the seemingly easier use of the cp(1) command, was necessary
because only one process PID at a time may be written to the tasks file.
The same effect (writing one PID at a time) as the while loop can be accomplished more
efficiently, in fewer keystrokes and in syntax that works on any shell, but alas more
obscurely, by using the -u (unbuffered) option of sed(1):
$ sed -un p < ../alpha/tasks > tasks
SEE ALSO
taskset(1), get_mempolicy(2), getcpu(2), mbind(2), sched_getaffinity(2), sched_setaffin‐
ity(2), sched_setscheduler(2), set_mempolicy(2), CPU_SET(3), proc(5), numa(7), sched(7),
migratepages(8), numactl(8)
Documentation/cpusets.txt in the Linux kernel source tree
COLOPHON
This page is part of release 3.74 of the Linux man-pages project. A description of the
project, information about reporting bugs, and the latest version of this page, can be
found at http://www.kernel.org/doc/man-pages/.
Linux 2014-05-21 CPUSET(7)
|
