2017 - Diagnosing process memory use in Linux
The Slovak original of this document: 2017 - Diagnostika využívania pamäte procesov v OS Linux (slovensky).
1 Tools for diagnosing memory use in Linux
ps displays information about a selection of the active processes. If you want a repetitive update of the selection and the displayed information, use top(1) instead.
The pmap command reports the memory map of a process or processes.
smem is a tool that can give numerous reports on memory usage on Linux systems. Unlike existing tools, smem can report proportional set size (PSS), which is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system. Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. PSS instead measures each application's "fair share" of each shared area to give a realistic measure.
github raw - ps_mem.py Try to determine how much RAM is currently being used per program. Note per _program_, not per process. So for example this script will report RAM used by all httpd process together. In detail it reports: sum(private RAM for program processes) + sum(Shared RAM for program processes). The shared RAM is problematic to calculate, and this script automatically selects the most accurate method available for your kernel.
2 Reading the memory figures the diagnostic tools report
A short introduction to reading the memory figures the diagnostic tools report:
- VSS (Virtual Set Size) / VSZ (Virtual Memory Size):
The size of the whole virtual memory of the process, that is, of the address space available to it, whether or not its pages are resident in physical memory. The trouble with this figure is that it includes memory that was allocated, say with "malloc()", but never written to. So it is not a good figure for judging what a program really uses.
- RSS (Resident Set Size):
How much memory the process is using right now — the part of it held in RAM. Put another way, the total number of the process's pages currently resident in physical memory. The trouble with this figure is that it includes the memory used by the shared libraries the process uses. If a shared library "lib" is used by two programs ("proces1" and "proces2"), size "V" is allocated for "lib" once — but adding the two RSS figures counts that shared memory twice, needlessly and wrongly. So this is not a good figure for judging what a single program really uses.
- PSS (Proportional Set Size):
Much like RSS, except that the memory allocated for a shared library is divided by the number of processes using it. If a shared library "lib" takes 100 KB and two programs ("proces1" and "proces2") use it, PSS = 100/2 = 50 KB. That makes it better than RSS, but still not a good figure for judging what a single program uses. Its best use is summed over every process, which gives a good view of what the whole system is using.
- USS (Unique set size):
How much private memory the process is using right now — the memory that is unique to it. That makes this the best figure for judging what a single program really uses.
The tools that report memory use give the same figures different column names. The table below sets the differences out.
+--------------------+---------+-------------------+---------+-------------------------------+ | Program\Figure | VSS | RSS | PSS | USS | ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ | ps | VSZ | RSS | - | - | +--------------------+---------+-------------------+---------+-------------------------------+ | pmap -x | Kbytes | RSS | - | - | +--------------------+---------+-------------------+---------+-------------------------------+ | pmap -X | Size | RSS | PSS | - | +--------------------+---------+-------------------+---------+-------------------------------+ | pmap -XX | Size | RSS | PSS | Private_Clean + Private_Dirty | +--------------------+---------+-------------------+---------+-------------------------------+ | smem | VSS | RSS | PSS | USS | +--------------------+---------+-------------------+---------+-------------------------------+ | ps_mem.py | - | Shared + Private | Shared | Private | +--------------------+---------+-------------------+---------+-------------------------------+
Points [2.1] to [2.4] run "ps", "smem", "pmap" and "ps_mem.py" so that their output is as close to the same as it can be made.
2.1 Running "ps" and reading the memory figures for PID=100886
# ps -eo pid,comm,vsz,rss,drs,trs -q 100886 ---------------------------------------------------------------------------------------------------------------- PID COMMAND VSZ RSS 100886 mc 163996 5364
2.2 Running "smem" and reading the memory figures for PID=100886
Note 1: For some reason "smem" always includes a row for the "smem" process itself, which is written in Python, so row 2 holds the memory figures for the Python interpreter.
Note 2: Given note 1, it might be worth writing my own tool.
# ./smem -c 'pid name maps vss uss pss rss' -P /usr/bin/mc ---------------------------------------------------------------------------------------------------------------- PID Name Maps VSS USS PSS RSS 100886 mc 118 164000 2932 3213 5364 101585 python 82 146072 4660 5342 6836
2.3 Running "pmap" and reading the memory figures for PID=100886
Note: the output is long and fairly involved, so only the first two and last two rows are shown.
# pmap -XX 100886 | head -n 2; pmap -XX 100886 | tail -n 2;
----------------------------------------------------------------------------------------------------------------
100886: /usr/bin/mc -P /tmp/mc-root/mc.pwd.100867
Address Perm Offset Device Inode Size Rss Pss Shared_Clean Shared_Dirty Private_Clean Private_Dirty Referenced Anonymous AnonHugePages Swap KernelPageSize MMUPageSize Locked
====== ==== ==== ============ ============ ============= ============= ========== ========= ============= ==== ============== =========== ======
164000 5364 3210 2432 0 1200 1732 5364 1732 0 0 472 472 02.4 Running "ps_mem.py" and reading the memory figures for PID=100886
# ./ps_mem.py -p 100886
----------------------------------------------------------------------------------------------------------------
Private + Shared = RAM used Program
2.9 MiB + 340.0 KiB = 3.2 MiB mc
---------------------------------
3.2 MiB3 Installing "smem" and an example of its use
3.1 Downloading and "installing" smem
# cd /install # wget https://www.selenic.com/smem/download/smem-1.4.tar.gz # tar -xvzf ./smem-1.4.tar.gz
3.2 Running smem
# cd /install/smem-1.4 # ./smem
4 Installing "ps_mem.py" and an example of its use
4.1 Downloading and "installing" ps_mem.py
# cd /install # wget https://raw.githubusercontent.com/pixelb/ps_mem/master/ps_mem.py # chmod +x ps_mem.py # ./ps_mem.py
4.2 Running ps_mem.py for PID=100886
# /install/ps_mem.py -p 100886
4.3 Running ps_mem.py for PID=100886 with the output refreshed every second
# /install/ps_mem.py -p 100886 -w 1
5 Using "smem" in earnest to measure the memory used by "mc" (Midnight Commander)
Two log files are used to measure the memory the "mc" process uses:
+----+--------------------------------+---------------+ | ID | File | Size | +++++++++++++++++++++++++++++++++++++++++++++++++++++++ | 1 | /var/log/messages-20161114 | 1,5 MB | +----+--------------------------------+---------------+ | 2 | /var/log/messages-20161120 | 588 KB | +----+--------------------------------+---------------+
Measuring the memory the "mc" process uses was done in these steps:
5.1 - The output shows what the "mc" process (Midnight Commander) is using right now. USS for "mc" is 2.9 MB.
Showing memory use
- pre konkretny proces (-P)
- vypiseme sumar/total (-t)
- vypiseme jednotky K,M,G,... (-k)
- vypiseme stlpce "pid", "name", "maps", "vss", "uss", "pss", "rss", "swap" (-c)
----------------------------------------------------------------------------------------------------------------
# ./smem -c 'pid name maps vss uss pss rss swap' -P /usr/bin/mc -t -k
----------------------------------------------------------------------------------------------------------------
PID Name Maps VSS USS PSS RSS Swap
100886 mc 118 160.2M 2.9M 3.1M 5.2M 0
101240 python 81 142.7M 4.6M 5.2M 6.7M 0
----------------------------------------------------------------------------------------------------------------
2 199 302.9M 7.4M 8.4M 11.9M 05.2 - The output shows what the "mc" process (Midnight Commander) is using with file ID 1 open for editing. USS for "mc" has risen from 2.9 to 4.2 MB.
# ./smem -c 'pid name maps vss uss pss rss swap' -P /usr/bin/mc -t -k
----------------------------------------------------------------------------------------------------------------
PID Name Maps VSS USS PSS RSS Swap
100886 mc 118 161.7M 4.2M 4.5M 6.6M 0
101245 python 81 142.7M 4.6M 5.2M 6.7M 0
----------------------------------------------------------------------------------------------------------------
2 199 304.4M 8.8M 9.8M 13.3M 05.3 - The output shows what the "mc" process (Midnight Commander) is using after editing of file ID 1 has finished. The process has given back the memory it allocated for editing, and USS is comparable with output [5.1].
# ./smem -c 'pid name maps vss uss pss rss swap' -P /usr/bin/mc -t -k
----------------------------------------------------------------------------------------------------------------
PID Name Maps VSS USS PSS RSS Swap
100886 mc 118 160.2M 2.9M 3.1M 5.2M 0
101250 python 81 142.7M 4.6M 5.2M 6.7M 0
----------------------------------------------------------------------------------------------------------------
2 199 302.9M 7.4M 8.4M 11.9M 05.4 - The output shows what the "mc" process (Midnight Commander) is using with file ID 2 open for editing. USS for "mc" has risen from 2.9 to 3.4 MB.
# ./smem -c 'pid name maps vss uss pss rss swap' -P /usr/bin/mc -t -k
----------------------------------------------------------------------------------------------------------------
PID Name Maps VSS USS PSS RSS Swap
100886 mc 118 160.7M 3.4M 3.6M 5.7M 0
101273 python 81 142.7M 4.6M 5.2M 6.7M 0
----------------------------------------------------------------------------------------------------------------
2 199 303.4M 7.9M 8.9M 12.4M 05.5 - The output shows what the "mc" process (Midnight Commander) is using after editing of file ID 2 has finished. The process has given back the memory it allocated for editing, and USS is comparable with output [5.1] and [5.3].
# ./smem -c 'pid name maps vss uss pss rss swap' -P /usr/bin/mc -t -k
----------------------------------------------------------------------------------------------------------------
PID Name Maps VSS USS PSS RSS Swap
100886 mc 118 160.2M 2.9M 3.1M 5.2M 0
101275 python 81 142.7M 4.6M 5.2M 6.7M 0
----------------------------------------------------------------------------------------------------------------
2 199 302.9M 7.4M 8.4M 11.9M 05.6 - The output shows what the "mc" process (Midnight Commander) is using right now, listing every mapping the process holds as well.
Show memory use:
- for one particular process (-P)
- print the total (-t)
- print the units K, M, G, ... (-k)
- print every mapping (-m)
- print the columns "pids", "map", "vss", "uss", "pss", "rss", "swap" (-c)
----------------------------------------------------------------------------------------------------------------
# ./smem -c 'pids map vss uss pss rss swap' -m -P /usr/bin/mc -t -k
----------------------------------------------------------------------------------------------------------------
PIDs Map VSS USS PSS RSS Swap
2 [vdso] 16.0K 0 0 8.0K 0
2 [vsyscall] 8.0K 0 0 0 0
1 /usr/lib64/gconv/gconv-modules.cache 28.0K 0 3.0K 24.0K 0
2 /usr/lib/locale/locale-archive 202.3M 0 7.0K 84.0K 0
1 /usr/lib64/libcom_err.so.2.1 2.0M 8.0K 8.0K 16.0K 0
1 /usr/lib64/libkeyutils.so.1.5 2.0M 8.0K 8.0K 16.0K 0
1 /usr/lib64/libpcre.so.1.2.0 2.4M 8.0K 8.0K 16.0K 0
1 /usr/lib64/libutil-2.17.so 2.0M 8.0K 8.0K 12.0K 0
1 /usr/lib64/libz.so.1.2.7 2.1M 8.0K 8.0K 20.0K 0
1 /usr/lib64/libkrb5support.so.0.1 2.1M 8.0K 9.0K 24.0K 0
1 /usr/lib64/libselinux.so.1 2.1M 8.0K 9.0K 40.0K 0
1 /usr/bin/python2.7 12.0K 8.0K 10.0K 12.0K 0
1 /usr/lib64/libgmodule-2.0.so.0.4600.2 2.0M 8.0K 10.0K 20.0K 0
1 /usr/lib64/libnss_files-2.17.so 2.1M 8.0K 10.0K 40.0K 0
1 /usr/lib64/libnss_dns-2.17.so 2.0M 8.0K 11.0K 20.0K 0
1 /usr/lib64/python2.7/lib-dynload/_functo 2.0M 8.0K 12.0K 16.0K 0
1 /usr/lib64/libk5crypto.so.3.1 2.2M 12.0K 13.0K 32.0K 0
1 /usr/lib64/python2.7/lib-dynload/_locale 2.0M 8.0K 16.0K 24.0K 0
1 /usr/lib64/python2.7/lib-dynload/grpmodu 2.0M 16.0K 16.0K 16.0K 0
2 /usr/lib64/libdl-2.17.so 4.0M 16.0K 16.0K 32.0K 0
1 /usr/lib64/libgssapi_krb5.so.2.2 2.3M 12.0K 17.0K 64.0K 0
1 /usr/lib64/python2.7/lib-dynload/_heapq. 2.0M 12.0K 18.0K 24.0K 0
1 /usr/lib64/libresolv-2.17.so 2.1M 8.0K 19.0K 64.0K 0
1 /usr/lib64/libgpm.so.2.1.0 2.0M 20.0K 20.0K 20.0K 0
1 /usr/lib64/python2.7/lib-dynload/_collec 2.0M 12.0K 20.0K 28.0K 0
1 /usr/lib64/python2.7/lib-dynload/_struct 2.0M 12.0K 20.0K 28.0K 0
1 /usr/lib64/python2.7/lib-dynload/stropmo 2.0M 12.0K 20.0K 28.0K 0
1 /usr/lib64/python2.7/lib-dynload/timemod 2.0M 12.0K 20.0K 28.0K 0
2 /usr/lib64/libpthread-2.17.so 4.2M 16.0K 20.0K 136.0K 0
2 /usr/lib64/ld-2.17.so 272.0K 16.0K 23.0K 244.0K 0
2 /usr/lib64/libm-2.17.so 6.0M 16.0K 24.0K 148.0K 0
1 /usr/lib64/python2.7/lib-dynload/operato 2.0M 12.0K 26.0K 40.0K 0
1 /usr/lib64/python2.7/lib-dynload/itertoo 2.1M 24.0K 36.0K 48.0K 0
1 /usr/lib64/libssh2.so.1.0.1 2.2M 40.0K 40.0K 40.0K 0
1 /usr/lib64/libssl.so.1.0.1e 2.4M 44.0K 55.0K 124.0K 0
2 [stack] 272.0K 64.0K 64.0K 64.0K 0
1 /usr/lib64/libkrb5.so.3.3 2.9M 68.0K 82.0K 212.0K 0
2 /usr/lib64/libc-2.17.so 7.5M 48.0K 111.0K 1.4M 0
1 /usr/lib64/libcrypto.so.1.0.1e 3.9M 152.0K 199.0K 564.0K 0
1 /usr/lib64/libglib-2.0.so.0.4600.2 3.2M 72.0K 199.0K 528.0K 0
1 /usr/lib64/libslang.so.2.2.4 3.1M 528.0K 528.0K 528.0K 0
1 /usr/bin/mc 1.1M 704.0K 704.0K 704.0K 0
1 /usr/lib64/libpython2.7.so.1.0 3.7M 232.0K 800.0K 1.3M 0
2 <anonymous> 2.3M 1.8M 1.8M 1.8M 0
2 [heap] 3.7M 3.4M 3.4M 3.4M 0
--------------------------------------------------------------------------------------------
56 45 302.8M 7.4M 8.4M 11.9M 0