MySQL, OOM killer, overcommitting and other memory related issues
Posted Under: Internet,Linux,Server admin,Virtualization
It started with an error message
This post is a bit of a coredump of myself attempting to resolve a sudden web server failure. And even more important, understand why it happened (check on that) and try avoiding it from happening in the future (not as lucky there).
I’ve noticed that there are many threads in the Internet on why mysqld died suddenly, so to make a long story short: mysqld has the exact profile that the OOM killer is looking for: Lots of resident RAM, and it’s not a system process. Apache gets killed every now and then for the same reason.
This post relates to a VPS hosted Debian 8, kernel 3.10.0, x86_64. The MySQL server is a 5.5.62-0+deb8u1 (Debian).
As always, it started with a mail notification from some cronjob complaining about something. Soon enough it was evident that the MySQL server was down. And as usual, the deeper I investigated this issue, the more I realized that this was just the tip of the iceberg (the kind that doesn’t melt due to global warming).
The crash
So first, it was clear that the MySQL had restarted itself a couple of days before disaster:
191007 9:25:17 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 191007 9:25:17 [Note] Plugin 'FEDERATED' is disabled. 191007 9:25:17 InnoDB: The InnoDB memory heap is disabled 191007 9:25:17 InnoDB: Mutexes and rw_locks use GCC atomic builtins 191007 9:25:17 InnoDB: Compressed tables use zlib 1.2.8 191007 9:25:17 InnoDB: Using Linux native AIO 191007 9:25:17 InnoDB: Initializing buffer pool, size = 128.0M 191007 9:25:17 InnoDB: Completed initialization of buffer pool 191007 9:25:17 InnoDB: highest supported file format is Barracuda. InnoDB: The log sequence number in ibdata files does not match InnoDB: the log sequence number in the ib_logfiles! 191007 9:25:17 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 191007 9:25:19 InnoDB: Waiting for the background threads to start 191007 9:25:20 InnoDB: 5.5.62 started; log sequence number 1427184442 191007 9:25:20 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306 191007 9:25:20 [Note] - '127.0.0.1' resolves to '127.0.0.1'; 191007 9:25:20 [Note] Server socket created on IP: '127.0.0.1'. 191007 9:25:21 [Note] Event Scheduler: Loaded 0 events 191007 9:25:21 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.62-0+deb8u1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Debian) 191007 9:25:28 [ERROR] /usr/sbin/mysqld: Table './mydb/wp_options' is marked as crashed and should be repaired 191007 9:25:28 [Warning] Checking table: './mydb/wp_options' 191007 9:25:28 [ERROR] /usr/sbin/mysqld: Table './mydb/wp_posts' is marked as crashed and should be repaired 191007 9:25:28 [Warning] Checking table: './mydb/wp_posts' 191007 9:25:28 [ERROR] /usr/sbin/mysqld: Table './mydb/wp_term_taxonomy' is marked as crashed and should be repaired 191007 9:25:28 [Warning] Checking table: './mydb/wp_term_taxonomy' 191007 9:25:28 [ERROR] /usr/sbin/mysqld: Table './mydb/wp_term_relationships' is marked as crashed and should be repaired 191007 9:25:28 [Warning] Checking table: './mydb/wp_term_relationships'
And then, two days layer, it crashed for real. Or actually, got killed. From the syslog:
Oct 09 05:30:16 kernel: OOM killed process 22763 (mysqld) total-vm:2192796kB, anon-rss:128664kB, file-rss:0kB
and
191009 5:30:17 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
191009 5:30:17 [Note] Plugin 'FEDERATED' is disabled.
191009 5:30:17 InnoDB: The InnoDB memory heap is disabled
191009 5:30:17 InnoDB: Mutexes and rw_locks use GCC atomic builtins
191009 5:30:17 InnoDB: Compressed tables use zlib 1.2.8
191009 5:30:17 InnoDB: Using Linux native AIO
191009 5:30:17 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
191009 5:30:17 InnoDB: Completed initialization of buffer pool
191009 5:30:17 InnoDB: Fatal error: cannot allocate memory for the buffer pool
191009 5:30:17 [ERROR] Plugin 'InnoDB' init function returned error.
191009 5:30:17 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
191009 5:30:17 [ERROR] Unknown/unsupported storage engine: InnoDB
191009 5:30:17 [ERROR] Aborting
191009 5:30:17 [Note] /usr/sbin/mysqld: Shutdown complete
The mmap() is most likely anonymous (i.e. not related to a file), as I couldn’t find any memory mapped file that is related to the mysql processes (except for the obvious mappings of shared libraries).
The smoking gun
But here comes the good part: It turns out that the OOM killer had been active several times before. It just so happen that the processes are being newborn every time this happens. It was the relaunch that failed this time — otherwise I wouldn’t have noticed this was going on.
This is the output of plain “dmesg”. All OOM entries but the last one were not available with journalctl, as old entries had been deleted.
[3634197.152028] OOM killed process 776 (mysqld) total-vm:2332652kB, anon-rss:153508kB, file-rss:0kB [3634197.273914] OOM killed process 71 (systemd-journal) total-vm:99756kB, anon-rss:68592kB, file-rss:4kB [4487991.904510] OOM killed process 3817 (mysqld) total-vm:2324456kB, anon-rss:135752kB, file-rss:0kB [4835006.413510] OOM killed process 23267 (mysqld) total-vm:2653112kB, anon-rss:131272kB, file-rss:4404kB [4835006.767112] OOM killed process 32758 (apache2) total-vm:282528kB, anon-rss:11732kB, file-rss:52kB [4884915.371805] OOM killed process 825 (mysqld) total-vm:2850312kB, anon-rss:121164kB, file-rss:5028kB [4884915.509686] OOM killed process 17611 (apache2) total-vm:282668kB, anon-rss:11736kB, file-rss:444kB [5096265.088151] OOM killed process 23782 (mysqld) total-vm:4822232kB, anon-rss:105972kB, file-rss:3784kB [5845437.591031] OOM killed process 24642 (mysqld) total-vm:2455744kB, anon-rss:137784kB, file-rss:0kB [5845437.608682] OOM killed process 3802 (systemd-journal) total-vm:82548kB, anon-rss:51412kB, file-rss:28kB [6896254.741732] OOM killed process 11551 (mysqld) total-vm:2718652kB, anon-rss:144116kB, file-rss:220kB [7054957.856153] OOM killed process 22763 (mysqld) total-vm:2192796kB, anon-rss:128664kB, file-rss:0kB
Or, after calculating the time stamps (using the last OOM message as a reference):
Fri Aug 30 15:17:36 2019 OOM killed process 776 (mysqld) total-vm:2332652kB, anon-rss:153508kB, file-rss:0kB Fri Aug 30 15:17:36 2019 OOM killed process 71 (systemd-journal) total-vm:99756kB, anon-rss:68592kB, file-rss:4kB Mon Sep 9 12:27:30 2019 OOM killed process 3817 (mysqld) total-vm:2324456kB, anon-rss:135752kB, file-rss:0kB Fri Sep 13 12:51:05 2019 OOM killed process 23267 (mysqld) total-vm:2653112kB, anon-rss:131272kB, file-rss:4404kB Fri Sep 13 12:51:05 2019 OOM killed process 32758 (apache2) total-vm:282528kB, anon-rss:11732kB, file-rss:52kB Sat Sep 14 02:42:54 2019 OOM killed process 825 (mysqld) total-vm:2850312kB, anon-rss:121164kB, file-rss:5028kB Sat Sep 14 02:42:54 2019 OOM killed process 17611 (apache2) total-vm:282668kB, anon-rss:11736kB, file-rss:444kB Mon Sep 16 13:25:24 2019 OOM killed process 23782 (mysqld) total-vm:4822232kB, anon-rss:105972kB, file-rss:3784kB Wed Sep 25 05:31:36 2019 OOM killed process 24642 (mysqld) total-vm:2455744kB, anon-rss:137784kB, file-rss:0kB Wed Sep 25 05:31:36 2019 OOM killed process 3802 (systemd-journal) total-vm:82548kB, anon-rss:51412kB, file-rss:28kB Mon Oct 7 09:25:13 2019 OOM killed process 11551 (mysqld) total-vm:2718652kB, anon-rss:144116kB, file-rss:220kB Wed Oct 9 05:30:16 2019 OOM killed process 22763 (mysqld) total-vm:2192796kB, anon-rss:128664kB, file-rss:0kB
So first, what do these mean numbers mean? There doesn’t seem to be an authoritative information source about this, but judging from different sources on the web, it goes like this:
- total-vm is the total size of the Virtual Memory in use. This isn’t very relevant (I think), as it involves shared libraries, memory mapped files and other segments that don’t consume any actual RAM or other valuable resources.
- anon-rss is the resident in physical RAM consumed by the process itself (anonymous = not memory mapped to a file or something like that).
- file-rss is the amount of memory that is resident in physical RAM, and is memory mapped to a file (for example, the executable binary).
Jusding from “top”, it’s quite typical for the mysql daemon to have a virtual memory allocation of about 4 GB, and resident memory of about 100-150 MB. The file-rss is most likely the database itself, that happened to be memory mapped (if at all) when the OOM happened to look for a victim.
So now it’s clear what happened, and it’s also quite clear that the mysql daemon did nothing irregular to get killed.
The MySQL keepaliver
The MySQL daemon is executed by virtue of an SysV init script, which launches /usr/bin/mysqld_safe, a patch-on-patch script to keep the daemon alive, no matter what. It restarts the mysqld daemon if it dies for any or no reason, and should also produce log messages. On my system, it’s executed as
/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
The script issues log messages when something unexpected happens, but they don’t appear in /var/log/mysql/error.log or anywhere else, even though the file exists, is owned by the mysql user, and has quite a few messages from the mysql daemon itself.
Changing
/usr/bin/mysqld_safe > /dev/null 2>&1 &
to
/usr/bin/mysqld_safe --syslog > /dev/null 2>&1 &
Frankly speaking, I don’t think this made any difference. I’ve seen nothing new in the logs.
It would have been nicer having the messages in mysql/error.log, but at least they are visible with journalctl this way.
Shrinking the InnoDB buffer pool
As the actual failure was on attempting to map memory for the buffer pool, maybe make it smaller…?
Launch MySQL as the root user:
$ mysql -u root --password
and check the InnoDB status, as suggested on this page:
mysql> SHOW ENGINE INNODB STATUS;
[ ... ]
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 137363456; in additional pool allocated 0
Dictionary memory allocated 1100748
Buffer pool size 8192
Free buffers 6263
Database pages 1912
Old database pages 725
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1894, created 18, written 1013
0.00 reads/s, 0.00 creates/s, 0.26 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 1912, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
I’m really not an expert, but if “Free buffers” is 75% of the total allocated space, I’ve probably allocated too much. So I reduced it to 32 MB — it’s not like I’m running a high-end server. I added /etc/mysql/conf.d/innodb_pool_size.cnf (owned by root, 0644) reading:
# Reduce InnoDB buffer size from default 128 MB to 32 MB [mysqld] innodb_buffer_pool_size=32M
Restarting the daemon, it says:
---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 34340864; in additional pool allocated 0 Dictionary memory allocated 1100748 Buffer pool size 2047 Free buffers 856 Database pages 1189 Old database pages 458
And finally, repair the tables
Remember those warnings that the tables were marked as crashed? That’s the easy part:
$ mysqlcheck -A --auto-repair
That went smoothly, with no complaints. After all, it wasn’t really a crash.
Some general words on OOM
This whole idea that the kernel should do Roman Empire style decimation of processes is widely criticized by many, but it’s probably not such a bad idea. The root cause lies in the fact that the kernel agrees to allocate more RAM than it actually has. This is even possible because the kernel doesn’t really allocate RAM when a process asks for memory with a brk() call, but it only allocates the memory space segment. The actual RAM is allocated only when the process attempts to access a page that hasn’t been RAM allocated yet. The access attempt causes a page fault, the kernel quickly fixes some RAM and returns from the page fault interrupt as if nothing happened.
So when the kernel responds with an -ENOMEM, it’s not because it doesn’t have any RAM, but because it doesn’t want to.
More precisely, the kernel keeps account on how much memory it has given away (system-wise and/or cgroup-wise) and make a decision. The common policy is to overcommit to some extent — that is, to allow the total allocated RAM allocated to exceed the total physical RAM. Even, and in particular, if there’s no swap.
The common figure is to overcommit by 50%: For a 64 GiB RAM computer, there might be 96 GiB or promised RAM. This may seem awfully stupid thing to do, but hey, it works. If that concept worries you, modern banking (with real money, that is) might worry you even more.
The problem rises when the processes run to the bank. That is, when the processes access the RAM they’ve been promised, and at some point the kernel has nowhere to take memory from. Let’s assume there’s no swap, all disk buffers have been flushed, all rabbits have been pulled. There’s a process waiting for memory, and it can’t go back running until the problem has been resolved.
Linux’ solution to this situation is to select a process with a lot of RAM and little importance. How the kernel does that judgement is documented everywhere. The important point is that it’s not necessarily the process that triggered the event, and that it will usually be the same victim over and over again. In my case, mysqld is the favorite. Big, fat, and not a system process.
Thinking about it, the OOM is a good solution to get out of a tricky situation. The alternative would have been to deny memory to processes just launched, including the administrator’s attempt to rescue the system. Or an attempt to shut it down with some dignity. So sacrificing a large and hopefully not-so-important process isn’t such a bad idea.
Why did the OOM kick in?
This all took place on a VPS virtual machine with 1 GB leased RAM. With the stuff running on that machine, there’s no reason in the world that the total actual RAM consumption would reach that limit. This is a system that typically has 70% of its memory marked as “cached” (i.e. used by disk cache). This should be taken with a grain of salt, as “top” displays data from some bogus /proc/vmstat, and still. See below on how to check the actual memory consumption.
As can be seen in the dmesg logs above, the amount of resident RAM of the killed mysqld process was 120-150 MB or so. Together with the other memory hog, apache2, they reach 300 MB. That’s it. No reason for anything drastic.
Having said that, it’s remarkable that the total-vm stood at 2.4-4.3 GB when killed. This is much higher than the typical 900 MB visible usually. So maybe there’s some kind of memory leak, even if it’s harmless? Looking at mysql over time, its virtual memory allocation tends to grow.
VPS machines do have a physical memory limit imposed, by virtue of the relevant cgroup’s memory.high and memory.max limits. In particular the latter — if the cgroup’s total consumption exceeds memory.max, OOM kicks in. This is how the illusion of an independent RAM segment is made on a VPS machine. Plus faking some /proc files.
But there’s another explanation: Say that a VPS service provider takes a computer with 16 GB RAM, and places 16 VPS machines with 1 GB leased RAM each. What will the overall actual RAM consumption be? I would expect it to be much lower than 16 GB. So why not add a few more VPS machines, and make some good use of the hardware? It’s where the profit comes from.
Most of the time, there will be no problem. But occasionally, this will cause RAM shortages, in which case the kernel’s global OOM looks for a victim. I suppose there’s no significance to cgroups in this matter. In other words, the kernel sees all processes in the system the same, regardless of which cgroup (and hence VPS machine) they belong to. Which means that the process killed doesn’t necessarily belong to the VPS that triggered the problem. The processes of one VPS may suddenly demand their memory, but some other VPS will have its processes killed.
Conclusion
- Shrinking the buffer pool of mysqld was probably a good idea, in particular if a computer-wide OOM killed the process — odds are that it will kill some other mysqld instead this way.
- Possibly restart mysql with a cronjob every day to keep its memory consumption in control. But this might create problems of its own.
- It’s high time to replace the VPS guest with KVM or similar.
Does my VPS need more RAM?
There’s “free” and “top” and several other utilities for telling you the status of the memory, but they don’t answer a simple question: Do the application eat up too much memory?
So the way to tell, is asking /proc/meminfo directly (note that this was made on my openVZ machine, not bare metal):
$ cat /proc/meminfo MemTotal: 1048576 kB MemFree: 34256 kB MemAvailable: 741536 kB Cached: 719728 kB Buffers: 0 kB Active: 485536 kB Inactive: 454964 kB Active(anon): 142584 kB Inactive(anon): 110608 kB Active(file): 342952 kB Inactive(file): 344356 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 1048576 kB SwapFree: 984960 kB Dirty: 48 kB Writeback: 0 kB AnonPages: 253192 kB Shmem: 32420 kB Slab: 35164 kB SReclaimable: 19972 kB SUnreclaim: 15192 kB
The tricky part about memory estimation is that the kernel attempts to use all RAM it has for something. So a lot goes to disk cache (“Cached”, 719728 kB in the example). This memory is reclaimed immediately if needed by an application, so it should be counted as free RAM.
Therefore, MemAvailable is the place to look. It’s a rough estimation of how much memory applications can ask for, and comparing it with MemTotal, clearly 70% of the memory is free. So the VPS server has plenty of RAM.
And yes, it’s a bit surpsiring that each VPS has its own disk cache, but it actually makes sense. Why should one guest wipe out another one’s disk cache?
There’s also swap memory allocated on the machine, almost all of which is unused. This is a common situation when there’s no shortage of memory. The amount of swap memory in use (SwapTotal – SwapFree) should be added to calculation on how much memory application use (63616 kB in the case above). So in reality, applications eat up 307040 kB of real RAM and 63616 kB of swap, total 370656 kB. That’s 35% of the real RAM.
The reason I get into the topic of swap is that one of the answers I got from my VPS service provider’s support was that the problem with swap is that all VPS machines have a common pool of swap on the disk, so getting swap can fail. But that doesn’t explain an OOM kill, as there’s plenty of real RAM to begin with.
If you insist looking at the output of “top”, the place is “cached Mem” + free:
KiB Mem: 1048576 total, 990028 used, 58548 free, 0 buffers KiB Swap: 1048576 total, 62756 used, 985820 free. 715008 cached Mem
but each “top” utility displays this data differently. In this case, putting it on the same line as info on swap memory is misleading.
Or, using “free”:
$ free
total used free shared buffers cached
Mem: 1048576 1000164 48412 35976 0 715888
-/+ buffers/cache: 284276 764300
Swap: 1048576 62716 985860
Once again, it’s cached memory + free. With this utility, they are both on the same line, as they should be.
—————————————————————-
Rambling epilogue: Some thoughts about overcomitting
The details for how overcomitting is accounted for is given on the kernel tree’s Documentation/vm/overcommit-accounting. But to make a long story short, it’s done in a sensible way. In particular, if a piece of memory is shared by threads and processes, it’s only accounted for once.
Relevant files: /proc/meminfo and /proc/vmstat
It seems like CommitLimit and Committed_AS are not available on a VPS guest system. But the OOM killer probably knows these values (or was it because /proc/sys/vm/overcommit_memory was set to 1 on my system, meaning “Always overcommit”?).
To get a list of the current memory hogs, run “top” and press shift-M as it’s running.
To get an idea on how a process behaves, use pmap -x. For example, looking at a mysqld process (run as root, or no memory map will be shown):
# pmap -x 14817 14817: /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 Address Kbytes RSS Dirty Mode Mapping 000055c5617ac000 10476 6204 0 r-x-- mysqld 000055c5623e6000 452 452 452 r---- mysqld 000055c562457000 668 412 284 rw--- mysqld 000055c5624fe000 172 172 172 rw--- [ anon ] 000055c563e9b000 6592 6448 6448 rw--- [ anon ] 00007f819c000000 2296 320 320 rw--- [ anon ] 00007f819c23e000 63240 0 0 ----- [ anon ] 00007f81a0000000 3160 608 608 rw--- [ anon ] 00007f81a0316000 62376 0 0 ----- [ anon ] 00007f81a4000000 9688 7220 7220 rw--- [ anon ] 00007f81a4976000 55848 0 0 ----- [ anon ] 00007f81a8000000 132 8 8 rw--- [ anon ] 00007f81a8021000 65404 0 0 ----- [ anon ] 00007f81ac000000 132 4 4 rw--- [ anon ] 00007f81ac021000 65404 0 0 ----- [ anon ] 00007f81b1220000 4 0 0 ----- [ anon ] 00007f81b1221000 8192 8 8 rw--- [ anon ] 00007f81b1a21000 4 0 0 ----- [ anon ] 00007f81b1a22000 8192 8 8 rw--- [ anon ] 00007f81b2222000 4 0 0 ----- [ anon ] 00007f81b2223000 8192 8 8 rw--- [ anon ] 00007f81b2a23000 4 0 0 ----- [ anon ] 00007f81b2a24000 8192 20 20 rw--- [ anon ] 00007f81b3224000 4 0 0 ----- [ anon ] 00007f81b3225000 8192 8 8 rw--- [ anon ] 00007f81b3a25000 4 0 0 ----- [ anon ] 00007f81b3a26000 8192 8 8 rw--- [ anon ] 00007f81b4226000 4 0 0 ----- [ anon ] 00007f81b4227000 8192 8 8 rw--- [ anon ] 00007f81b4a27000 4 0 0 ----- [ anon ] 00007f81b4a28000 8192 8 8 rw--- [ anon ] 00007f81b5228000 4 0 0 ----- [ anon ] 00007f81b5229000 8192 8 8 rw--- [ anon ] 00007f81b5a29000 4 0 0 ----- [ anon ] 00007f81b5a2a000 8192 8 8 rw--- [ anon ] 00007f81b622a000 4 0 0 ----- [ anon ] 00007f81b622b000 8192 12 12 rw--- [ anon ] 00007f81b6a2b000 4 0 0 ----- [ anon ] 00007f81b6a2c000 8192 8 8 rw--- [ anon ] 00007f81b722c000 4 0 0 ----- [ anon ] 00007f81b722d000 79692 57740 57740 rw--- [ anon ] 00007f81bc000000 132 76 76 rw--- [ anon ] 00007f81bc021000 65404 0 0 ----- [ anon ] 00007f81c002f000 2068 2052 2052 rw--- [ anon ] 00007f81c03f9000 4 0 0 ----- [ anon ] 00007f81c03fa000 192 52 52 rw--- [ anon ] 00007f81c042a000 4 0 0 ----- [ anon ] 00007f81c042b000 192 52 52 rw--- [ anon ] 00007f81c045b000 4 0 0 ----- [ anon ] 00007f81c045c000 192 64 64 rw--- [ anon ] 00007f81c048c000 4 0 0 ----- [ anon ] 00007f81c048d000 736 552 552 rw--- [ anon ] 00007f81c0545000 20 4 0 rw-s- [aio] (deleted) 00007f81c054a000 20 4 0 rw-s- [aio] (deleted) 00007f81c054f000 3364 3364 3364 rw--- [ anon ] 00007f81c0898000 44 12 0 r-x-- libnss_files-2.19.so 00007f81c08a3000 2044 0 0 ----- libnss_files-2.19.so 00007f81c0aa2000 4 4 4 r---- libnss_files-2.19.so 00007f81c0aa3000 4 4 4 rw--- libnss_files-2.19.so 00007f81c0aa4000 40 20 0 r-x-- libnss_nis-2.19.so 00007f81c0aae000 2044 0 0 ----- libnss_nis-2.19.so 00007f81c0cad000 4 4 4 r---- libnss_nis-2.19.so 00007f81c0cae000 4 4 4 rw--- libnss_nis-2.19.so 00007f81c0caf000 28 20 0 r-x-- libnss_compat-2.19.so 00007f81c0cb6000 2044 0 0 ----- libnss_compat-2.19.so 00007f81c0eb5000 4 4 4 r---- libnss_compat-2.19.so 00007f81c0eb6000 4 4 4 rw--- libnss_compat-2.19.so 00007f81c0eb7000 4 0 0 ----- [ anon ] 00007f81c0eb8000 8192 8 8 rw--- [ anon ] 00007f81c16b8000 84 20 0 r-x-- libnsl-2.19.so 00007f81c16cd000 2044 0 0 ----- libnsl-2.19.so 00007f81c18cc000 4 4 4 r---- libnsl-2.19.so 00007f81c18cd000 4 4 4 rw--- libnsl-2.19.so 00007f81c18ce000 8 0 0 rw--- [ anon ] 00007f81c18d0000 1668 656 0 r-x-- libc-2.19.so 00007f81c1a71000 2048 0 0 ----- libc-2.19.so 00007f81c1c71000 16 16 16 r---- libc-2.19.so 00007f81c1c75000 8 8 8 rw--- libc-2.19.so 00007f81c1c77000 16 16 16 rw--- [ anon ] 00007f81c1c7b000 88 44 0 r-x-- libgcc_s.so.1 00007f81c1c91000 2044 0 0 ----- libgcc_s.so.1 00007f81c1e90000 4 4 4 rw--- libgcc_s.so.1 00007f81c1e91000 1024 128 0 r-x-- libm-2.19.so 00007f81c1f91000 2044 0 0 ----- libm-2.19.so 00007f81c2190000 4 4 4 r---- libm-2.19.so 00007f81c2191000 4 4 4 rw--- libm-2.19.so 00007f81c2192000 944 368 0 r-x-- libstdc++.so.6.0.20 00007f81c227e000 2048 0 0 ----- libstdc++.so.6.0.20 00007f81c247e000 32 32 32 r---- libstdc++.so.6.0.20 00007f81c2486000 8 8 8 rw--- libstdc++.so.6.0.20 00007f81c2488000 84 8 8 rw--- [ anon ] 00007f81c249d000 12 8 0 r-x-- libdl-2.19.so 00007f81c24a0000 2044 0 0 ----- libdl-2.19.so 00007f81c269f000 4 4 4 r---- libdl-2.19.so 00007f81c26a0000 4 4 4 rw--- libdl-2.19.so 00007f81c26a1000 32 4 0 r-x-- libcrypt-2.19.so 00007f81c26a9000 2044 0 0 ----- libcrypt-2.19.so 00007f81c28a8000 4 4 4 r---- libcrypt-2.19.so 00007f81c28a9000 4 4 4 rw--- libcrypt-2.19.so 00007f81c28aa000 184 0 0 rw--- [ anon ] 00007f81c28d8000 36 28 0 r-x-- libwrap.so.0.7.6 00007f81c28e1000 2044 0 0 ----- libwrap.so.0.7.6 00007f81c2ae0000 4 4 4 r---- libwrap.so.0.7.6 00007f81c2ae1000 4 4 4 rw--- libwrap.so.0.7.6 00007f81c2ae2000 4 4 4 rw--- [ anon ] 00007f81c2ae3000 104 12 0 r-x-- libz.so.1.2.8 00007f81c2afd000 2044 0 0 ----- libz.so.1.2.8 00007f81c2cfc000 4 4 4 r---- libz.so.1.2.8 00007f81c2cfd000 4 4 4 rw--- libz.so.1.2.8 00007f81c2cfe000 4 4 0 r-x-- libaio.so.1.0.1 00007f81c2cff000 2044 0 0 ----- libaio.so.1.0.1 00007f81c2efe000 4 4 4 r---- libaio.so.1.0.1 00007f81c2eff000 4 4 4 rw--- libaio.so.1.0.1 00007f81c2f00000 96 84 0 r-x-- libpthread-2.19.so 00007f81c2f18000 2044 0 0 ----- libpthread-2.19.so 00007f81c3117000 4 4 4 r---- libpthread-2.19.so 00007f81c3118000 4 4 4 rw--- libpthread-2.19.so 00007f81c3119000 16 4 4 rw--- [ anon ] 00007f81c311d000 132 112 0 r-x-- ld-2.19.so 00007f81c313e000 8 0 0 rw--- [ anon ] 00007f81c3140000 20 4 0 rw-s- [aio] (deleted) 00007f81c3145000 20 4 0 rw-s- [aio] (deleted) 00007f81c314a000 20 4 0 rw-s- [aio] (deleted) 00007f81c314f000 20 4 0 rw-s- [aio] (deleted) 00007f81c3154000 20 4 0 rw-s- [aio] (deleted) 00007f81c3159000 20 4 0 rw-s- [aio] (deleted) 00007f81c315e000 20 4 0 rw-s- [aio] (deleted) 00007f81c3163000 20 4 0 rw-s- [aio] (deleted) 00007f81c3168000 1840 1840 1840 rw--- [ anon ] 00007f81c3334000 8 0 0 rw-s- [aio] (deleted) 00007f81c3336000 4 0 0 rw-s- [aio] (deleted) 00007f81c3337000 24 12 12 rw--- [ anon ] 00007f81c333d000 4 4 4 r---- ld-2.19.so 00007f81c333e000 4 4 4 rw--- ld-2.19.so 00007f81c333f000 4 4 4 rw--- [ anon ] 00007ffd2d68b000 132 68 68 rw--- [ stack ] 00007ffd2d7ad000 8 4 0 r-x-- [ anon ] ffffffffff600000 4 0 0 r-x-- [ anon ] ---------------- ------- ------- ------- total kB 640460 89604 81708
The KBytes and RSS column’s Total at the bottom matches the VIRT and RSS figures shown by “top”.
I should emphasize that this a freshly started mysqld process. Give it a few days to run, and some extra 100 MB of virtual space is added (not clear why) plus some real RAM, depending on the setting.
I’ve marked six anonymous segments that are completely virtual (no resident memory at all) summing up to ~360 MB. This means that they are counted in as 360 MB at least once — and that’s for a process that only uses 90 MB for real.
My own anecdotal test on another machine with a 4.4.0 kernel showed that putting /proc/sys/vm/overcommit_ratio below what was actually committed (making /proc/meminfo’s CommitLimit smaller than Committed_AS) didn’t have any effect unless /proc/sys/vm/overcommit_memory was set to 2. And when I did that, the OOM wasn’t called, but instead I had a hard time running new commands:
# echo 2 > /proc/sys/vm/overcommit_memory # cat /proc/meminfo bash: fork: Cannot allocate memory
So this is what it looks like when memory runs out and the system refuses to play ball.