Docker container high performance file handle

BDZ of tipping excavator 2022-01-15 02:01:39

One 、 Preface

Some time ago, my colleague encountered a problem of file handle leakage in the container , Accidental occurrence of... Inside the container Too many open files Report errors . The file handle configuration at that time is as follows :

 picture 1.png

The question is whether it's coming , Is there a relationship between the file handle of the host and the number of file handles of the container ? What is the relationship ? This is the question to be explored and discussed below .

The process of the experiment is recorded in the middle , It's a little long , Anxious friends can jump directly to chapter four 《 Conclusion analysis 》.

Two 、docker Container start process

docker Is a typical client-server Application of type , have access to docker version Command to view server and client information :

image.png

docker The general process of starting the container is as follows :

DOCKER Container startup process _001.jpg

notes : The yellow text box on the left corresponds to Linux Resident process name under ;

A focus on runc When creating container subprocesses , Will pass in the relevant namespace Parameter for resource isolation ( User isolation is based on Linux Of CLONE_NEWUSER This namespace);

3、 ... and 、 Local experimental process

There happens to be a ready-made war package , Just used it. tomcat Container to test , use Dockerfile take war copy to webapps Let's go .
Dockerfile as follows :

FROM tomcat:8.5.72-jdk8
RUN useradd dockertester -m -d /home/dockertester \
&& gpasswd -a dockertester dockertester \
&& chown dockertester:dockertester -R /usr/local/tomcat
ADD --chown=dockertester pftest.war /usr/local/tomcat/webapps
USER dockertester
CMD ["catalina.sh", "run"]
  • CentOS Linux release 7.4.1708 (Core)
  • docker-19.03.9
  • tomcat:8.5.72-jdk8

( One ) Use Linux The default configuration ,2000 Concurrent test

Environmental preparation 1:

  1. Use Dockerfile Build an application image ;
  2. Use locust【 It saves resources 】 Make concurrent requests , You need to write your own pressure test script ;
  3. ulimit -n View current file handle limit ( The default common users are 1024):
 [[email protected] ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 7216
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 7216
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
  1. cat /proc/sys/fs/file-max View the maximum number of system handles ( Default 181953);
[[email protected] ~]# cat /proc/sys/fs/file-max
181953
  1. Look inside the container java Current file handle limit of the process ( The current container java The maximum number of process file handles is 1048576, Where does this value come from? It will be explained later ):
[email protected]:/usr/local/tomcat$ cat /proc/$(ps -A | grep java | awk '{print $1}')/limits | grep "files"
Max open files 1048576 1048576 files 
  • The test execution 2 minute , There is no mistake .

image.png
image.png

  • View the number of currently used handles in the container , Found that the number of handles used has reached 3488, Far more than the default 1024
[[email protected] ~]# docker exec -it tomcat-test bash
[email protected]:/usr/local/tomcat$ cat /proc/sys/fs/file-nr
3488 0 181953

Test summary

docker The number of file handles in the container is not limited Linux Host user file handle limit . The same is true for launching in container applications using ordinary users , This experiment is a new dockertester user .

( Two ) Modify the container's limit Parameters ,2000 Concurrent test

  1. Environmental preparation 1 There is no need to modify the configuration of ;
  2. Add restriction parameters to the container startup command :--ulimit nofile=1024:1024, After the start of the container , Look inside the container java Limit on the number of process handles ( Has been successfully modified to 1024):
[email protected]:/usr/local/tomcat$ cat /proc/$(ps -A | grep java | awk '{print $1}')/limits | grep "files"
Max open files 1024 1024 files
  • The pressure cannot be measured 1 A large number of errors began to appear in minutes :

image.png
image.png

  • View the current number of system handles and usage , Stay in 2432( beyond 1024 Those are Linux Occupied by other processes of the host ):
[email protected]:/usr/local/tomcat$ cat /proc/sys/fs/file-nr
2432 0 181953
  • see tomcat Log , You can find a lot of Too many open files Report errors :

toomanyopenfile.png

Test summary

docker The number of handles in the container is controlled by the startup parameter --ulimit nofile=1024:1024 The limitation of .

( 3、 ... and ) modify Linux System maximum file handle limit ,2000 Concurrent test

Environmental preparation 2:

  1. Remove the container startup command and add limiting parameters :--ulimit nofile=1024:1024;
  2. Modify the maximum number of file handles in the system :
[[email protected] ~]# cat /proc/sys/fs/file-max
181953
[[email protected] ~]# echo 2000 > /proc/sys/fs/file-max
[[email protected] ~]# cat /proc/sys/fs/file-max
2000
  1. View the current... In the container java Limit on the number of process file handles ( It's still the default 1048576):
[email protected]:/usr/local/tomcat$ cat /proc/$(ps -A | grep java | awk '{print $1}')/limits | grep "files"
Max open files 1048576 1048576 files
  • The pressure measurement cannot be performed 1 minute , Start reporting a lot of errors :

image.png
image.png

  • View the current number of file handles and usage in the container , The number of file handles is full :
bash: start_pipeline: pgrp pipe: Too many open files in system
bash: /bin/cat: Too many open files in system
  • tomcat There are a lot of... In the log Too many open files Report errors :

image.png

Test summary

docker The number of handles in the container is limited by the number of file handles on the host system (/proc/sys/fs/file-max or /etc/sysctl.conf Parameters for file configuration )

Four 、 Conclusion analysis

  • notes : This document does not apply to Rootless Installed by docker application , Installed in this way docker Haven't studied .

( One )namespace Process isolation 【 Incomplete isolation 】

  1. docker The startup container is based on namespace Do process isolation , And have CAP_SYS_RESOURCE The ability of , Parameters like file handles docker You can reset , However, after reset, it is still limited by the host system parameters , namely /proc/sys/fs/file-max or /etc/sysctl.conf Parameters for file configuration ;
     
  2. Default docker The startup user of the internal process is root, Inside the container root The permission is slightly lower than that of the host . Under normal circumstances , Similar cannot be modified in the container /proc/sys/fs/file-max These system configurations , A read-only error will appear bash: /proc/sys/fs/file-max: Read-only file system. But if docker run Add --privileged Parameters , You can modify these kernel parameters , Moreover, the configuration of the host will be modified synchronously , This is very dangerous ️ What happened ;
     
  3. docker Share kernel with host , therefore cat /proc/sys/fs/file-max Such system level commands are executed both inside and outside the container . also top Command to see CPU Auditing is not docker Number of cores limited by startup parameters , Or the number of cores of the host .

( Two ) Support namespace Kernel configuration item

See official documentation :Configure namespaced kernel parameters (sysctls) at runtime

IPC( Used to isolate resources needed for interprocess communication )
  • kernel.msgmax, kernel.msgmnb, kernel.msgmni, kernel.sem, kernel.shmall, kernel.shmmax, kernel.shmmni, kernel.shm_rmid_forced;
  • fs.mqueue Parameters at the beginning ,fs.mqueue.*;
Network relevant :
  • similar net.* Configuration parameters for ;
  • If used --network=host Options , Then... Is not allowed docker run --sysctl net.ipv4.ip_forward=1 someimage Mode start ;

( 3、 ... and ) The source of the number of default file handles for processes in the container

Contact the container startup process in Chapter 2 , Process of container yes runc Can be inherited by child processes. ( After creation ),runc yes containerd Can be inherited by child processes. . stay fork Sub process time , The child process inherits all the information of the parent process , Therefore, the default number of file handles of processes in the container is inherited containerd The number of handles to the process is limited . You can order to see , In fact, it was the default 1048576;

cat /proc/$(ps -A | grep containerd-PID | awk '{print $1}')/limits | grep "files"

and containerd The number of handles to the process is limited to containerd.service The default setting in the file is 1048576

  • The new version is not in docker.service Set in
  • from Docker 1.11 Start ,Docker Daemon No more runtime code , This part of the logic is migrated separately OCI Compatibility layer , and containerd It's just daemon and runc Where OCI Between layers , Responsible for container creation , And the destruction is left to shim.

image.png

( Four ) An interesting user namespace mapping

stay Dockerfile When building an application image , If the designation is not root Start the container , that docker It is mapped by default uid by 1000 Users of , If the local host is not set uid by 1000 Users of , After the container starts ,ps -ef You will see that the user column directly displays 1000.


thank
Similar articles

2022-01-15

2022-01-15