Centos7 Jenkins 日志记录爆满磁盘问题

问题描述：上周准备线上上Jenkens,然后配置好之后……

今天还在睡懒觉，一通电话惊醒，服务器有部分服务不能用了…

1、分析问题

服务一直很稳定，再测试一把出现问题的服务。发现问题，连运行日志都没有了，默默的猜服务器是不是文件哪里挂了，直接上杀手：

第一步：清除nginx日志

第二步：删除服务日志

第三步：重启服务

结果：日志照样没有

2、只能是服务器的磁盘爆满了

第一步：df -sh

果然是磁盘爆了。

第二步：找服务器磁盘哪里被爆了

首先看哪个文件夹爆了-这个是我清理后的结果

[root@iZ23evimvf8Z ~]# df -lh
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       40G   39G   0     100% /
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.7G   44K  7.7G   1% /dev/shm
tmpfs           7.7G  384K  7.7G   1% /run
tmpfs           7.7G     0  7.7G   0% /sys/fs/cgroup
tmpfs           1.6G     0  1.6G   0% /run/user/1000
tmpfs           1.6G     0  1.6G   0% /run/user/0

[root@iZ23evimvf8Z ~]# du -sh /*
4.0K	/1
0	/bin
103M	/boot
624K	/C:
24K	/deployer
44K	/dev
143M	/etc
52K	/file:
4.0G	/ftpDir
19M	/home
0	/lib
0	/lib64
16K	/lost+found
4.0K	/media
4.0K	/mnt
116M	/opt
...........................

[root@iZ23evimvf8Z ~]# du -sh /var/log/*
36K	/var/log/agent.log
16K	/var/log/aliyun-util.log
4.0K	/var/log/anaconda
4.0K	/var/log/audit
8.0K	/var/log/boot.log
4.0K	/var/log/btmp
56K	/var/log/btmp-20190801
16K	/var/log/cron
100K	/var/log/cron-20190729
76K	/var/log/cron-20190804
88K	/var/log/cron-20190811
84K	/var/log/cron-20190818
36K	/var/log/dmesg
36K	/var/log/dmesg.old
0	/var/log/grubby
24K	/var/log/gshell.log
1.9G	/var/log/journal

找到最终的文件，然后干掉：

[root@iZ23evimvf8Z log]# rm -rf /var/log/jenkins

然后看看我们的服务器：

[root@iZ23evimvf8Z log]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       40G   39G     0 100% /
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.7G   44K  7.7G   1% /dev/shm
tmpfs           7.7G  356K  7.7G   1% /run
tmpfs           7.7G     0  7.7G   0% /sys/fs/cgroup
tmpfs           1.6G     0  1.6G   0% /run/user/1000
tmpfs           1.6G     0  1.6G   0% /run/user/0

原因：我们彻底删除了是文件的磁盘，但是实际Jenkins还在运行中，还有占用文件：于是我们这么做：

[root@iZ23evimvf8Z log]# ps aux | grep jenkins
root      2468  6.6 10.2 7796180 1634752 ?     Ssl  10:50   0:47 /usr/local/www/zcsjw/jdk1.8.0_172/bin/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -DJENKINS_HOME=/var/libjenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=9000 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
root     10184  0.0  0.0 112652   956 pts/2    S+   11:02   0:00 grep --color=auto jenkins
[root@iZ23evimvf8Z log]# kill -9 2468

彻底干掉Jenkins就好了：

[root@iZ23evimvf8Z log]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       40G   23G   15G  61% /
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.7G   44K  7.7G   1% /dev/shm
tmpfs           7.7G  356K  7.7G   1% /run
tmpfs           7.7G     0  7.7G   0% /sys/fs/cgroup
tmpfs           1.6G     0  1.6G   0% /run/user/1000
tmpfs           1.6G     0  1.6G   0% /run/user/0

大功告成。

其他更方便简单的处理方式

第一步：查找系统大文件

[root@iZ23evimvf8Z log]# find / -type f -size +100M -print0 | xargs -0 du -h | sort -nr

[root@iZ23evimvf8Z log]# find / -type f -size +100M  -print0 | xargs -0 du -h | sort -nr
find: ‘/proc/24277/task/24277/fdinfo/6’: No such file or directory
find: ‘/proc/24277/fdinfo/6’: No such file or directory
131M	/ftpDir/sl/tjsljs.flv
129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-00000000004e09fc-00058f83c0d11719.journal
129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-00000000004b5d72-00058e8ce84241fd.journal
129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-000000000048bfe4-00058db36555cbcc.journal
129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-0000000000462262-00058ca95dffa790.journal
129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-000000000042e589-00058b7a553abdb1.journal
129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-0000000000402986-000589de23e00a9d.journal
129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-00000000003b49e7-00057a645c72469f.journal
129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-000000000038cb1a-00057a5bac12ab59.journal
129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-0000000000364c51-00057a52fbda482e.journal
129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-000000000033cd8c-00057a4a4b4daa56.journal
129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-0000000000314ec9-00057a419adf7984.journal
129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-00000000002ecfee-00057a38ea9ac8e7.journal
102M	/usr/lib/locale/locale-archive
1.4G	/usr/local/csjw/elasticsearch-6.3.5-node2/bin/java_pid1453.hprof
1.4G	/usr/local/csjw/elasticsearch-6.3.5-node2/bin/java_pid1215.hprof
1.4G	/usr/local/csjw/elasticsearch-6.3.5-node1/bin/java_pid1857.hprof
1.4G	/usr/local/csjw/elasticsearch-6.3.5-node1/bin/java_pid1388.hprof
1.4G	/usr/local/csjw/elasticsearch-6.3.5-node1/bin/java_pid1223.hprof
1.4G	/usr/local/csjw/elasticsearch-6.3.5-node1/bin/java_pid1188.hprof

第二步：删除查找到的大文件，然后干掉。部分步骤如上面处理。