In OpenStack Kolla, they use docker for their services. Recently, I was alerted that I was running out of disk space and cleared out some unused kernels, but also realized, that I had 12Gb of docker logs as shown below. So I wanted to find an elegant solution to free up space for the long term.
# du -ch /var/lib/docker/containers/*/*-json.log 8.0K /var/lib/docker/containers/0244f6373b66bb31dc959e14ec7bb00aeb59d1b49341490acc6bb4f09a5d402a/0244f6373b66bb31dc959e14ec7bb00aeb59d1b49341490acc6bb4f09a5d402a-json.log 80K /var/lib/docker/containers/03298b99be331d066481badb01d0d21398525661edf7ab7b06218ba5abc5b25f/03298b99be331d066481badb01d0d21398525661edf7ab7b06218ba5abc5b25f-json.log 70M /var/lib/docker/containers/07ca03e21b0771149a8367fc9fcce02fd9049fd5f82c8e3f1c3414283868b84d/07ca03e21b0771149a8367fc9fcce02fd9049fd5f82c8e3f1c3414283868b84d-json.log 1.1M /var/lib/docker/containers/263dfc25ff10994ea39f8c7759da032dda828a7fda70e70333fe95bb84b4737a/263dfc25ff10994ea39f8c7759da032dda828a7fda70e70333fe95bb84b4737a-json.log 1.5G /var/lib/docker/containers/28ef28add064fd0c86a115b2a1e39e29c22aaff55dd3fa686b5c5737e1dcfd1f/28ef28add064fd0c86a115b2a1e39e29c22aaff55dd3fa686b5c5737e1dcfd1f-json.log 5.7M /var/lib/docker/containers/2a63fa0aeacc83272012b8a6a4faea3a61c725b0f343d7478dc5db5a88090668/2a63fa0aeacc83272012b8a6a4faea3a61c725b0f343d7478dc5db5a88090668-json.log 28K /var/lib/docker/containers/37288a8131e2c12498e90421dd84e24774138bc1dc3c0940589cb2def21ec507/37288a8131e2c12498e90421dd84e24774138bc1dc3c0940589cb2def21ec507-json.log 1.2M /var/lib/docker/containers/3a9e2f81083e882b1aebbbd9776ee270aafea8ff3a4fff1a546d1dbf09d84058/3a9e2f81083e882b1aebbbd9776ee270aafea8ff3a4fff1a546d1dbf09d84058-json.log 23M /var/lib/docker/containers/3e3fbc9ac409567e9a393d5b634c05168515a0fd05cdcd59dce5c29f840d9365/3e3fbc9ac409567e9a393d5b634c05168515a0fd05cdcd59dce5c29f840d9365-json.log 5.2G /var/lib/docker/containers/40f7a732c894e11c820a657621127e1a6aa80ec78d3c2d24e912e808ac9f7246/40f7a732c894e11c820a657621127e1a6aa80ec78d3c2d24e912e808ac9f7246-json.log 32K /var/lib/docker/containers/4974c9676191d92883944096d510923f8436964d4a49b611285ebc123fd32415/4974c9676191d92883944096d510923f8436964d4a49b611285ebc123fd32415-json.log 240K /var/lib/docker/containers/4c89ee12ac2df14577a86cc1c7f2e520836b7cc993d2b04a9693e3fc1f20540c/4c89ee12ac2df14577a86cc1c7f2e520836b7cc993d2b04a9693e3fc1f20540c-json.log 8.0M /var/lib/docker/containers/5032e981a2a20b44b0d0bc7d40f1b86bd41e57eab8e1690fce3b5833b684c254/5032e981a2a20b44b0d0bc7d40f1b86bd41e57eab8e1690fce3b5833b684c254-json.log 28K /var/lib/docker/containers/50afdd850d3f930fd31e8d6a0a993439ee2a6f7961c3e4390dea86671df20833/50afdd850d3f930fd31e8d6a0a993439ee2a6f7961c3e4390dea86671df20833-json.log 76K /var/lib/docker/containers/5404b2699e5732771d2bc75b138c2a46424678221004e35dd1e26505e2739d57/5404b2699e5732771d2bc75b138c2a46424678221004e35dd1e26505e2739d57-json.log 12M /var/lib/docker/containers/57086d7387b02a320cc89aa67e267e7c5a0f74be281ead6967e59a8b965ba52f/57086d7387b02a320cc89aa67e267e7c5a0f74be281ead6967e59a8b965ba52f-json.log 16K /var/lib/docker/containers/74961d08f8c33915d88c92014fcbcb433a4dc64a29bd582cbd9d90952613b46d/74961d08f8c33915d88c92014fcbcb433a4dc64a29bd582cbd9d90952613b46d-json.log 8.0K /var/lib/docker/containers/74c6befb80aac9631959716b4a31fd5ab12177f38a22753948532b5345b82801/74c6befb80aac9631959716b4a31fd5ab12177f38a22753948532b5345b82801-json.log 653M /var/lib/docker/containers/7780889aafd4cf17ceed613088684b0d763040f79c3e8a7aa09bbcb16bca456b/7780889aafd4cf17ceed613088684b0d763040f79c3e8a7aa09bbcb16bca456b-json.log 400K /var/lib/docker/containers/7a6429feea40206c9749714311759b728e7bb8a93783ad2b4f551e2d0880541c/7a6429feea40206c9749714311759b728e7bb8a93783ad2b4f551e2d0880541c-json.log 1.5G /var/lib/docker/containers/7f99dcab576b332336725b291c9b313b09515e31fb0b6501b42c8dd8f82d235e/7f99dcab576b332336725b291c9b313b09515e31fb0b6501b42c8dd8f82d235e-json.log 17M /var/lib/docker/containers/858ed8872a4d0006f5c88ee3de02670c80ff59e7db22506dd0617e8b71f2c42d/858ed8872a4d0006f5c88ee3de02670c80ff59e7db22506dd0617e8b71f2c42d-json.log 1.1M /var/lib/docker/containers/87133b4de74f382bc5f66774f0734e7b333d010c021e8f6b75d81fc847d50e2d/87133b4de74f382bc5f66774f0734e7b333d010c021e8f6b75d81fc847d50e2d-json.log 24K /var/lib/docker/containers/974f49af9963cf24a6a488e5c0a7f969808355cbfc25b42ac41531db2023202a/974f49af9963cf24a6a488e5c0a7f969808355cbfc25b42ac41531db2023202a-json.log 16K /var/lib/docker/containers/99eb06c3bc25697bab223375e0065709dca21133448642574b69a4773f8f9c5d/99eb06c3bc25697bab223375e0065709dca21133448642574b69a4773f8f9c5d-json.log 12K /var/lib/docker/containers/9a02926e775da34f0098babc84943342e2aaaedaa56caa3aeb5233e71b797a38/9a02926e775da34f0098babc84943342e2aaaedaa56caa3aeb5233e71b797a38-json.log 236K /var/lib/docker/containers/9e2303a0dd36170b53a146e4326ef070216533085b67143e07e6fc82e509f954/9e2303a0dd36170b53a146e4326ef070216533085b67143e07e6fc82e509f954-json.log 56K /var/lib/docker/containers/a144afb0b5530c2ebbb9a4b669c3e160e9b1adeef12fdd154689612ca5d3cd05/a144afb0b5530c2ebbb9a4b669c3e160e9b1adeef12fdd154689612ca5d3cd05-json.log 664M /var/lib/docker/containers/abce7b945a9fb94bc148ba2f53255f3300f97996fbef7373aee8b5fa4394f633/abce7b945a9fb94bc148ba2f53255f3300f97996fbef7373aee8b5fa4394f633-json.log 32K /var/lib/docker/containers/c0a67ef24ea31d5bd98896bb2ea2bd13683ac03a7705803aeefec993f4df6278/c0a67ef24ea31d5bd98896bb2ea2bd13683ac03a7705803aeefec993f4df6278-json.log 1.6G /var/lib/docker/containers/c0a711c5b59baf775e47d230f8713308ad1c8fd3421c4d18b2bc8bff1f67f36e/c0a711c5b59baf775e47d230f8713308ad1c8fd3421c4d18b2bc8bff1f67f36e-json.log 376K /var/lib/docker/containers/c0b54e1439d9094c6c1caad7cdaa4a777e26107eb53ac11a13697e46c2090757/c0b54e1439d9094c6c1caad7cdaa4a777e26107eb53ac11a13697e46c2090757-json.log 23M /var/lib/docker/containers/caacff44c1a32c44b5ae662a011f8755374df085bc285f24a268fb6b0f33742e/caacff44c1a32c44b5ae662a011f8755374df085bc285f24a268fb6b0f33742e-json.log 36K /var/lib/docker/containers/cd683ed0a3086d2736d46c2248fcf64f8c25a8153bd3024f5bc5caaaa3b2966c/cd683ed0a3086d2736d46c2248fcf64f8c25a8153bd3024f5bc5caaaa3b2966c-json.log 56K /var/lib/docker/containers/d9c03c2dd2f70f3cc345f2f8e35b2c215277b93148f13b72f9a970293d9afa63/d9c03c2dd2f70f3cc345f2f8e35b2c215277b93148f13b72f9a970293d9afa63-json.log 66M /var/lib/docker/containers/ddc02fe22c41b75240cf71bf07079834c1deea95063c935ba8eb287e67af791e/ddc02fe22c41b75240cf71bf07079834c1deea95063c935ba8eb287e67af791e-json.log 8.0K /var/lib/docker/containers/ddc9a874ee47275fe7d139d2ccbf173f2e16d091edd515278abb54398bc50912/ddc9a874ee47275fe7d139d2ccbf173f2e16d091edd515278abb54398bc50912-json.log 21M /var/lib/docker/containers/df13c5b52ec59a0bcce696c24236ac549bdde863a29d661b3ac398650447a64f/df13c5b52ec59a0bcce696c24236ac549bdde863a29d661b3ac398650447a64f-json.log 24K /var/lib/docker/containers/e04e35323cd7f64e9a6dccb5c420f72e7800963fbb233b0790314209072ede2b/e04e35323cd7f64e9a6dccb5c420f72e7800963fbb233b0790314209072ede2b-json.log 53M /var/lib/docker/containers/e396fcc6e624dd0fafcc35823b9e1133031bf64d1fd1bc7409916dc405fa4b03/e396fcc6e624dd0fafcc35823b9e1133031bf64d1fd1bc7409916dc405fa4b03-json.log 548K /var/lib/docker/containers/f58b6542f00de95aa1395663e4fd813094da2d8f23f220109c2d263c6ab21933/f58b6542f00de95aa1395663e4fd813094da2d8f23f220109c2d263c6ab21933-json.log 12K /var/lib/docker/containers/f9a66b5458de3b4620eeb80ef5651c2d8ba5826edfdc6a07365711453f53da11/f9a66b5458de3b4620eeb80ef5651c2d8ba5826edfdc6a07365711453f53da11-json.log 4.0K /var/lib/docker/containers/fde7cadb362d7645a436290f97866a584a375eebac831d14690aded45a51805c/fde7cadb362d7645a436290f97866a584a375eebac831d14690aded45a51805c-json.log 12G total
My solution was to create a file here: /etc/logrotate.d/docker-logs with the following content.
/var/lib/docker/containers/*/*.log { rotate 7 daily compress size=50M missingok delaycompress copytruncate }
Once written, I force ran my logrotate:
logrotate --force /etc/logrotate.d/docker-logs It took a couple of minutes and the size was down to 4.0K and hopefully wont get to the GB range again. Resources: https://blog.birkhoff.me/devops-truncate-docker-container-logs-periodically-to-free-up-server-disk-space/ Note: I found much of this through various blogs and common knowledge, but I came across this blog and he had a similar issue and ended up being very similar. So the meat of the content is the same, but this is written by me. Your comment will be posted after it is approved.
Leave a Reply. |
AuthorJames Benson is an IT professional. Archives
August 2022
Categories
All
|