In OpenStack Kolla, they use docker for their services. Recently, I was alerted that I was running out of disk space and cleared out some unused kernels, but also realized, that I had 12Gb of docker logs as shown below. So I wanted to find an elegant solution to free up space for the long term.
# du -ch /var/lib/docker/containers/*/*-json.log 8.0K /var/lib/docker/containers/0244f6373b66bb31dc959e14ec7bb00aeb59d1b49341490acc6bb4f09a5d402a/0244f6373b66bb31dc959e14ec7bb00aeb59d1b49341490acc6bb4f09a5d402a-json.log 80K /var/lib/docker/containers/03298b99be331d066481badb01d0d21398525661edf7ab7b06218ba5abc5b25f/03298b99be331d066481badb01d0d21398525661edf7ab7b06218ba5abc5b25f-json.log 70M /var/lib/docker/containers/07ca03e21b0771149a8367fc9fcce02fd9049fd5f82c8e3f1c3414283868b84d/07ca03e21b0771149a8367fc9fcce02fd9049fd5f82c8e3f1c3414283868b84d-json.log 1.1M /var/lib/docker/containers/263dfc25ff10994ea39f8c7759da032dda828a7fda70e70333fe95bb84b4737a/263dfc25ff10994ea39f8c7759da032dda828a7fda70e70333fe95bb84b4737a-json.log 1.5G /var/lib/docker/containers/28ef28add064fd0c86a115b2a1e39e29c22aaff55dd3fa686b5c5737e1dcfd1f/28ef28add064fd0c86a115b2a1e39e29c22aaff55dd3fa686b5c5737e1dcfd1f-json.log 5.7M /var/lib/docker/containers/2a63fa0aeacc83272012b8a6a4faea3a61c725b0f343d7478dc5db5a88090668/2a63fa0aeacc83272012b8a6a4faea3a61c725b0f343d7478dc5db5a88090668-json.log 28K /var/lib/docker/containers/37288a8131e2c12498e90421dd84e24774138bc1dc3c0940589cb2def21ec507/37288a8131e2c12498e90421dd84e24774138bc1dc3c0940589cb2def21ec507-json.log 1.2M /var/lib/docker/containers/3a9e2f81083e882b1aebbbd9776ee270aafea8ff3a4fff1a546d1dbf09d84058/3a9e2f81083e882b1aebbbd9776ee270aafea8ff3a4fff1a546d1dbf09d84058-json.log 23M /var/lib/docker/containers/3e3fbc9ac409567e9a393d5b634c05168515a0fd05cdcd59dce5c29f840d9365/3e3fbc9ac409567e9a393d5b634c05168515a0fd05cdcd59dce5c29f840d9365-json.log 5.2G /var/lib/docker/containers/40f7a732c894e11c820a657621127e1a6aa80ec78d3c2d24e912e808ac9f7246/40f7a732c894e11c820a657621127e1a6aa80ec78d3c2d24e912e808ac9f7246-json.log 32K /var/lib/docker/containers/4974c9676191d92883944096d510923f8436964d4a49b611285ebc123fd32415/4974c9676191d92883944096d510923f8436964d4a49b611285ebc123fd32415-json.log 240K /var/lib/docker/containers/4c89ee12ac2df14577a86cc1c7f2e520836b7cc993d2b04a9693e3fc1f20540c/4c89ee12ac2df14577a86cc1c7f2e520836b7cc993d2b04a9693e3fc1f20540c-json.log 8.0M /var/lib/docker/containers/5032e981a2a20b44b0d0bc7d40f1b86bd41e57eab8e1690fce3b5833b684c254/5032e981a2a20b44b0d0bc7d40f1b86bd41e57eab8e1690fce3b5833b684c254-json.log 28K /var/lib/docker/containers/50afdd850d3f930fd31e8d6a0a993439ee2a6f7961c3e4390dea86671df20833/50afdd850d3f930fd31e8d6a0a993439ee2a6f7961c3e4390dea86671df20833-json.log 76K /var/lib/docker/containers/5404b2699e5732771d2bc75b138c2a46424678221004e35dd1e26505e2739d57/5404b2699e5732771d2bc75b138c2a46424678221004e35dd1e26505e2739d57-json.log 12M /var/lib/docker/containers/57086d7387b02a320cc89aa67e267e7c5a0f74be281ead6967e59a8b965ba52f/57086d7387b02a320cc89aa67e267e7c5a0f74be281ead6967e59a8b965ba52f-json.log 16K /var/lib/docker/containers/74961d08f8c33915d88c92014fcbcb433a4dc64a29bd582cbd9d90952613b46d/74961d08f8c33915d88c92014fcbcb433a4dc64a29bd582cbd9d90952613b46d-json.log 8.0K /var/lib/docker/containers/74c6befb80aac9631959716b4a31fd5ab12177f38a22753948532b5345b82801/74c6befb80aac9631959716b4a31fd5ab12177f38a22753948532b5345b82801-json.log 653M /var/lib/docker/containers/7780889aafd4cf17ceed613088684b0d763040f79c3e8a7aa09bbcb16bca456b/7780889aafd4cf17ceed613088684b0d763040f79c3e8a7aa09bbcb16bca456b-json.log 400K /var/lib/docker/containers/7a6429feea40206c9749714311759b728e7bb8a93783ad2b4f551e2d0880541c/7a6429feea40206c9749714311759b728e7bb8a93783ad2b4f551e2d0880541c-json.log 1.5G /var/lib/docker/containers/7f99dcab576b332336725b291c9b313b09515e31fb0b6501b42c8dd8f82d235e/7f99dcab576b332336725b291c9b313b09515e31fb0b6501b42c8dd8f82d235e-json.log 17M /var/lib/docker/containers/858ed8872a4d0006f5c88ee3de02670c80ff59e7db22506dd0617e8b71f2c42d/858ed8872a4d0006f5c88ee3de02670c80ff59e7db22506dd0617e8b71f2c42d-json.log 1.1M /var/lib/docker/containers/87133b4de74f382bc5f66774f0734e7b333d010c021e8f6b75d81fc847d50e2d/87133b4de74f382bc5f66774f0734e7b333d010c021e8f6b75d81fc847d50e2d-json.log 24K /var/lib/docker/containers/974f49af9963cf24a6a488e5c0a7f969808355cbfc25b42ac41531db2023202a/974f49af9963cf24a6a488e5c0a7f969808355cbfc25b42ac41531db2023202a-json.log 16K /var/lib/docker/containers/99eb06c3bc25697bab223375e0065709dca21133448642574b69a4773f8f9c5d/99eb06c3bc25697bab223375e0065709dca21133448642574b69a4773f8f9c5d-json.log 12K /var/lib/docker/containers/9a02926e775da34f0098babc84943342e2aaaedaa56caa3aeb5233e71b797a38/9a02926e775da34f0098babc84943342e2aaaedaa56caa3aeb5233e71b797a38-json.log 236K /var/lib/docker/containers/9e2303a0dd36170b53a146e4326ef070216533085b67143e07e6fc82e509f954/9e2303a0dd36170b53a146e4326ef070216533085b67143e07e6fc82e509f954-json.log 56K /var/lib/docker/containers/a144afb0b5530c2ebbb9a4b669c3e160e9b1adeef12fdd154689612ca5d3cd05/a144afb0b5530c2ebbb9a4b669c3e160e9b1adeef12fdd154689612ca5d3cd05-json.log 664M /var/lib/docker/containers/abce7b945a9fb94bc148ba2f53255f3300f97996fbef7373aee8b5fa4394f633/abce7b945a9fb94bc148ba2f53255f3300f97996fbef7373aee8b5fa4394f633-json.log 32K /var/lib/docker/containers/c0a67ef24ea31d5bd98896bb2ea2bd13683ac03a7705803aeefec993f4df6278/c0a67ef24ea31d5bd98896bb2ea2bd13683ac03a7705803aeefec993f4df6278-json.log 1.6G /var/lib/docker/containers/c0a711c5b59baf775e47d230f8713308ad1c8fd3421c4d18b2bc8bff1f67f36e/c0a711c5b59baf775e47d230f8713308ad1c8fd3421c4d18b2bc8bff1f67f36e-json.log 376K /var/lib/docker/containers/c0b54e1439d9094c6c1caad7cdaa4a777e26107eb53ac11a13697e46c2090757/c0b54e1439d9094c6c1caad7cdaa4a777e26107eb53ac11a13697e46c2090757-json.log 23M /var/lib/docker/containers/caacff44c1a32c44b5ae662a011f8755374df085bc285f24a268fb6b0f33742e/caacff44c1a32c44b5ae662a011f8755374df085bc285f24a268fb6b0f33742e-json.log 36K /var/lib/docker/containers/cd683ed0a3086d2736d46c2248fcf64f8c25a8153bd3024f5bc5caaaa3b2966c/cd683ed0a3086d2736d46c2248fcf64f8c25a8153bd3024f5bc5caaaa3b2966c-json.log 56K /var/lib/docker/containers/d9c03c2dd2f70f3cc345f2f8e35b2c215277b93148f13b72f9a970293d9afa63/d9c03c2dd2f70f3cc345f2f8e35b2c215277b93148f13b72f9a970293d9afa63-json.log 66M /var/lib/docker/containers/ddc02fe22c41b75240cf71bf07079834c1deea95063c935ba8eb287e67af791e/ddc02fe22c41b75240cf71bf07079834c1deea95063c935ba8eb287e67af791e-json.log 8.0K /var/lib/docker/containers/ddc9a874ee47275fe7d139d2ccbf173f2e16d091edd515278abb54398bc50912/ddc9a874ee47275fe7d139d2ccbf173f2e16d091edd515278abb54398bc50912-json.log 21M /var/lib/docker/containers/df13c5b52ec59a0bcce696c24236ac549bdde863a29d661b3ac398650447a64f/df13c5b52ec59a0bcce696c24236ac549bdde863a29d661b3ac398650447a64f-json.log 24K /var/lib/docker/containers/e04e35323cd7f64e9a6dccb5c420f72e7800963fbb233b0790314209072ede2b/e04e35323cd7f64e9a6dccb5c420f72e7800963fbb233b0790314209072ede2b-json.log 53M /var/lib/docker/containers/e396fcc6e624dd0fafcc35823b9e1133031bf64d1fd1bc7409916dc405fa4b03/e396fcc6e624dd0fafcc35823b9e1133031bf64d1fd1bc7409916dc405fa4b03-json.log 548K /var/lib/docker/containers/f58b6542f00de95aa1395663e4fd813094da2d8f23f220109c2d263c6ab21933/f58b6542f00de95aa1395663e4fd813094da2d8f23f220109c2d263c6ab21933-json.log 12K /var/lib/docker/containers/f9a66b5458de3b4620eeb80ef5651c2d8ba5826edfdc6a07365711453f53da11/f9a66b5458de3b4620eeb80ef5651c2d8ba5826edfdc6a07365711453f53da11-json.log 4.0K /var/lib/docker/containers/fde7cadb362d7645a436290f97866a584a375eebac831d14690aded45a51805c/fde7cadb362d7645a436290f97866a584a375eebac831d14690aded45a51805c-json.log 12G total
My solution was to create a file here: /etc/logrotate.d/docker-logs with the following content.
/var/lib/docker/containers/*/*.log { rotate 7 daily compress size=50M missingok delaycompress copytruncate }
Once written, I force ran my logrotate:
logrotate --force /etc/logrotate.d/docker-logs It took a couple of minutes and the size was down to 4.0K and hopefully wont get to the GB range again. Resources: https://blog.birkhoff.me/devops-truncate-docker-container-logs-periodically-to-free-up-server-disk-space/ Note: I found much of this through various blogs and common knowledge, but I came across this blog and he had a similar issue and ended up being very similar. So the meat of the content is the same, but this is written by me. |
AuthorJames Benson is an IT professional. Archives
August 2022
Categories
All
|