Docker uses a lot more disk-space than docker system df reports

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          7         2         5.129GB   4.806GB (93%)
Containers      3         3         12.46MB   0B (0%)
Local Volumes   4         3         7.627GB   230MB (3%)
Build Cache     165       0         2.378GB   2.378GB

Using the -v flag like this : docker system df -v shows the same information in way more details that can help you understand what part of Docker uses all that data.

Except in my case, the folder where Docker stored its data used about 20 times more space than reported by Docker. Naturally I only noticed it when my server ran out of disk-space putting me in panic-mode.

It took me a while till I realized there are some huge log-files in the containers folder.

I am sure you know that you can look at the output that was printed to the console of your container by running the command:

docker container logs NAME_OF_A_CONTAINER

As it turns out this output is saved in a file in the folder of the individual container and by default it is kept forever.

docker system df can provide detailed reports on disk usage, but it does not include the disk space used by the log files.

One of the services I used was very, I mean really talkative and printed tons of stuff on the console. Docker shoved all of that into an ever growing file.

## Solution

The immediate solution was to stop docker and remove the huge file.

The longer-term solution was to configure Docker to keep the log files limited in size.

It can be done in the daemon configuration file which, at least in my installation is located in

/etc/docker/daemon.json

I already had this in the file:

{
    "data-root": "/home/data/docker",
}

That's because a while ago I moved all the data to a disk separate from the disk of the operating system.

Now I updated it:

{
    "data-root": "/home/data/docker",

    "log-driver": "json-file",
    "log-opts": {
        "max-size": "10m",
        "max-file": "3"
    }
}

Then I restarted the Docker daemon with

sudo systemctl restart docker

and I also had to stop and remove the existing containers and start new ones!

Without that the changed did not take effect.

To tell the truth I also rebooted the host machine. It is probably not necessary, but I like to do that after such changes to verify that the computer can be booted and all the services start as expected. I rather have a few seconds downtime when I still remember the changes I made than later when something breaks because of some changes a few weeks or month earlier.

For more details see how to [configure logging drivers for Docker](https://docs.docker.com/config/containers/logging/configure/).