A Docker
Logging Guide

Managing stats, events and Docker logs

In my daily job, I get the chance to meet and exchange with various passionate tech teams from all sorts of companies building various applications. Among these teams and companies, a trend is emerging: the rise of microservices.

docker microservices

Splitting up an application into smaller and more maintainable parts has plenty of advantages in the modern development world as issues become isolated. When teams build code bases easier to manage, adding new features and deploying is facilitated, and so is the scaling of independant services. And since your application is not dependant on a machine anymore, it is even better than the common VMs architecture.

Everybody seems to agree that Docker is the best open source project to build microservices – I would even say that Docker seems to be the biggest influencer in encouraging people to adopt/move to microservices.

So Docker has become very trendy – that’s a fact – and they’ve already become quite common in production. However, when containers with short life spans are used in large numbers, collecting and analyzing monitoring data requires significant investments. Many apps using containers are not monitoring themselves – which makes them vulnerable to downtime or functional bugs.

Docker logging & monitoring is challenging

Docker does not provide a clear answer to this problem, nor do they advertise for a specific solution either. Lots of articles have popped up on the topic over the last 2 years covering the different options available to collect data. But, they usually have a focus on monitoring.

Focusing solely on Docker monitoring leaves a whole set of very interesting data aside. Since Docker and app performance depends on the tech team’s ability to get visibility over metrics & events as well as logs, this article will explore:

  • Why all 3 Docker data sets – stats, events & logs – are important to look into to promote proper Docker performance
  • How to more specifically extract Docker logs easily
  • I. Why monitor Docker Stats

    Docker stats were released for the first time with Docker 1.5. It provides a whole set of monitoring constants.

    docker stats

    The good news is that the Docker provided API streams real nice JSON objects that look like logs. You can now set alerts for critical metrics going out of allocated boundaries.

    But how important is it to look at these Docker constants really? We find the insights that can be extracted from them are not often clear. Take CPU for example. What this metric is really showing you is how much CPU the container consumes over the Docker machine it is deployed into. If the container consumes 100% of the CPU, it means other containers may suffer from the lack of CPU. That’s why CPU throttling – also available in the stats – is an important metric too. When deploying a container you can define CPU shares constraints. You can thus decide that if 2 containers run side-by-side, each of them cannot exceed 50% of the available CPU. When this threshold is met, Docker will report it in the throttling metric (count or time). If your system encounters too much of it, it means that one of your applications cannot perform at full capacity, possibly generating bad user experience.

    This story aside here are the metrics you should really focus on:

    NameAttribute in JSON docDescription
    user CPUcpu_stats.cpu_usage.total_usagePercent of time that CPU is under direct control of processes
    system CPUcpu_stats.system_cpu_usagePercent of time that CPU is executing system calls on behalf of processes
    throttling (count)cpu_stats.throttling_data.throttled_periodsNumber of CPU throttling enforcements for a container
    throttling (time)cpu_stats.throttling_data.throttled_timeTotal time that a container’s CPU usage was throttled
    Memorymemory_stats.usageMemory usage of a container
    RSSmemory_stats.stats.rssNon-cache memory for a process (stacks, heaps, etc.)
    Cache memorymemory_stats.stats.cacheData from disk cached in memory
    Bytesrx_bytes & tx_bytesNetwork traffic volume (send/receive)
    Packetsrx_packets & tx_packetsNetwork packet count (send/receive)
    Errors (transmit)Rx_errors & tx_errorsErrors in packet transmission
    Droppedrx_dropped & tx_errorsPackets dropped (send/receive)

    While monitoring Docker constants is a good first step towards Docker performance, events and logs still need to be taken into account for clear visibility over your system.

    II. Why monitor Docker Events

    Docker events provide critical information about your containers lifecycle. With events, you can see how and why they come and go. Events type would be the following: start, stop, pause, restart, kill or die. But Docker is continuously adding new ones with new versions.

    docker events

    The Docker remote API makes it very easy for you to subscribe to the stream of JSON events coming from containers.

    This is what an event looks like:

    {
      "image": "<image_name>",
      "id": "<container_id>",
      "name": "<container_name>",
      "host": "<your_host>",
      "type": "create",
      "message": "[Docker event] host=\"a5900527eead577df14c7917e83f0b6ebeb7b3d103e44d0a93a1c05316c6d391\" name=\"boring_hypatia\" event=\"create\""
    }
    

    Understanding why it matters to have events is quite straightforward. As each container embodies a service, knowing that somebody paused a container or that one simply died can explain the ups and downs of the whole service.

    When using auto-scaling and orchestrating solutions such as Kubernetes or AWS ECS, the number of containers becomes quite high. Accessing such events becomes critical to monitor how the orchestrators (or the OPS guys) manage the fleet and optimize your Docker production environment.

    III. Watching over Docker Logs

    1) Why monitor Docker logs

    Well, of course you should gather logs! Why make do without such interesting information? It’s no secret that logs are an incredibly convenient and powerful vehicle to extract everything you need to know about your application. To know more about what you should collect, refer to our article Collecting Logs Best Practices.

    2) How to monitor Docker logs

    1. Plenty of Docker logging methodologies
    2. But here comes the trouble: there are plenty of options at your disposable to get these docker logs out. And many of these existing options require too many manipulations that end up increasing both the complexity and maintenance cost of the overall infrastructure. So here are according to us the options you should not follow:

      • Log directly from your apps (log appenders)
      • Deploy a file appender in your container
      • Install a file collector in the container (Rsyslog, Logstash, FluentD, …)
      • Install a file collector as a container
      • Send logs using Syslog protocol
      • Collect files from Docker filesystems

    3. The best Docker logging collection methodology according to us
    4. Since pretty much the beginning of Docker, the community decided that it was a microservice architecture best practice to have one and only one application per container. In this scenario, deploying N applications means deploying N containers. It made the definition of a logging standard really easy as standard out (STDOUT) and standard error (STDERR) messages – the backbone of languages, logging libraries and operating systems – are correlated to a specific application.

      Docker makes it straightforward to grab these logs thanks to the Docker logs API or CLI and more and more people are thus publishing images in the Docker hub. So this would be an OK way to proceed to collect your Docker logs.

      However, from a practical point of view, using CLI or coding agents for the API takes up resources and time to implement for each application/container. We find there’s an even better way to go with the open source project Logspout.
      The Logspout approach is very neat, so it’s no surprise it is getting momentum. Let me explain how it works:

      • You deploy a Logspout container
      • It attaches to ALL containers on the host by using the Events API
      • It then routes ALL docker logs wherever you want ( thanks to output plugins)

      Pretty efficient, isn’t it?

    Getting further – Centralizing Docker stats, events & logs

    Docker stats, events & logs all have a role to play to ensure optimal application performance. While your system is small, centralizing all Docker data in one place and simply searching through when something happens looks manageable. The problem is, it usually gets out of hand sooner than expected.

    Though Logspout is a practical way to centralize Docker logs from various containers, you would still miss the essential Docker events and Docker stats… That you would miss once a container is killed and the data is simply not available anymore.

    At Logmatic.io, we solved this problem by developing the logmatic-docker container. It works as follows:

    • You add the logmatic-docker container in the docker machine as you usually do
    • The logmatic-docker container listens to events produced by the Docker daemon
    • The logmatic-docker container collects stats and logs from each Docker container
    • All Docker stats, events and logs are shipped in JSON format to Logmatic.io. Once all your Docker data is available in Logmatic.io, visualizing, filtering and searching become simple.

    Stay tuned for our upcoming article on what type of dashboards and alerts on Docker data are particularly interesting to improve your Docker performance! Or check out our Docker integration and get started in a flash.

    Related Posts

    Get notified when new content is published