Logmatic.io Blog

A Docker logging guide

Managing stats, events and Docker logs

In my daily job I get the chance to meet and exchange with various passionate tech teams from all sorts of companies building various applications. From this very variety a trend is emerging: the rise of microservices.

docker microservices

Splitting up an application in smaller and more maintainable parts has plenty of advantages in the modern development world as issues become isolated. As teams build code bases easier to manage, adding new features and deploying is facilitated. So is facilitated the scaling of independant services. Plus your application is not dependant on a machine anymore, which makes it even better than the common VMs architecture.

Everybody seems to agree that Docker is the best open source project to build microservices – I would even say that Docker seems to be the biggest influencer in encouraging people to adopt /move to microservices.

So Docker has become very trendy – that’s a fact – and they’ve already become quite common in production. However, as containers with short live spans are used in large numbers, collecting and analyzing monitoring data requires significant investments. And so we witnessed many apps using containers that are not monitoring themselves – which makes them vulnerable to downtime or functional bugs.

Docker logging & monitoring is challenging

Docker does not provide a clear answer to this problem, and does not advertise for a specific solution either. Lots of articles have popped up on the topic over the last 2 years, covering all the different options available to collect your data, but usually focusing on monitoring.

We find that focusing on Docker monitoring leaves a whole set of very interesting data aside, and that app performance comes out of tech team’s ability to get visibility over metrics & events as well as logs. And our own experience with Docker tells us the same goes for Docker performance. So we’re sharing in this article:

  • Why all 3 Docker data sets – stats, events & logs – are important to look into to promote proper Docker performance
  • How to more specifically extract Docker logs easily

I. Why monitor Docker Stats

Docker stats were released for the first time with Docker 1.5. It provides a whole set of monitoring constants.

docker stats

The good news is that the Docker provided API streams real nice JSON objects that look like logs. Setting alerts for critical metrics going out of allocated boundaries thus becomes possible.

But how important is it to look at these Docker constants really? We find the insights that can be extracted from them are not often clear. Take CPU for example. What this metric is really showing you is how much CPU the container consumes over the Docker machine it is deployed into. If the container consumes 100% of the CPU, it means other containers may suffer from the lack of CPU. That’s why CPU throttling – also available in the stats – is an important metric too. When deploying a container you can define CPU shares constraints. You can thus decide that if 2 containers run side-by-side, each of them cannot exceed 50% of the available CPU. When this threshold is met, Docker will report it in the throttling metric (count or time). If your system encounters too much of it, it means that one of your applications cannot perform at full capacity, possibly generating bad user experience.

This story aside here are the metrics you should really focus on:

NameAttribute in JSON docDescription
user CPUcpu_stats.cpu_usage.total_usagePercent of time that CPU is under direct control of processes
system CPUcpu_stats.system_cpu_usagePercent of time that CPU is executing system calls on behalf of processes
throttling (count)cpu_stats.throttling_data.throttled_periodsNumber of CPU throttling enforcements for a container
throttling (time)cpu_stats.throttling_data.throttled_timeTotal time that a container’s CPU usage was throttled
Memorymemory_stats.usageMemory usage of a container
RSSmemory_stats.stats.rssNon-cache memory for a process (stacks, heaps, etc.)
Cache memorymemory_stats.stats.cacheData from disk cached in memory
Bytesrx_bytes & tx_bytesNetwork traffic volume (send/receive)
Packetsrx_packets & tx_packetsNetwork packet count (send/receive)
Errors (transmit)Rx_errors & tx_errorsErrors in packet transmission
Droppedrx_dropped & tx_errorsPackets dropped (send/receive)

While monitoring Docker constants is already a good first step towards Docker performance, it remains just a first step. Events and logs still need to be taken into account for clear visibility over your system.

II. Why monitor Docker Events

Docker events provide critical information about your containers lifecycle. With events, you can see how and why they come and go. Events type would be the following: start, stop, pause, restart, kill or die. But Docker is continuously adding new ones with new versions.

docker events

The Docker remote API makes it very easy for you to subscribe to the stream of JSON events coming from containers. For the curious ones out there, this is what an event looks like:

  "image": "<image_name>",
  "id": "<container_id>",
  "name": "<container_name>",
  "host": "<your_host>",
  "type": "create",
  "message": "[Docker event] host=\"a5900527eead577df14c7917e83f0b6ebeb7b3d103e44d0a93a1c05316c6d391\" name=\"boring_hypatia\" event=\"create\""

Understanding why having events matters is then quite straightforward. As each container embodies a service, knowing that somebody paused a container or that one simply died can explain the ups and downs of the whole service.

When using auto-scaling and orchestrating solutions such as Kubernetes or AWS ECS, the number of containers becomes quite high, so accessing such events becomes critical for you to monitor how the orchestrators (or the OPS guys) manage the fleet and optimize your Docker production environment.

III. Watching over Docker Logs

1) Why monitor Docker logs

Well, of course you should gather logs! Why make do without such interesting information? It’s no secret that logs are this incredibly incredibly convenient and powerful vehicle to extract everything you need to know about your application. To know more about what you should collect, refer to our article Collecting Logs Best Practices.

2) How to monitor Docker logs

  1. Plenty of Docker logging methodologies
  2. But here comes the trouble: there are plenty of options at your disposable to get these docker logs out. And many of these existing options require too many manipulations that end up increasing both the complexity and maintenance cost of the overall infrastructure. So here are according to us the options you should not follow:

    • Log directly from your apps (log appenders)
    • Deploy a file appender in your container
    • Install a file collector in the container (Rsyslog, Logstash, FluentD, …)
    • Install a file collector as a container
    • Send logs using Syslog protocol
    • Collect files from Docker filesystems

  3. The best Docker logging collection methodology according to us

Since pretty much the beginning of Docker, the community decided that it was a microservice architecture best practice to have one and only one application per container. In this scenario, deploying N applications means deploying N containers. It made the definition of a logging standard really easy as standard out (STDOUT) and standard error (STDERR) messages – the backbone of languages, logging libraries and operating systems – are correlated to a specific application.

Docker makes it straightforward to grab these logs thanks to the Docker logs API or CLI and more and more people are thus publishing images in the Docker hub. So this would be an OK way to proceed to collect your Docker logs.

However, from a practical point of view, using CLI or coding agents for the API takes up resources and time to implement for each application/container. We find there’s an even better way to go with the open source project Logspout.
Indeed, the Logspout approach is very neat, so it’s no surprise it is getting momentum. Let me explain how it works:

    • You deploy a Logspout container
    • It attaches to ALL containers on the host by using the Events API
    • It then routes ALL docker logs wherever you want ( thanks to output plugins)

Pretty efficient, isn’t it?

Getting further – Centralizing Docker stats, events & logs

Docker stats, events & logs have all a role to play to ensure optimal application performance. While your system is small, centralizing all Docker data in one place and simply searching it when something happens looks manageable. The problem is, it usually goes out of hand sooner than expected.

Logspout is a practical way to centralize Docker logs coming from various containers. Yet you would still miss the essential Docker events and Docker stats… That you would miss once a container is killed and the data is simply not available anymore.

At Logmatic.io, we solved this conundrum by developing the logmatic-docker container. It works as follows:

  • You add the logmatic-docker container in the docker machine as you usually do
  • The logmatic-docker container listens to events produced by the Docker daemon
  • The logmatic-docker container collects stats and logs from each Docker container
  • All Docker stats, events and logs are shipped in JSON format to Logmatic.io. Once all your Docker data is available in Logmatic.io, visualizing, filtering and searching become simple.

Stay tuned for our upcoming article on what type of dashboards and alerts on Docker data are particularly interesting to improve your Docker performance!

Related Posts