Intent

Log aggregation is a pattern for centralizing application behavior in distributed systems in order to increase visibility to application behavior.

Problem

It is difficult to understand and diagnose application behaviour in modern distributed architectures (service-oriented, microservice, serverless, etc), because:

  • Logs are local — Access is an issue. Security best practices (such as the principle of least privilege) suggest access to individual systems be restricted, meaning application debugging becomes opaque to those who do not have access.
  • Logs are inconsistent — Each source has a proprietary format with varying levels of metadata.
  • Logs are noisy — Practically every process on every host will produce logs and deciding what to persist is as much an art as science.

These characteristics make debugging opaque which leads to much longer resolution times.

Solution

What should you log?

Your first instinct with logging would be to simply track everything, but having an overabundance of data can make debugging more difficult.

If your logging platform has limited retention, you will need to adjust how verbose your logs are in order to maintain enough context to properly diagnose issues or opt to spend more on the solution.

Knowing the questions you want to answer will help direct what you should be including in your logs. By adding additional data to your log messages, such as structured metadata, you will more easily answer these questions in a timely manner.

Log formatting

Your logs will be of varying levels of importance, defined by their level. Defining a level for a given message helps you increase visibility in development, while maintaining a lower noise level in production.

Create a balance between human readable logs, and formatting that a machine can consume easily. Structured data in JSON, or key-value pairs are ideal.

Add context to your logs such as traceable IDs, and state information which facilitate an understanding of the system at time of the log occurring.

Personally identifiable information

To maintain the privacy of your users and systems you must make sure you haven’t included any personally identifying information (PII) in your log metadata. Your logs will be archived and depending on your solutions, stored in a cloud platform.

Basic things like names or emails should not be included. Passwords, API keys, and should be especially avoided in logs.

Aggregation makes your logs accessible

Log aggregation is the term used to describe the method of collecting logs from disparate systems and centralizing them in a single interface. This makes it easier to analyze events happening across systems.

Log aggregation platforms then give an accessible interface with access controls to dictate who can see which logs. This interface then also gives tools for searching, as well as sharing or communicating subsets of events.

This reduces the complexity and length of time necessary to diagnose application behavior.

Structure

Pros and Cons

Pros

  • Easy to get started
  • Shared access
  • Alerting

Cons

  • Can be overwhelming due to volume of data
  • Investment required for uniform logs
  • Third party storage

Relevant Services

Timber.io

Timber is a cloud-based logging system that is designed for applications and developers, allowing you to spend less time debugging and more time shipping.

Timber.io Logging
Timber.io LoggingLog better. Solve problems faster.
Try Freearrow_right

LogDNA

LogDNA enables you to instantly capture, search and graph your logs from any platform at any volume.

LogDNA
LogDNAThe best logging service you will ever use
Try Freearrow_right
Manifold Background

Log Aggregation

Make application debugging simple by centralizing logs from microservices