Log aggregation is a pattern for centralizing application behavior in distributed systems in order to increase visibility to application behavior.
It is difficult to understand and diagnose application behaviour in modern distributed architectures (service-oriented, microservice, serverless, etc), because:
These characteristics make debugging opaque which leads to much longer resolution times.
Your first instinct with logging would be to simply track everything, but having an overabundance of data can make debugging more difficult.
If your logging platform has limited retention, you will need to adjust how verbose your logs are in order to maintain enough context to properly diagnose issues or opt to spend more on the solution.
Knowing the questions you want to answer will help direct what you should be including in your logs. By adding additional data to your log messages, such as structured metadata, you will more easily answer these questions in a timely manner.
Your logs will be of varying levels of importance, defined by their level. Defining a level for a given message helps you increase visibility in development, while maintaining a lower noise level in production.
Create a balance between human readable logs, and formatting that a machine can consume easily. Structured data in JSON, or key-value pairs are ideal.
Add context to your logs such as traceable IDs, and state information which facilitate an understanding of the system at time of the log occurring.
To maintain the privacy of your users and systems you must make sure you haven’t included any personally identifying information (PII) in your log metadata. Your logs will be archived and depending on your solutions, stored in a cloud platform.
Basic things like names or emails should not be included. Passwords, API keys, and should be especially avoided in logs.
Log aggregation is the term used to describe the method of collecting logs from disparate systems and centralizing them in a single interface. This makes it easier to analyze events happening across systems.
Log aggregation platforms then give an accessible interface with access controls to dictate who can see which logs. This interface then also gives tools for searching, as well as sharing or communicating subsets of events.
This reduces the complexity and length of time necessary to diagnose application behavior.
Timber is a cloud-based logging system that is designed for applications and developers, allowing you to spend less time debugging and more time shipping.
LogDNA enables you to instantly capture, search and graph your logs from any platform at any volume.