🦖 🧡 Loki 2.5.0 🧡 🦖

It has been nearly 6 months since Loki 2.4 was released, and we’ve been busy making Loki better than ever. Here's a summary of new enhancements and important fixes.

Features and enhancements

• Thanks to @bboreham for taking a deep dive on the Go regexp library, and for creating a fork which greatly improves the performance of common regex uses cases in Loki.
• Binary operations are now significantly faster, taking full advantage of Loki's parallelism.
• A new schema is available, which uses more path prefixes to avoid rate limits on S3.
• That same schema change was also added to the filesystem store, which avoids using one directory to store every chunk.
• A new capability for hedging requests to storage improves performance on highly parallelized queries. Refer to the hedging configuration block for more information.
• Promtail has several new ways to ingest logs:
  • The ability to do service discovery and tailing directly from the Docker daemon.
  • Fetching logs directly from Cloudflare.
  • Receiving logs directly in Graylog Extended Log Format (GELF).
• Promtail permits client-side global rate limiting and pipeline-configurable rate limiting.

For a full list of all changes please look at the CHANGELOG.

Upgrade Considerations

As always, please read the upgrade guide before upgrading Loki.

Changes to the config split_queries_by_interval

The most likely impact many people will see is Loki failing to start because of a change in the YAML configuration for split_queries_by_interval. It was previously possible to define this value in two places.

query_range:
  split_queries_by_interval: 10m

and/or

limits_config:
  split_queries_by_interval: 10m

In 2.5.0 it can only be defined in the limits_config section.

Loki will fail to start if you do not remove the split_queries_by_interval configuration parameter from the query_range section.

Additionally, it has a new default value of 30m rather than 0.

The CLI flag is not changed and remains querier.split-queries-by-interval.

More parallelism by default

We continue to move defaults in Loki to leverage parallelism in all configurations, including the single binary.

In v2.5, all queries will be split and sharded by default. This will likely result in increased memory and CPU usage for Loki processes during queries, if you didn’t previously have these values enabled.

Usage reporting

Loki 2.5 includes code we added to report anonymous usage statistics back to Grafana Labs, an issue was created to outline the intent, and what went into the final implementation can be seen here in the source.

Usage reporting helps provide anonymous information on how people use Loki and what the Loki team should focus on for features and documentation. No private information is collected, and all reports are completely anonymous.

If possible, we ask you to leave the usage reporting feature enabled and help us understand more about Loki! We are also working to figure out how we can share this info with the community so everyone can watch Loki grow.

If you would rather not participate in usage stats reporting, the feature can be disabled in config

analytics:
  reporting_enabled: false

Bug fixes

2.5.0 bug fixes

V2.5.0 fixes numerous bugs. The CHANGELOG has the complete list.

A summary of some of the more important fixes:

• Query correctness:
• PR 5474 cyriltovena: Disable sharding of count/avg when labels are mutated
• PR 5444 cyriltovena: Do not insert missing point when sharding
• PR 5423 cyriltovena: Correctly sets hash value for headblock iterator
• PR 5289 cyriltovena: Fixes log deduplication when mutating Labels using LogQL
• PR 5006 sandeepsukhani: Fix splitting of queries when step is larger than split interval
• PR 5413 owen-d: A deadlock in the Azure blob client
• PR 5334 kavirajk: A fix in live tailing that could lead to exploding memory usage
• PR 5144 owen-d: A fix for the ruler when trying to use Basic Auth for remote write
• Query cancellation:
• PR 5113 kavirajk: Fix cancel issue between query-frontend and query scheduler
• PR 5080 kavirajk: Handle context cancellation in some of the querier downstream requests
• PR 5075 cyriltovena: Fixes a possible cancellation issue in the frontend
• PR 4741 sandeepsukhani: An issue where retention didn't cleanup the index completely

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

• https://hub.docker.com/r/grafana/loki
• https://hub.docker.com/r/grafana/promtail

$ docker pull "grafana/loki:2.5.0"
$ docker pull "grafana/promtail:2.5.0"

Binary

We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:

$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.5.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"