Breaking changes

This release updates OpenTelemetry libraries version to v0.40.0, and with that, it updates OTLP gRPC's default listening port from the legacy 55680 to the new 4317. There are two main routes to avoid downtime: configuring the receiver to listen in the old port 55680 and/or pushing traces to both ports simultaneously until the rollout is complete.

As part of adding support for full backend search, a search config parameter has had its name change from query_frontend.search.max_result_limit to query_frontend.search.default_result_limit.

• [CHANGE] BREAKING CHANGE The OTEL GRPC receiver's default port changed from 55680 to 4317. #1142 (@tete17)
• [CHANGE] BREAKING CHANGE Moved querier.search_max_result_limit and querier.search_default_result_limit to query_frontend.search.max_result_limit and query_frontend.search.default_result_limit #1174.

New Features and Enhancements

• [FEATURE]: Add support for inline environments. #1184 (@irizzant)
• [FEATURE] Added support for full backend search. #1174 (@joe-elliott)
• [ENHANCEMENT] Expose upto parameter on hedged requests for each backend with hedge_requests_up_to. #1085](https://github.com/grafana/tempo/pull/1085) (@joe-elliott)
• [ENHANCEMENT] Search: drop use of TagCache, extract tags and tag values on-demand #1068 (@kvrhdn)
• [ENHANCEMENT] Jsonnet: add $._config.namespace to filter by namespace in cortex metrics #1098 (@mapno)
• [ENHANCEMENT] Add middleware to compress frontend HTTP responses with gzip if requested #1080 (@kvrhdn, @zalegrala)
• [ENHANCEMENT] Allow query disablement in vulture #1117 (@zalegrala)
• [ENHANCEMENT] Improve memory efficiency of compaction and block cutting. #1121 #1130 (@joe-elliott)
• [ENHANCEMENT] Include metrics for configured limit overrides and defaults: tempo_limits_overrides, tempo_limits_defaults #1089 (@zalegrala)
• [ENHANCEMENT] Add Envoy Proxy panel to Tempo / Writes dashboard #1137 (@kvrhdn)
• [ENHANCEMENT] Reduce compactionCycle to improve performance in large multitenant environments #1145 (@joe-elliott)
• [ENHANCEMENT] Added max_time_per_tenant to allow for independently configuring polling and compaction cycle. #1145 (@joe-elliott)
• [ENHANCEMENT] Add tempodb_compaction_outstanding_blocks metric to measure compaction load #1143 (@mapno)
• [ENHANCEMENT] Update mixin to use new backend metric #1151 (@zalegrala)
• [ENHANCEMENT] Make TempoIngesterFlushesFailing alert more actionable #1157 (@dannykopping)
• [ENHANCEMENT] Switch open-telemetry/opentelemetry-collector to grafana/opentelemetry-collectorl fork, update it to 0.40.0 and add missing dependencies due to the change #1142 (@tete17)
• [ENHANCEMENT] Allow environment variables for Azure storage credentials #1147 (@zalegrala)
• [ENHANCEMENT] jsonnet: set rollingUpdate.maxSurge to 3 for distributor, frontend and queriers #1164 (@kvrhdn)
• [ENHANCEMENT] Reduce search data file sizes by optimizing contents #1165 (@mdisibio)
• [ENHANCEMENT] Add tempo_ingester_live_traces metric #1170 (@mdisibio)
• [ENHANCEMENT] Update compactor ring to automatically forget unhealthy entries #1178 (@mdisibio)
• [ENHANCEMENT] Added the ability to pass ISO8601 date/times for start/end date to tempo-cli query api search #1208 (@joe-elliott)
• [ENHANCEMENT] Prevent writes to large traces even after flushing to disk #1199 (@mdisibio)

Bug Fixes

• [BUGFIX] Add process name to vulture traces to work around display issues #1127 (@mdisibio)
• [BUGFIX] Fixed issue where compaction sometimes dropped spans. #1130 (@joe-elliott)
• [BUGFIX] Ensure that the admin client jsonnet has correct S3 bucket property. (@hedss)
• [BUGFIX] Publish tenant index age correctly for tenant index writers. #1146 (@joe-elliott)
• [BUGFIX] Ingester startup panic slice bounds out of range #1195 (@mdisibio)

Other Changes

• [CHANGE] Search: Add new per-tenant limit max_bytes_per_tag_values_query to limit the size of tag-values response. #1068 (@annanay25)
• [CHANGE] Reduce MaxSearchBytesPerTrace ingester.max-search-bytes-per-trace default to 5KB #1129 @annanay25
• [CHANGE] BREAKING CHANGE The OTEL GRPC receiver's default port changed from 55680 to 4317. #1142 (@tete17)
• [CHANGE] Remove deprecated method Push from tempopb.Pusher #1173 (@kvrhdn)
• [CHANGE] Upgrade cristalhq/hedgedhttp from v0.6.0 to v0.7.0 #1159 (@cristaloleg)
• [CHANGE] Export trace id constant in api package #1176
• [CHANGE] GRPC 1.33.3 => 1.38.0 broke compatibility with gogoproto.customtype. Enforce the use of gogoproto marshalling/unmarshalling for Tempo, Cortex & Jaeger structs. #1186 (@annanay25)
• [CHANGE] BREAKING CHANGE Remove deprecated ingester gRPC endpoint and data encoding. The current data encoding was introduced in v1.0. If running earlier versions, first upgrade to v1.0 through v1.2 and allow time for all blocks to be switched to the "v1" data encoding. #1215 (@mdisibio)