Release v1.36

Table of contents

• Release highlights
  • Metric correlations
    • New metric correlation algorithm (tech preview)
    • Cooperation of the Metric Correlations (MC) component with the Anomaly Advisor
    • Metric correlations dashboard
    • What's next with Metric Correlations
  • Tiering, providing almost unlimited metrics for your nodes
  • Kubernetes
  • Anomaly Rate on every chart
  • Centralized Admin Interface & Bulk deletion of offline nodes
  • Agent and Cloud chart metadata syncing
  • Visualization improvements
  • Labels on every chart
• Acknowledgments
• Contributions
• Deprecation notice
• Netdata Release Meetup
• Support options

❗ We're keeping our codebase healthy by removing features that are end of life. Read the deprecation notice to check if you are affected.

Netdata open-source growth

• 7.6M+ troubleshooters monitor with Netdata
• 1.6M unique nodes currently live
• 3.3k+ new nodes per day
• Over 557M Docker pulls all-time total
• Over 60,000 stargazers on GitHub

Release highlights

Metric correlations

New metric correlation algorithm (tech preview)

The Agent's default algorithm to run a metric correlations job (ks2) is based on Kolmogorov-Smirnov test. In this release, we also included the Volume algorithm, which is an heuristic algorithm based on the percentage change in averages between the highlighted window and a baseline, where various edge cases are sensibly controlled. You can explore our implementation in the Agent's source code

This algorithm is almost 73 times faster than the default algorithm (named ks2) with near the same accuracy. Give it a try by enabling it by default in your netdata.conf.

[global]
   # enable metric correlations = yes
   metric correlations method = volume

Cooperation of the Metric Correlations (MC) component with the Anomaly Advisor

The Anomaly Advisor feature lets you quickly surface potentially anomalous metrics and charts related to a particular highlight window of interest. When the Agent trains its internal Machine Learning models, it produces an Anomaly Rate for each metric.

With this release, Netdata can now perform Metric Correlation jobs based on these Anomalous Rate values for your metrics.

Metric correlations dashboard

In the past, you used to run MC jobs from the Node's dashboard with all the settings predefined. Now, Netdata gives you some extra functionality to run an MC job for a window of interest with the following options:

1. To run an MC job on both Metrics and their Anomaly Rate
2. To change the aggregation method of datapoints for the metrics.
3. To choose between different algorithms

All this from the same, single dashboard.

What's next with Metric Correlations

Troubleshooting complicated infrastructures can get increasingly hard, but Netdata wants to continually provide you with the best troubleshooting experience. On that note, here are some next logical steps for for our Metric Correlations feature, planned for upcoming releases:

1. Enriching the Agent with more Metric Correlation algorithms.
2. Making the Metric Correlation component run seamless (you can explore the /weights endpoint in the Agent's API; this is a WIP).
3. Giving you the ability to run Metric Correlation Jobs across multiple nodes.

Be on the lookout for these upgrades and feel free to reach us in our channels with your ideas.

Tiering, providing almost unlimited metrics for your nodes

Netdata is a high fidelity monitoring solution. That comes with a cost, the cost of keeping those data in your disks. To help remedy this cost issue, Netdata introduces with this release the Tiering mechanism for the Agent's time-series database (dbengine).

Tiering is the mechanism of providing multiple tiers of data with different granularity on metrics by doing the following:

1. Downsampling the data into lower resolution data.
2. Keeping statistical information about the metrics to recreate the original* metrics.

Visit the Tiering in a nutshell section in our docs to understand the maximum potential of this feature. Also, don't hesitate to enable this feature to change the retention of your metrics

Note: *Of course the metric may vary; you can just recreate the exact time series without taking into consideration other parameters.

Kubernetes

A Kubernetes Cluster can easily have hundreds (or even thousands) of pods running containers. Netdata is now able to provide you with an overview of the workloads and the nodes of your Cluster. Explore the full capabilities of the k8s_state module

Anomaly Rate on every chart

In a previous release, we introduced unsupervised ML & Anomaly Detection in Netdata with Anomaly Advisor. With this next step, we’re bringing anomaly rates to every chart in Netdata Cloud. Anomaly information is no longer limited to the Anomalies tab and will be accessible to you from the Overview and Single Node view tabs as well. We hope this will make your troubleshooting journey easier, as you will have the anomaly rates for any metric available with a single click, whichever metric or chart you happen to be exploring at that instant.

If you are looking at a particular metric in the overview or single node dashboard and are wondering if the metric was truly anomalous or not, you can now confirm or disprove that feeling by clicking on the anomaly icon and expanding the anomaly rate view. Anomaly rates are calculated per second based on ML models that are trained every hour.

For more details please check our blog post and video walkthrough.

Centralized Admin Interface & Bulk deletion of offline nodes

We've listened and understood the your pain around Space and War Room settings in Netdata Cloud. In response, we have simplified and organized these settings into a Centralized Administration Interface!

In a single place, you're now able to access and change attributes around:

• Space
• War Rooms
• Nodes
• Users
• Notifications
• Bookmarks

Along with this change, the deletion of individual offline nodes has been greatly improved. You can now access the Space settings, and on Nodes within which it is possible to filter all Offline nodes, you can now mass select and bulk delete them.

Agent and Cloud chart metadata syncing

On this release, we are doing a major improvement on our chart metadata syncing protocol. We moved from a very granular message exchange at chart dimension level to a higher level at context.

This approach will allow us to decrease the complexity and points of failure on this flow, since we reduced the number of events being exchanged and scenarios that need to be dealt with. We will continuously fix complex and hard-to-track existing bugs and any potential unknown ones.

This will also bring a lot of benefits to data transfer between Agents to Cloud, since we reduced the number of messages being transmitted.

To sum up these changes:

1. The traffic between Netdata cloud and Agents is reduced significantly.
2. Netdata Cloud scales smoother with hundreds of nodes.
3. Netdata Cloud is aware of charts and nodes metadata.

Visualization improvements

Composite chart enhancements

We have restructured composite charts into a more natural presentation. You can now read composite charts as if reading a simple sentence, and make better sense of how and what queries are being triggered.

In addition to this, we've added additional control over time aggregations. You can now instruct the agent nodes on what type of aggregation you want to apply when multiple points are grouped into a single one.

The options available are: min, max, average, sum, incremental sum (delta), standard deviation, coefficient of variation, media, exponential weighted moving average and double exponential smoothing.

Theme restyling

We've also put some effort to improve our light and dark themes. The focus was put on:

• optimizing space for the information that is crucial to you when you're exploring and/or troubleshooting your nodes.
• improving contrast ratios so that the components and data that are more relevant don't get lost among other noise.

Labels on every chart

Most of the time, you will group metrics by their dimension or their instance, but there are some benefits to other groupings. So, you can now group them by logical representations.

For instance, you can represent the traffic in your network interfaces by their interface type, virtual or physical.

This is still a work in progress, but you can explore the newly added labels on the following areas/charts:

• Disks
• Mountpoints in your system
• Network interfaces both wired and wireless
• MD arrays
• Power supply units
• Filesystem (like BTRFS)

Acknowledgments

We would like to thank our dedicated, talented contributors that make up this amazing community. The time and expertise that you volunteer is essential to our success. We thank you and look forward to continue to grow together to build a remarkable product.

• @didier13150 for fixing boolean value for ProtectControlGroups in the systemd unit file.
• @kklionz for fixing a base64_encode bug in Exporting Engine.
• @kralewitz for fix parsing multiple values in nginx upstream_response_time in go.d/web_log.
• @mhkarimi1383 for adding an alternative way to get ansible plays to Ansible quickstart.
• @tnyeanderson for fixing netdata-updater.sh sha256sum on BSDs.
• @xkisu for fixing cgroup name detection for docker containers in containerd cgroup.
• @boxjan for adding Chrony collector.

Contributions

Collectors

New

⚙️ Enhancing our collectors to collect all the data you need.

• Add PgBouncer collector (go.d/pgbouncer) (#748, @ilyam8)
• Add WireGuard collector (go.d/wireguard) (#744, @ilyam8)
• Add PostgresSQL collector (go.d/postgres) (#718, @ilyam8)
• Add Chrony collector (go.d/chrony) (#678, @boxjan)
• Add Kubernetes State collector (go.d/k8s_state) (#673, @ilyam8)

Improvements

⚙️ Enhancing our collectors to collect all the data you need.

Bug fixes

🐞 Improving our collectors one bug fix at a time.

eBPF

• Fix data collection frequency (#13351, @thiagoftsm)
• Fix crash on cleanup (#13259, @thiagoftsm)

Exporting

Documentation

📄 Keeping our documentation healthy together with our awesome community.

Packaging / Installation

📦 "Handle with care" - Just like handling physical packages, we put in a lot of care and effort to publish beautiful software packages.

Other Notable Changes

Improvements

⚙️ Greasing the gears to smoothen your experience with Netdata.

Bug fixes

🐞 Increasing Netdata's reliability one bug fix at a time.

Code organization

🏋️ Changes to keep our code base in good shape.

Deprecation notice

The following items will be removed in our next minor release (v1.37.0):

Patch releases (if any) will not be affected.

| Component | Type | Will be replaced by |
|----------------------------------------------------------------------------------------------------------|:---------:|:------------------------------------------------------------------------------------:|
| python.d/postgres | collector | go.d/postgres |

All the deprecated components will be moved to the netdata/community repository.

Deprecated in this release

In accordance with our previous deprecation notice, the following items have been removed in this release:

| Component | Type | Replaced by |
|------------------------------------------------------------------------------------------------------------------------|:---------:|:--------------------------------------------------------------------------------------------------------:|
| python.d/chrony | collector | go.d/chrony |
| python.d/ovpn_status_log | collector | go.d/openvpn_status_log |

Netdata Release Meetup

Join the Netdata team on the 11th of August for the Netdata Agent Release Meetup, which will be held on the Netdata Discord.

Together we’ll cover:

• Release Highlights
• Acknowledgements
• Q&A with the community

RSVP now

We look forward to meeting you.

Support options

As we grow, we stay committed to providing the best support ever seen from an open-source solution. Should you encounter an issue with any of the changes made in this release or any feature in the Netdata Agent, feel free to contact us through one of the following channels:

• Netdata Learn: Find documentation, guides, and reference material for monitoring and troubleshooting your systems with Netdata.
• Github Issues: Make use of the Netdata repository to report bugs or open a new feature request.
• Github Discussions: Join the conversation around the Netdata development process and be a part of it.
• Community Forums: Visit the Community Forums and contribute to the collaborative knowledge base.
• Discord: Jump into the Netdata Discord and hangout with like-minded sysadmins, DevOps, SREs and other troubleshooters. More than 1100 engineers are already using it!