Netdata - v1.43.0

Groundbreaking: systemd-journal logs release!

Table of Contents

• Netdata Growth
• Release Summary
• Release Highlights
• systemd journal improvements
• Virtual Machine monitoring (VMWare vSphere)
• What is coming next
• Acknowledgments
• Contributions
  • Collectors
  • Packaging/Installation
  • Documentation
  • Health
  • Other Notable Changes
• Deprecation notice
• Netdata Release Meetup
• Support options

Steady to our schedule, this is another great Netdata release!

Netdata Growth

• 65.5 k GitHub Stars ⭐
  Since October 2023, Netdata is leading the observability category in the CNCF landscape, surpassing Elasticsearch. Thank you for your love ❤️! Give Netdata a ⭐ too, on GitHub!

• 595 M docker hub pulls
  Netdata runs with about 200k docker hub downloads per day. Since June 2023 we are a Verified Publisher, so that Netdata pulls don't count against docker hub pull limits for our users, allowing all our users to integrate Netdata to their CI/CD toolchains.

Release Summary

This release is the most robust and reliable Netdata we have ever built.

These are the main areas Netdata has improved since the last release:

1. Logs
   Today we release an almost rewritten version of systemd-journal, to improve its performance and visualization capabilities. systemd-journal holds critical systems and security information and given the lack of systemd-journal visualization tools, we focused first on filling this gap. At the same time, we are standardizing the way logs should be as a part of Netdata, enabling us to support more log management engines, like Loki and Elasticsearch.

2. Instances Slice and Dice
   Given the capabilities of the new Netdata Agent UI (v2), we are changing the way some of our collectors collect and expose metrics, to allow easier slicing and dicing of the data and be more OpenTelemetry compatible in terms of specifications. So, in this release we changed the way apps.plugin exposes charts in the Applications section of the dashboard. Following the NIDL framework, each application group is now an instance, allowing better aggregation of processes utilization across nodes. Similarly, our systemd units charts have been updated to have an instance for each systemd unit. For the same reasons, disk charts now have additional labels (id, model and serial) to help us identify disks from the charts. Unfortunately, such changes tend to make the older dashboards (v1, v0) less usable, especially on servers with many hundreds of instances.

3. Stock Alerts
   A number of changes have been implemented to the Netdata Health engine, to allow better integration with the new dashboard. More changes in this area are about to come, as part of the next release: a) allow multi-node alerts on parents, b) allow evaluating and configuring alerts from the UI.

4. Alerts Accuracy
   Netdata has by default 3 tiers of metrics, each with a different resolution. The Netdata query planner is automatically picking the right tier to satisfy a query, based on the number of points requested in the response. For alerts there was a side effect. Since alerts request only 1 point of data in the response, the query planner was picking the "easier" tier to query, which is of course the one with the lower resolution. Now alerts are always run on tier 0, the higher resolution one.

5. Lower Resources Utilization
   Several changes have been implemented for Netdata to better take care of itself. That includes lower memory usage, lower disk footprint, self vacuuming of SQLite databases, and more. Probably the most notable change is that now Netdata needs only 1 pointer (8 bytes on 64 bit, 4 bytes on 32 bit) for each use of a label name-value combination. This improves drastically Netdata's memory requirements in setups like busy k8s clusters, that containers come and go all the time, increasing the labels cardinality significantly.

6. 32bit Netdata on 64bit IoT machines
   A common request when Netdata is installed on 64bit IoT devices, is to have a 32bit Netdata running there. Before this release, this was not possible. Now a 32bit Netdata will nicely run on a 64bit operating system.

7. Netdata Cloud on prem
   Netdata Cloud is now available to be installed on-prem! Several companies have already deployed it and are currently testing it. If you want to join them, submit this form.

Release Highlights

systemd-journal

systemd-journal was first included in Netdata v1.42.0. Immediately after release, we recognized the wider need for this feature, so we've rewritten the plugin almost entirely, to provide the best possible experience. This work is also fundamental for supporting more log monitoring integrations - stay tuned!

The major improvements done on systemd-journal logs function were:
* addition of the histogram for log entries over time, with a break down per field-value, for any field and any time-frame
* enable of the PLAY mode provides the same experience as journalctl -f, showing new logs entries immediately after they are received
* allow filtering on any journal field or field value, for any time-frame
* add support for coloring log entries, the same way journalctl does

If you want to take a look at a full presentation of the systemd-journal plugin, how it works, how you can take full advantage of this and even instructions on configuration of a logs centralization server, check the documentation for the plugin.

You can experience the power of systemd-journal logs function in one of our Netdata demo rooms here
or check our latest YouTube video on it.

Want to know why you should untap the full potential of systemd-journal logs? Check out Netdata's founder, Costa Tsaousis @ktsaou, blogpost on it here.

Virtual Machine monitoring (VMWare vSphere)

With the increased feedback and requests on VMware vCenter Server collectors we have:
* Reviewed our out-of-the-box charts
* Added labels to the charts, e.g. host, datacenter, cluster, vm
* Reviewed the metadata on alerts
* Added summary charts section

It is with this feedback from the Community that we can keep working on improving Netdata to ensure it meets
your needs!

What is coming next

We are currently working on the following areas, which we hope to release next month:

1. Logs Explorer for Loki and Elasticsearch
   Similar to systemd-journal, allow Netdata to explore, query and visualize logs from Loki and Elasticsearch.

2. Collectors Configuration from the UI
   In the last release we presented the Integrations Marketplace. Since then, we work to make all integrations configurable via the dashboard. This will allow all of us to configure our Netdata servers directly from the UI, without touching configuration files, improving significantly the usability and easiness of Netdata.

3. Alerts Configuration from the UI
   Similarly, we work to allow configuring alerts directly from the UI, without text file configurations, so the all of us can create powerful alerts on the spot.

4. Netdata Mobile App
   We are at the final stage of releasing our Netdata Mobile App (iOS and Android) for receiving mobile push notifications and exploring alerts statuses.

5. Scalability
   Given the wide adoption of Netdata, we are committed to make Netdata scale better in larger environments. Especially when it comes to Netdata parents, we aim to provide the best scalability possible. We are currently finalizing the necessary changes to allow Netdata achieve:

   • 1 CPU core per 1 million metrics/s for data collection
   • 1 CPU core per 1 million metrics/s for ML and health (alerts)
   • 1 CPU core per 1 million metrics/s for re-streaming (pushing metrics to another parent)

   Of course, the numbers depend on the CPU and its clock, but they shouldn't vary significantly on modern systems.

   At the same time, we work to integrate Gorilla compression to our database. This will provide a significantly better overall memory footprint for Netdata.

Acknowledgments

We would like to thank our dedicated, talented contributors that make up this amazing community. The time and expertise that you volunteer are essential to our success. We thank you and look forward to continuing to grow together to build a remarkable product.

• @MAH69IK for improving ntfy notification title.
• @chpfm for fixing slave/user metrics collection stopping when query times out in go.d/mysql.
• @k0ste for various installation improvements on CentOS-Stream.
• @kylemanna for fixing an issue where a properly functioning sensors was skipped due to limits in python.d/sensors.
• @miversen33 for adding access control configuration to ntfy notification method.
• @novotnyJiri for fixing the wrong path in ansible-playbook deployment guide.
• @theggs for adding installation description for Homebrew on Apple Silicon.
• @vpnable for fixing counting UNDEF as users in go.d/openvpn_status_log.
• @zhqu1148980644 for fixing docker-compose example.
• @luisj1983 for implementing molecule tests in the netdata/ansible playbook

Contributions

Collectors

Packaging / Installation

Documentation

Health

Other Notable Changes

Deprecation notice

Changed in this release

In accordance with our previous deprecation notice, the following items in this release have been changed:

| Component | Type | Change | Action |
|--------------------------------------------------------------------------------------------------------------|------------------------------------|--------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------:|
| apps.plugin | collector | a dimension for each group/user/user group => a chart for each group/user/user group | |
| cgroups.plugin | collector | a dimension for each systemd service => a chart for each systemd service | |
| proc.plugin | collector | all "Networking Stack" metrics except "tcp" have been moved to "IPv4 Networking" | |
| family attribute | alert configuration and Health API | deprecated | use chart labels |

Will be changed in the next release

We plan to change in the next release (v1.44.0):

| Component | Type | Change | Action |
|-----------------------------------------------------------------------------------------------------------------|-----------|----------------------------------------------------|:----------------------------------------------------------------------------------------:|
| charts.d/nut | collector | deprecated | use go.d/upsd |

Netdata Release Meetup

Join the Netdata team on the 18th of October at 16:30 UTC for the Netdata Release Meetup.

Together we’ll cover:

• Release Highlights.
• Acknowledgments.
• Q&A with the community.

RSVP now - we look forward to meeting you.

Support options

As we grow, we stay committed to providing the best support ever seen from an open-source solution. Should you encounter an issue with any of the changes made in this release or any feature in the Netdata Agent, feel free to contact us through one of the following channels:

• Netdata Learn: Find documentation, guides, and reference material for monitoring and troubleshooting your systems with Netdata.
• GitHub Issues: Make use of the Netdata repository to report bugs or open a new feature request.
• GitHub Discussions: Join the conversation around the Netdata development process and be a part of it.
• Community Forums: Visit the Community Forums and contribute to the collaborative knowledge base.
• Discord Server: Jump into the Netdata Discord and hang out with like-minded sysadmins, DevOps, SREs, and other troubleshooters. More than 1700 engineers are already using it!