How to Remove Fields with Empty Values From Your Logs

Much of the log data we handle doesn't offer substantial insight and can be conveniently removed from your logs, helping us reduce costs. What may seem like a small adjustment, like deleting an attribute, can have significant implications when scaled up.

A typical case involves fields in your logs presenting empty values or housing data considered irrelevant.

Below, we’ll take a look at a few examples of what this looks like and how you can take action in BindPlane OP.

Nginx

The Nginx logs presented below contain several fields where the value is simply a "-". This is a common occurrence with Nginx, providing no substantial value. It's recommended to eliminate these fields.

Postgres

The Postgres logs displayed below have a 'role' and 'user' field, both of which are entirely empty.

Here’s how you can remove these fields in BindPlane OP

Incorporate the “Delete Empty Values” processor into your pipeline.
Determine the types of values you want the processor to search for. By default, it targets null values, but it can also identify empty lists or maps.
Advanced configuration:

If there are any fields you don’t ever want to be removed, specify them as an exclusion.
Specify strings that the processor should consider as empty. For instance, in the case of the Nginx logs above, we want to regard “-” as an empty string and should include it in this list.