Much of the log data we handle doesn't offer substantial insight and can be conveniently removed from your logs, helping us reduce costs. What may seem like a small adjustment, like deleting an attribute, can have significant implications when scaled up.
A typical case involves fields in your logs presenting empty values or housing data considered irrelevant.
Below we’ll take a look at a few examples of what this looks like and how you can take action in BindPlane OP.
The Nginx logs presented below contain several fields where the value is simply a "-". This is a common occurrence with Nginx, providing no substantial value. It's recommended to eliminate these fields.
The Postgres logs displayed below have a 'role' and 'user' field, both of which are entirely empty.
Here’s how you can remove these fields in BindPlane OP
- Incorporate the “Delete Empty Values” processor into your pipeline.
- Determine the types of values you want the processor to search for. By default, it targets null values, but it can also identify empty lists or maps.
- Advanced configuration:
- If there are any fields you don’t ever want to be removed, specify them as an exclusion.
- Specify strings that the processor should consider as empty. For instance, in the case of the Nginx logs above, we want to regard “-” as an empty string and should include it in this list.
- Use Live Preview to validate the processor is working as expected!
That’s it! If you’d like to see this in action, check out the video tutorial below.