Log Management 101: Log Sources to Monitor

Log management software gives the primary diagnostic data in your applications’ development, deployment, and maintenance. However, choosing the log sources to log and monitor could often be daunting. The primary cause of concern in monitoring all sources is the high price tag that many SIEM tools in the market charge based on the number of users and sources ingesting logs. At observIQ, we offer unlimited users and sources. You only pay for what you ingest and how long you retain; if your retention needs are minimal, you can use observIQ for free. Ingesting logs from some sources, such as firewalls, IDS, active directories, and endpoint tools, is pretty straightforward. However, some critical sources of your incident response plan could have complex configurations that may deter you away from implementing Log Management for those sources. With advanced log management tools such as observIQ, most sources come preconfigured to work with the log agent, making adding sources simple. As a best practice in logging, businesses evaluate and implement logging for sources that:

• Are required for actionable solutions such as monitoring, troubleshooting, etc. The suggested best practice is to archive non-actionable logs if necessary for compliance. This evaluation can tone down the noise in your log ingestion and make a more need-based log management process.
• Maximize the return on investment through increased visibility into application events.
• Address the existing need for compliance.
• Cover event logs from areas of their network/ infrastructure prone to hacks and malicious attacks.
• Voids the blind spots in a network.

On the web, you will find generic information about log sources and what you can do with data ingested into any SIEM tool. However, there is a critical step that every business/ individual evaluating a log management tool must take: take stock of all your network/application components. In-house or custom applications developed recently often create logs in a standard format such as JavaScript Object Notation (JSON) or Syslog. All logs are never saved in the standard format. Logs from servers and firewalls are easily ingestible and are parsed seamlessly. However, log management is challenging when working with DNS and other physical security platforms. Not logging DNS or security platforms could deter the security and block insights into crucial network components. Many businesses must pay more attention to challenging logging components, fearing the high human effort involved in developing the necessary plug-ins for ingesting from many sources. Multiple surveys from organizations of all levels estimate that businesses consume less than half of the sources in their network. Please be aware of all the aspects of your application and the infrastructure you need to log in.

Related Content: How to Manage Sensitive Log Data

1. DNS

It provides highly detailed information about DNS data sent and received by the DNS server. DNS attacks include DNS hijacking, DNS tunneling, Denial-of-Service (DoS) attacks, Command and control, and cache poisoning. Hence, DNS logs help identify information related to these attacks so that the source can be found. These include detailed data on records requested, client IP, request flags, zone transfers, query logs, rate timing, and DNS signing.

DNS logs are a wealth of data from user site access, malicious site requests, DNS attacks, Denial of Service (DoS) events, etc. Based on the DNS logs, businesses can make critical network security decisions. DNS is the primary protocol for applications and web browsers to operate based on domain names. Although DNS was initially not intended for general-purpose tunneling, many utilities have been developed to tunnel over DNS. Since the data transfer capabilities of DNS are unintentional, DNS is often an ignored space in log ingestion. If the tunneling capabilities of DNS are not monitored, it could lead to a severe risk to a network. Payload and traffic analysis are the two core components that need monitoring in DNS.

Logging all components of DNS can become very noisy and make analysis impossible. The DNS log data is often voluminous and is in a multi-line format. Listed below are a few common scenarios where the DNS logs ingestion is mandatory and helpful:

• When the DNS queries are done using the TCP instead of the UDP
• When requests from an internal RFC 1918 IP address are not associated with the business domain.
• When a zone transfer occurs to an unauthorized/ unknown DNS server
• When a record request mentions an unconventional file type with many meaningless characters embedded in it.
• When there are requests to non-standard ports at hours that are not standard usage times.
• When country domain extensions are uncommon from the business’ network, such as .ru(Russia) or .cn(China), a widespread trigger will occur if the company does not operate in those countries.
• When there are increased lookup requests and failures.

2. Packet Capture Logs

Packet Capture(PCAP) is an API that captures packet data for network traffic in the OSI layers. It is important to note that PCAP does not log by itself. Instead, a network analyzer collects and records the packet data information. The events logged in a .pcap file include:

• Malware detections
• Bandwidth usage
• DHCP server issues
• DNS events

During network threat analysis, the packet file data helps detect all network intrusions and all other suspicious packet transfers. The packet data helps drill down to the root cause of a network issue. For example, if you see a response failure from an application call, a study of the packet rate and packet information can reveal if the problem is with the request or the response. PCAP logs are in a simple format, making ingestion and parsing simple for most log agents.

3. Cloud Platform Logs

Most businesses host their applications on cloud platforms such as Google Cloud Platform(GCP), Amazon Web Services(AWS), etc. The log management software will inevitably ingest the logs from these cloud platforms. Many businesses must refrain from ingesting cloud platform logs due to discrepancies in the log formats and the parsing agent. Companies like observIQ have ready-to-use plug-ins for these platforms, making log ingestion possible in minutes. Our log agent, Stanza, can efficiently manage the number of events in the cloud platforms, an area where most log parsers fail. However, limiting the ingestion to actionable events is recommended only to reduce the noise in the cloud platform events to a manageable level. Some of the most critical cloud platform events that you need to consider monitoring are:

• Changes in user permissions and roles
• Any changes made to the instances, such as creation/deletion in the cloud infrastructure
• Requests to data buckets containing sensitive data buckets are made by users accessing them remotely.
• Unauthorized account or file access
• Communication with external sources
• Alerts for the transformation of personally identifiable information

4. Windows Events Logs

Event logs from Windows are critical for securing the network, troubleshooting issues, and retaining events for compliance. Windows events such as network connections, errors, traffic, system behavior, and unauthorized access are logged. A Windows system can produce a large volume of log data daily. While managing such volume is difficult, log management software makes handling Windows event logs easier. In a Windows environment, three primary categories of event logs are tracked: system logs, application logs, and security logs. Logs from the System Monitor, a driver, also log to the Windows events log. The system monitors and logs events such as file creation and modification, driver installations/ deletions, process creation, accessing raw disks, etc. The recommended practice is centralizing the Windows events onto a server from which the SIEM/ log agent can read the logs. The log management software chosen should be able to aggregate all Windows logs into a standard format, provide an alerting mechanism for network anomalies, and offer visualization capabilities. observIQ offers simplified logging for Windows events; check our user documentation for simple plug-in configuration for Windows logs.

5. Database Logs

Companies are apprehensive about logging database events because they worry that logging could inversely affect the server’s performance. Capturing events from databases and tables could be challenging, considering the large number of application databases. If there are databases created by third-party service providers, gaining access to log events in those databases would be a challenge due to access restrictions. A sound Database Activity Monitoring system helps gain visibility into databases. Since the DAM works similarly to a firewall in its restrictive functionality, logging database events via it makes it a more compliance-oriented monitoring practice. They use a stored procedure to capture specific events and log accurate information about the event with the record ID. Stored procedures can be used to track administrator accesses, malicious and unauthorized access attempts, authorization failures, start and end event logs for servers, all schema-related operations, requests for modifications to a large set of records in the database, etc.

6. Linux Logs

Log all Linux events to get a timeline for system, application, and operation system-related events. Errors and issues on the desktop are logged in different locations. The location where the logs are written is configurable in most cases. All Linux logs are entered in plain text format. /var/log is the directory to which the logs are saved. In Linux, every event has the potential to be logged; anything from package manager events to Apache servers can be logged. All logs except authorization-related logs are logged to a Syslog in the /var/log directory. The most critical Linux logs are Application, event, service, and system logs.

7. Infrastructure Device Logs

Infrastructure devices are the lifeline of information transport architecture. Monitoring the routers, switches, and switches in a network gives an insight into the health and functioning of the network. Enriching the log data from infrastructure devices can increase machine and user interaction visibility. Logs from infrastructure are vital submissions in the compliance package. Infrastructure logs from highly distributed environments need a straightforward alerting mechanism in place. When there are network failures, only through a detailed log analysis can the DevOps triage and fix the issue. Sophisticated monitoring tools like observIQ have alerting mechanisms based on thresholds and metric indicators for problems. Logs from different infrastructure devices are output in various formats.
In most cases, these logs are unstructured data most suitable for batch processing through a tool rather than manual analysis. Most application developers are interested in logging configuration changes to infrastructure devices. Knowing the origination of configuration changes helps address any issues arising from misconfigurations.

Related Content: Reducing Log Volume with Log-based Metrics

8. Containerized Application Logs

Although containerized applications are a new and emerging form of application management, they are becoming increasingly business-critical. This is the most in-vogue source for log ingestion for businesses of all sizes. Collating the logs from the application, the host OS, and Docker is necessary to effectively log a containerized application.

9. WebServers Logs

Logging WebServer events can be challenging, but WebServer logs help businesses understand the end user’s interaction with the application. While businesses ignored the need to log web servers in the past, with the increasing need to study user behavior logs from web servers such as Apache, NGINX is being looked at with new interest. Web server logs contain helpful information such as:

• User visits and application access logs
• User login and duration for which the user is logged in
• Page view count
• User information such as browser used to access the application, version of the OS
• Bots' access to the application
• HTTP requests

10. Security Device Logs

As more businesses adopt a more cloud-based approach for application delivery, the devices at the edge of customer interaction are becoming extremely valuable. With this shift to cloud infrastructure, security devices such as firewalls are experiencing a significant spike in traffic loads. Logs from security devices give details about network security and user activity. Not logging security devices is equivalent to not fixing the final piece of a puzzle.

In this post, we outlined a generic set of log sources. However, the complexity of logging from many sources is a problem for most of our users, and observIQ is working to fix that. We have over 60 integrations with log sources at the time of this post, and this list is increasing. We work on these integrations based on popular requests. Feel free to pitch your log source request to our customer support team. In the next post, we would like you to join us in configuring these log sources for your observIQ account so that you can set it up for free!