Archiving Is In, And Your Logs Are Here To Stay!

Archiving is in, and your logs are here to stay!

We develop features that streamline the log management processes for our users. Logs are information assets, and we understand that you need to retrieve, re-assess, and draw insights from your historical logs. observIQ offers a simple integration with Amazon Web Services (AWS) for extended retention. It takes less than 30 seconds to set up and archive logs directly to an S3 bucket in your AWS account. Once compressed and uploaded from observIQ, your logs are retained indefinitely – as long as your cloud bucket exits. Your data in the S3 bucket is backed up in a SOC 2-compliant data center.

On the topic of archiving, we want to highlight some scenarios where you may turn to your S3 bucket to pull up historic logs:

1. Audits
2. Analyzing security status
3. Business Insights

Audits and Compliance

Extended log retention is critical for businesses with strict compliance and regulatory requirements, but that doesn’t mean it should be a hassle. The retention period and the type of logs retained vary based on the business’s industry standards. Creating an audit trail for every event in the network or applications is mandatory for common compliance standards such as HIPAA and PCI. Most mandated retention standards lean toward the one-year time frame. However, some businesses retain logs for extended periods to err on the side of caution in case their compliance certification norms change.

Every action in the cyber realm generates logs. Businesses choose the log data that are most necessary for their compliance. Common log types that are retained for compliance are:

1. Network and application access credentials include user ID, access locations, user information, time and date of access, terminal access identification, etc.
2. Changes to application and infrastructure such as containerized pod additions and deletions, activating and deactivating firewalls, malware, etc.
3. Changes the admin makes to the log data or log management tool. This could be importing log files, deleting a batch of logs, or changing the log management tool.

Analyzing Security Status

Cybersecurity is a top priority for all businesses, even outside the tech industry. Logs give you an unobstructed view of your application and infrastructure’s health and security. Cybersecurity auditors want to read through present and past logs to analyze security performance over time, identify potential vulnerabilities, and formulate process changes to tighten security. This is also tied to the compliance aspect of log retention. Before applying for a compliance certification, most companies employ legal and cybersecurity counsel to assess their security and infrastructure and make recommendations to navigate the certification process. A business’s logs are the first resource an independent auditor wants to examine.

Business Insights

“Data-driven decisions” is a pervasive buzz phrase in today's business world. Some of your most critical data lives in your logs. Generating deep, actionable insights from data requires deep, accurate, and organized data infrastructures. Retaining logs over long periods gives you visibility into the trends, usage, performance, and security of your applications and infrastructures. Without extended retention, formulating data-driven decisions around development, security, and user experience is impossible.

Businesses pull up historic logs for trend analysis. Metrics such as response time, response size, traffic source, volume of traffic, pod status, etc., are studied over time to chart a performance pattern, and decision-making ensues.

The Archiving Process in observIQ

The extended retention features in observIQ are available to all users, including users on the free tier. Free extended retention can go a long way for any business, and anyone with an S3 bucket can start archiving their logs with observIQ today!

1 Configure an AWS S3 Bucket

Within AWS:

1. Access your S3 Console
2. Use an existing bucket or create a new one.

2 Enable Archiving in observIQ

You should be signed in as an Owner or Admin to access the archive functionality.

1. Head to Settings → Archive
2. Enable archiving.
3. Select the region; the region selection in observIQ and AWS S3 should be the same.
4. Enter the name of the S3 bucket where the logs will be archived. Your logs are saved as zipped files that are time-stamped with the date and time of export.
5. You can enter your AWS account's access ID and secret access key.

Your logs are saved in the S3 bucket periodically. They are saved in batches, each enclosed within a zipped file. The zipped files are named with a timestamp. We ensure that all your logs are saved to the S3 bucket before the end of the retention period.

The list below the extended retention setup displays all the zipped files saved to the linked S3 bucket. Clicking the filename navigates you to the file within AWS.

observIQ makes extended retention a painless process for you and your team. Try out our log management platform today. Every feature is included in every plan, even the free tier. Please contact our support team with any questions, and we can talk about how observIQ can best fit your log management needs.