New Solutions to New Observability Needs

Brief History

“Observability” is the process in DataOps of recording data generated by digital systems as they go about their processes. Some great companies are in the observability space, generating a whopping $17 billion annually and contributing a significant portion to the modest 2.5 quintillion bytes of data created yearly. Observability became a point of interest in the late nineties during the tech boom when computers were landing, for the first time, on every desk in the country (and later, the world), and the U.S. economy shifted every industry it could to digital infrastructure. Information moved faster than ever thought possible, and though that led to an explosion of new opportunities and growth, it also opened the door to new problems and risks.

Observability emerged as an industry when companies with large, complex digital infrastructures needed to collect data from their systems to bolster security and efficiency or maintain compliance with government regulations. The earliest players, Solarwinds and Splunk, founded in 1999 and 2003, built the first enterprise-level observability tools that became the industry standard for Fortune 500 tech companies. Their solutions, while robust, are less accessible to companies without deep technical expertise and even deeper pockets. Several smaller but successful companies popped up between 2008 and 2015, capturing medium-sized companies and some small, tech-oriented companies.

In addition to new observability opportunities born from inefficiencies that have developed over the last twenty years among the big players as the tech industry evolved, there are still at least $5 billion worth of underserved observability needs. As more technical infrastructure is built, the underserved segment grows. What exactly is creating the void, and how can it be filled?

Related Content: When Two Worlds Collide: AI and Observability Pipelines

Human Effort

For many companies today, observability requires a dedicated team. Remember, the three main reasons any organization wants observability are security, insight for development and optimization, and compliance. Security and compliance are significant sources of frustration for companies in spaces that require them. The technical expertise needed to maintain proper security and compliance in a digital environment is often a barrier to entry for firms that would otherwise enter the market with phenomenal services (especially in the medical industry, where any company that wants to sell a product needs to guarantee HIPAA compliance). Strict security and compliance rules are good for consumers, and other than the recent disastrous Solarwinds hack, observability firms have kept a solid guard. There must be more sense in lobbying to reduce regulation or security standards. Still, it’s evident that if observability were more technically accessible so that less technical companies could meet observability requirements quickly, more firms in more industries would be able to enter the market with innovations.

The issue is evident in the second case, which provides insight for development and optimization. Most development teams are small and thrive on agility. It’s not that they don’t have the technical expertise to implement valuable observability solutions; they do not have the human capital needed to make that endeavor efficient. In both cases, there is a glaring lack of a plug-and-play observability solution that meets demand without the hassle.

Towering Costs

Existing observability solutions cost a lot. The pricing structures of the leaders, Datadog and Splunk, are convoluted at best. They advertise pricing “per host per month, billed annually,” which seems modest at $23 and $110, respectively, but in reality, even a small company is an outlier if they come in with bills under $50,000 per year. The cost is so exorbitant that many log management companies base their marketing strategy around undercutting Splunk. When a small team is trying to build a new product, that money is much better spent elsewhere, even if it causes inefficiencies or security risks during development. Many development teams have to live with low visibility into their applications.

While other solutions are cheaper on small scales, they do not price well on large scales. Most companies plan to grow. So, while it’s nice to have a cheap observability solution when your needs are modest, it’s not worth implementing them when it will break the bank as the company succeeds. It’s directly contrary to the purpose of observability. Small teams need solutions that are efficient and affordable while capable of scaling with them as they expand.

Growing Markets

The observability space is changing. Observability has been in its “Wild West” phase for two decades. Great companies emerged that jumped on the opportunities created by the tech boom, and plenty of other players arrived over time to capitalize on the niches that the early movers couldn’t fit. Still, that era has come to an end. Observability is now a standard for companies of all sizes in and outside tech. The existing solutions aren’t ready to service the new wave of demand, and it’s growing faster than they can adapt. The observability space needs something new.

Related Content: Splashing into Data Lakes: The Reservoir of Observability

A New Vision

The next generation of observability is seamlessly integrated with the development workflow, requires very little attention or expertise to create value, and comes at a fraction of the cost. Industry experts, who can move faster as individuals than larger companies have started addressing this new space with cooperative, open-source projects like OpenTelemetry. OpenTelemetry solves a lot of problems. It’s accessible. It’s free. It sets brilliant industry standards. It isn’t easy to take advantage of for teams with no dedicated DevOps engineer and isn’t easy to switch to for teams using any existing solutions.

observIQ is the company that built out OpenTelemetry’s logging solution. It’s also the first company to master this new frontier, directly bringing industry-standard, performance-exceeding, affordable solutions to users through a simple cloud platform. Templatized agents for nearly any stack and use case install in seconds and ingest logs in real-time. The best part is it’s free for small businesses and undercuts enterprise leaders by several orders of magnitude. It’s never been easier to start logging today.