The Observability Blog

Categories:
  • Uncategorized

New Solutions to New Observability Needs

Paul Stefanski headshot
by Paul Stefanski on
August 13, 2021

Brief History

 

“Observability,” is the process in DataOps of recording data generated by digital systems as they go about their processes. There are some great companies in the observability space, generating a whopping $17 billion annually, and contributing a significant portion to the modest 2.5 quintillion bytes of data created every year. Observability became a point of interest in the late nineties during the tech boom when computers were landing, for the first time, on every desk in the country (and later, the world) and the U.S. economy shifted every industry it could to a digital infrastructure. Information moved fast – faster than ever thought possible, and though that led to an explosion of new opportunities and growth, it also opened the door to new problems and risks. 

Observability emerged as an industry when companies with large, complex digital infrastructures needed to collect data from their systems to bolster security, efficiency, or maintain compliance with government regulations. The earliest players, Solarwinds and Splunk, founded in 1999 and 2003 respectively, built the first enterprise level observability tools that became industry standard for fortune 500 tech companies. Their solutions, while robust, are less accessible to companies without deep technical expertise and even deeper pockets. A number of smaller, but successful, companies popped up between 2008 and 2015, capturing medium-sized companies and some small, tech-oriented companies. 

In addition to new observability opportunities born from inefficiencies that have developed over the last twenty years among the big players as the tech industry evolved, there are still at least $5 billion worth of underserved observability needs. As more technical infrastructure is built, the underserved segment grows. What exactly is creating the void and how can it be filled?

 

 

Human Effort

 

For many companies today, observability requires a dedicated team. Remember, the three main reasons any organization wants observability is for security, insight for development and optimization, and compliance. Security and compliance are significant sources of frustration for companies in spaces that require them, and the technical expertise needed to maintain proper security and compliance in a digital environment is often a barrier to entry for firms that would otherwise enter the market with phenomenal services (especially in the medical industry, where any company that wants to sell a product needs to guarantee HIPAA compliance). Strict security and compliance rules are good for consumers, and other than the recent, disastrous Solarwinds hack, observability firms have kept a solid guard. There is no sense in lobbying to reduce regulation or security standards, but it’s obvious that if observability were more technically accessible so that less technical companies could meet observability requirements easily, more firms in more industries would be able to enter the market with new innovations.

In the second case, insight for development and optimization, the issue is obvious. Most development teams are small and thrive on agility. It’s not that they don’t have the technical expertise to implement valuable observability solutions, it’s that they do not have the human capital needed to make that endeavor efficient. In both cases, there is a glaring lack of a plug and play observability solution that meets demand without the hassle.

 

Towering Costs

 

Existing observability solutions cost a lot. The pricing structures of the leaders, Datadog and Splunk, are convoluted at best. They advertise pricing “per host per month, billed annually” which seems modest at $23 and $110 respectively, but in reality even a small company is an outlier if they come in with bills under $50,000 per year. The cost is so exorbitant that many log management companies base their entire marketing strategy around undercutting Splunk. When a small team is trying to build out a new product, that money is much better spent elsewhere, even if it causes inefficiencies or security risks during development. Many development teams just have to live with low visibility into their applications. 

While other solutions are cheaper at small scales, they do not price well at large scales. Most companies plan to grow. So while it’s nice to have an observability solution that’s cheap when your needs are modest, it’s not worth the cost of implementing them when it will break the bank as the company succeeds. It’s directly contrary to the purpose of observability. Small teams need solutions that are efficient and affordable while capable of scaling with them as they expand.

 

Growing Markets

 

The observability space is changing. For the past two decades, observability has been in its “wild west” phase. Great companies emerged that jumped on the opportunities created by the tech boom, and plenty of other players arrived over time to capitalize on the niches that the early movers couldn’t fit, but that era has come to an end. Observability is now a standard, must-have for companies of all sizes in and outside of tech. The solutions that exist aren’t ready to service the new wave of demand, and it’s growing faster than they can adapt. The observability space needs something new.

 

A New Vision

 

The next generation of observability is seamlessly integrated with the development workflow, requires very little attention or expertise to create value, and comes at a fraction of the cost. Industry experts, who can move faster as individuals than larger companies, have started addressing this new space with cooperative, open source projects like OpenTelemetry. OpenTelemetry solves a lot of problems. It’s accessible. It’s free. It sets brilliant industry standards. It isn’t easy to take advantage of for teams with no dedicated DevOps engineer, and isn’t easy to switch to for teams using any existing solutions. 

observIQ is the company that built out OpenTelemetry’s logging solution. It’s also the first company to master this new frontier, bringing industry standard, performance exceeding, affordable solutions directly to users through a simple cloud platform. There are templatized agents for nearly any stack and use case that install in seconds and ingest logs in real time. The best part is it’s free for small businesses and undercuts enterprise leaders by several orders of magnitude. It’s never been easier to start logging today.