Our take on New Relic’s observability projections for 2021
New Relic recently published a survey on the observability projections and trends for the year 2021. At observIQ, an emerging platform of choice in the observability space, we decided to give our users a rundown on what we agreed and disagreed within their survey results.
Push to implement observability:
The survey presents a convincing argument that organizations need to implement observability solutions earlier than they currently tend to. Logs help DevOps teams understand the system’s behavior through the system’s outputs and log management has been common practice since the outset of terminals and highly hierarchical computing language. In the early 2010s, companies released application libraries that are installed within the application to track the application’s performance and events. This led to observability as we know it now. The difference between application monitoring and observability is cardinality. Cardinality refers to the ability to establish unique identifiers to a data set. For instance, in a database of travelers on a flight, the highest cardinality would be their passport number followed by the first name and last name combination. With high cardinality, modern observability facilitators make debugging and system performance analysis exponentially easier. This could be the reason why more and more organizations and individuals are seeing the true value in implementing log management for their businesses.
Effective monitoring of containerized applications is the need of the hour:
The survey reiterated that observability for Kubernetes and containerized applications is trending towards becoming a necessity. This is a no-brainer. At observIQ, we predicted this and built one of the simplest ways to plugin our agent into Kubernetes to gain visibility of all events. It is good that New Relic factored this because companies like Splunk and honeycomb.io did not include any stats related to Kubernetes adoption among businesses and the increasing need for observability among Kubernetes applications. As more and more organizations embrace containerized and micro-services-based architecture models for their applications and infrastructure, incorporating observability into their new applications is easier.
Observability awareness is on the rise.
Earlier definitions of observability referred to the three pillars of observability, metrics, traces and logs, very theoretically. But any DevOps engineer following best practices would point out that metrics, traces and logs are merely three sets of observability data. The true sense of a functioning observability practice is using these sets of data efficiently within a tool that can parse, enrich, and visualize the data.
There is no perfect observability method, probably there never will be.
With distributed, containerized, and cloud applications, everything is transitory. Platforms such as OpenTelemetry have spelled out what is always known. As software systems change, processes for managing those systems are compelled to keep up with those changes. Hence, a prebuilt model that fits all businesses for the next 5 years does not exist. Observability is a space that will evolve as it needs to. For businesses to keep up with this evolution, there must be a common ground formed based on processes. As per the survey, the general consensus is that there is an immense opportunity to streamline and mature the observability practice, which is an agreeable statement.
Organizations are ready to invest higher in observability.
This brings up a seldom-talked-about question. Is an expensive solution really better? We started offering a free tier starting this June. We did this to offer our platform to users who don’t have an exhaustive list of items in their observability wishlist. Our free tier offers short-term log data storage while maintaining your access to all existing and new features of the system. Many tools in the market offer users observability that is often tied down by the cost. Companies are limited by the cost when implementing a detailed observability practice. Instead of pushing for organizations to invest more in observability, the best way to make organizations embark on their observability journey would be to offer meaningful pricing. The verbiage in the survey refers to the C-suite executives making higher budgetary allocations, and it does not factor other user personas in the budgetary study. This is an evident gap in analysis. This disparity extends to the survey’s data breakup of the type of subscription/licensing models. Without having an inclusive user persona, these observations speak only to one specific category of users.
Lack of engineers with Implementation skills is a dealbreaker in adaptability.
We agree with this statement. Companies that are still in the code library-based application monitoring practice find the code level retracing and modifying very intimidating. Although some businesses see that the pros of this transition outweigh the cons, the fear of issues that may arise during implementation in a functioning system stops businesses from pursuing modern-day observability solutions. Solutions like observIQ can help users overcome the need to fix or repatch. Instead, the agents are installed onto the application stacks via preconfigured plugins. If you would like to explore more about this, speak to our implementation support team.
Dev teams agree that observability should be implemented across the board
We have written in the past about how observability is now an integral part of every phase in the SDLC. The survey confirms this with a majority of development teams believing that they need observability across the board in every phase of application development, quality assurance, implementation, and maintenance. An interesting result observed from the survey’s results is that users in the APAC region felt more compelled to incorporate observability through the SDLC than their North American counterparts.
Simple observability platforms to promote faster implementation
The survey results tied adaptability to the skill level of an implementer. We have a different take on this. Instead of scaling up the implementation abilities of observability users, we believe it is more efficient to build simpler observability platforms. Platforms such as obervIQ, have a growing list of pre-built log sources. This is conducive to making organizations with smaller teams implement observability. With large, complicated observability platforms, evaluating engineers are thrown off by the complexity of the application.
Wider user personas for observability
The sampling size, occupational diversity, and geographic spread of responders used in the survey were good. However, the sample personas used seemed very old school. Observability is no longer a restrictive space. There is an identified observability need from businesses, public entities, professional gamers, independent contractors, homeowners using IoT for home security, technical influencers – the list of possible user personas is endless. So having a restrictive sample pool of individuals working in similar work streams makes this survey extremely unidirectional.
Fragmented monitoring is still common practice.
One-third of the users are still working with their observability data between multiple systems. With their analysis data housed in different systems, most developers must mend together siloed data manually to debug or gauge the system’s performance.
In conclusion, this survey highlights what is known for the past few years. Observability is vertical in the tech space that is gaining momentum. The earlier the adoption, the better for organizations of all sizes. This is also the era of cloud-native application emergence. Our recommendation would be for organizations to incorporate observability as part of their cloud-native migration.