This article originally appeared on TheNewStack
IT performance monitoring is undergoing a major transformation thanks to advances in artificial intelligence and real-time analytics. Many IT monitoring platforms promise new capabilities that will move them beyond monitoring to observability.
But before we get too far, let’s first define the goal of observability. Systems engineer and author Cindy Sridharan offers one of the clearest definitions to date in an essay on Medium. She describes how observability, a superset of monitoring, combines alerting/visualization, distributed systems tracing infrastructure and log aggregation/analytics to provide better visibility into IT systems health.
In order to achieve such a lofty goal, the industry must shift operational data collection from a focus on flat metrics to what we call “dimensional data.” Achieving observability starts with internal instrumentation and external data collection, the output of which is then passed to various analytical or graphing engines within a monitoring platform.
Why Current Metrics Fall “Flat”
To understand the need for dimensional data, it’s important to first understand the challenges
associated with the data collection status quo, something we refer to as “flat metrics.” Flat metrics provide only surface-level analysis, without segmenting a technology into its various roles and components.
This was not the case a decade ago, when most environments could be integrated using a handful of scripts or APIs. But since that time, the number of APIs has exploded, hybrid and even multi-cloud environments have mainstreamed and newer technologies like containers and serverless have clouded relationship visibility. The very abstraction techniques that make them so user-friendly to DevOps make them more challenging to monitor and analyze.
Simply put, it’s no longer scalable for an organization to be able to manually consider how relationships between different layers in their IT stack may be impacting alerts, performance problems and outages. Without the context of the complex relationships in the modern IT environment, flat metrics can create false-positives in monitoring platforms.
Key Elements of Dimensional Data for Observability
Dimensional data refers to the stream of information a next-generation monitoring integration strategy delivers, such as monitoring-integration-as-a-service (MIaaS). MIaaS is an emerging set of tools for connecting monitoring platforms that are deployed in different IT environments, so they can analyze the health and performance of the entire ecosystem. MIaaS is often used by large business-to-business enterprises that need to integrate on-premise infrastructure performance metrics with cloud service-performance metrics or in multicloud environments.
A dimensional data stream will include highly granular behavioral detail — beyond what a single endpoint API connection might include — as well as rich relational context. The term dimensional data is specific to IT system health and performance information and should not be confused with dimensional models used in data warehousing or low- and high-dimensional data sets in business analytics.
In contrast to flat metrics, dimensional data combines the standardization, relational visibility and super metrics that move performance monitoring and analytics platforms closer to observability.
Some traditional approaches to monitoring integration may cover one or two dimensions of data — but the only way to access all four dimensions of operational data is with a MIaaS.
Here are the four elements of dimensional data that observability requires:
The Path to Observability Requires Dimensional Data
Monitoring purists such as Sridharan — cited above — might argue the best route to observability is to look for a way to bring together insights from your monitoring, log analytics, tracing and other tools. That approach works best when each of the tools draws from the same well of information, like a dimensional data stream.
However, in practice, some organizations try to achieve observability by expanding the capabilities of their existing monitoring tools to take on more observability-like functions. Indeed, many monitoring, log analytics and tracing tools are designed as an attempt to become a single platform for observability.
Organizations also tend to deploy multiple monitoring tools. Many of the platforms are best-in-breed at certain functions such as APM, log analytics and tracing, which is why teams often deploy more than one based on the use case.
While it is unlikely any platform will become the single source for observability, dimensional data deployed through a MIaaS can offer any platform the highly granular intelligence and the relational context they need to achieve observability.