SLO from Methodology to Practice: Part 2 – SLO Tool Selection¶

Service Level Objectives, or SLOs, are a key part of the Site Reliability Engineering toolkit. SLOs provide a framework for defining clear targets around application performance, ultimately helping teams deliver a consistent customer experience, balance feature development with platform stability, and improve communication with internal and external users.

SLOs and SLIs¶

As we saw in Part 1, an SLO sets a precise target for your SLIs – metrics that reflect the health and performance of your service. By managing your SLOs in Guance, you can seamlessly access monitoring data – including trace metrics from APM, custom metrics, synthetic data, and metrics generated from logs – to use as SLIs. For example, if you want to ensure that typical user requests are processed quickly, you could use the median latency of a service from APM as an SLI. You could then define an SLO as: "99% of all user requests, calculated per minute, will have a median latency of less than 250 milliseconds in any calendar month."

To accurately track how actual performance compares to your set targets, you need a way not only to monitor real-time performance (e.g., calculating median latency every 60 seconds and comparing it to the 250 ms threshold) but also to measure how often that threshold is violated over a longer time span (to ensure the 99% target is met each calendar month). Guance tracks your SLIs and visualizes their status relative to the SLOs you have established, so you can immediately see how actual performance stacks up against your targets over a given period.

Tool Selection¶

For SLI metrics, the data primarily comes from three pillars: Metrics, Logs, and Traces.

Metrics: Indicators that collect system and application metrics. These have a significant impact on your service – for example, a sudden CPU spike of 100% can drastically reduce application processing capacity, causing severe lag for users.
Traces: Distributed tracing, based on a directed acyclic graph, that shows the direct invocation relationships between software modules. By examining service call chains, you can quickly identify which services or statements have a major impact on customers, such as slow database queries.
Logs: Time-related records output by systems or applications, typically generated by system/application developers to facilitate troubleshooting of errors and status. Extract relevant data from logs as Tags or Fields – Tags are useful for aggregation and grouping, while Fields are suitable for calculations and aggregations.

Common Tools

Metrics
Prometheus, OpenTelemetry...
Traces
SkyWalking, OpenTelemetry, Jaeger, Sleuth, Zipkin...
Logs
ELK, Filebeat, Flume...

There are many free and open-source tools, each with its own console and syntax. These tools come with high learning costs and expensive maintenance efforts. Fragmented systems and user systems force you to constantly switch between multiple platforms. SLIs precisely require a platform that can centralize components from various systems. A single platform that can integrate multiple tools and unify syntax can effectively reduce learning costs. Guance is the best choice. Guance provides a one-stop solution currently integrating over 200 open-source tools and components. Of course, you could build a team to construct such a platform, but it is not recommended.

Manage All Your SLOs in One Place¶

If your organization is committed to various SLOs across multiple products and teams, visualizing the status of all SLOs in one place helps you set priorities and resolve issues. Guance's Service Level Objectives view allows you to see the status of all SLOs and the remaining error budget for each SLO.

Create SLOs from Metrics and Monitors¶

In the SLO list view, you can create an SLO by clicking the Create SLO button. SLOs in Guance can be based on existing monitors (e.g., a monitor comparing p90 latency to a target threshold) or on live status calculated from metrics. Metric-based SLOs can be used to monitor the percentage of metrics that meet a specific definition, such as the number of non-5xx responses from a load balancer divided by the total number of responses.

View Your SLOs and Error Budget at a Glance¶

Clicking on an SLO or opening an event opens a side panel showing the details of the SLO, such as its status, target value, and remaining error budget. Guance automatically generates an error budget for each SLO, which represents the amount of unreliability you can tolerate before violating the SLO. This helps you quickly understand whether you are meeting your targets and whether your development velocity is appropriate for your established performance and stability goals. Guance automatically calculates the error budget based on the SLO target and time window you specify. For example, a 99% SLO target over a 7-day period gives you approximately three and a half hours of error budget for that period when performance is below standard.

Visualize SLO Status¶

To track the status of your SLOs in context with detailed data about related services or infrastructure components, you can add an SLO widget to your Guance dashboard. You can then share your dashboards internally or externally to communicate the real-time status of your SLOs to anyone who depends on your services.

You can also visualize how often the threshold is violated using common SLO baselines, such as last week, last month, week-to-date, or month-to-date. If you set a target of 99% over the past 30 days and a warning target of 99.5%, your SLO status will be green above 99.5%, yellow below 99.5%, and red below 99%.

Guance makes it easy to monitor and manage your SLOs in the same place where you already monitor your applications, infrastructure, user experience, and more. Perhaps equally important, Guance enables you to provide transparency to any stakeholders or users who depend on meeting these SLOs. If you aren't already using Guance to monitor the health and performance of your services, start with a free trial account.

Continue to the next and final part of this series, where we share best practices for getting the most out of your SLOs in Guance.