Published: September 7, 2026 · Last updated: September 7, 2026 · Author: Softix
Small SaaS teams often turn on OpenTelemetry everywhere—every service, every library, every environment—then get sticker shock when the trace backend bill or the Collector CPU graph arrives. You do not need 100% of healthy, low-latency traces to run the product. You need the right sample: enough baseline coverage to understand normal behavior, plus near-certainty that errors and latency outliers still reach your dashboards and on-call.
That is what OpenTelemetry’s sampling concepts are for. Softix’s frame for US SMB founders and product/ops leaders is Instrument–Sample–Alert: decide what deserves spans first, choose head vs tail sampling so you do not silently drop the traces that matter, then wire alerts to metrics and exemplars—not to “every span forever.”
This pairs with Softix’s SLO and error-budget operating model. Sampling is a cost and signal control; SLOs are the promise you still have to keep.
Head sampling vs tail sampling (from the OpenTelemetry docs)
OpenTelemetry uses precise language: a sampled span or trace is processed and exported; a not sampled one is not. “Sampling out” language that treats dropped data as “sampled” is incorrect per the official terminology.
Head sampling
Head sampling decides as early as possible—typically when a span starts—without inspecting the finished trace. The common form is consistent / deterministic probability sampling based on the trace ID and a desired percentage (for example, keep 5% of traces). Upsides called out in the docs: easy to understand, easy to configure, efficient, and usable anywhere in the pipeline. The primary downside: you cannot guarantee that every error or slow request is kept, because you have not seen the rest of the trace yet.
Tail sampling
Tail sampling waits until all or most spans in a trace are available, then decides using criteria across the whole trace. OpenTelemetry’s examples include always keeping traces with errors, sampling by overall latency, sampling by attributes (for example, a newly deployed service), and different rates for high- vs low-volume services. That sophistication is why larger systems that must sample almost always need tail sampling to balance volume against usefulness.
Tail sampling also has documented costs: harder to implement and operate (stateful buffers, sticky routing so all spans for a trace hit the same sampler), and many of the best managed options are vendor-specific. Teams sometimes combine both: light head sampling upstream to protect the pipeline, then richer tail policies before export.
Grafana Cloud’s sampling guide restates the same split for Alloy / Collector users: head sampling is low overhead but may miss rare errors and latency; tail sampling can select on outcome (status, duration, attributes).
When not to sample (and when you should)
OpenTelemetry is explicit that sampling is not always the right move. Avoid sampling when you generate very little data (tens of small traces per second or lower), when you only use observability in aggregate and can pre-aggregate, or when regulation (or contract) prohibits dropping data and you cannot park unsampled data in low-cost storage.
Consider sampling when you generate roughly 1,000+ traces per second, when most traffic is healthy with little variation, when you have clear “something is wrong” criteria (errors, high latency, domain rules), when you can treat high- and low-volume services differently, or when budget is tight but you can spend engineering time to sample well.
Also budget for three sampling costs the docs call out: compute for a tail-sampling path, ongoing engineering to keep policies honest as the system grows, and the opportunity cost of missing critical traces with bad rules. Sometimes buying more backend capacity is cheaper than a half-maintained sampler.
For a five-person SaaS still under tens of traces per second in production, Softix’s default advice is: instrument thoughtfully first; do not invent a sampling program to feel “enterprise.” Turn sampling on when volume or cost forces the trade-off—or when you need tail policies to keep all errors while thinning success traffic.
Softix Instrument–Sample–Alert
1) Instrument — what to span first
Sampling cannot rescue a noisy or empty instrumentation set. Before you pick percentages, lock a thin, high-value span surface:
- Edge and money paths — public HTTP/API gateway, auth, checkout/billing, and the one write path that defines your SLO.
- Dependency boundaries — outbound calls to Postgres, Redis, queues, and critical SaaS APIs (with stable
service.nameand route/operation names). - Release markers — deployment version / feature-flag evaluation on spans so you can correlate with Softix’s feature-flag release strategy during canaries.
- Skip vanity depth — avoid auto-instrumenting every internal helper and every batch job “because OTel supports it.” More spans raise cost and make sampling policies harder to reason about.
Keep metrics (RED / latency histograms) as the always-on reliability signal. Traces explain why; they should not be your only heartbeat. That split matches how Softix talks about reliability in the SLO / error-budget guide.
2) Sample — keep errors and outliers without keeping everything
A practical SMB progression:
| Stage | Approach | Use when |
|---|---|---|
| A | No sampling (or 100% in staging only) | Low volume; still learning attributes and service graph |
| B | Head / probabilistic baseline | Cost rising; healthy traffic dominates; you accept missing some rare events |
| C | Tail policies: errors + latency + probabilistic floor | You need “keep failures and slow traces” while thinning OK traffic |
| D | Managed tail (e.g. Grafana Cloud Adaptive Traces) | You want tail power without running a sticky, stateful sampling fleet |
Collector / Alloy path. The OpenTelemetry Collector documents a probabilistic sampling processor (per-item / TraceID-based, with consistency properties) and a tail sampling processor with policies such as status_code, latency, probabilistic, string_attribute, and / composite, and drop. Grafana documents configuring the same strategies with Grafana Alloy or the Collector, and points to Alloy’s tail-sampling docs alongside the contrib processor.
Policy pattern Softix recommends for small production SaaS (conceptual):
- Always sample traces with
ERRORstatus (and/or HTTP 5xx attributes). - Sample traces above a latency threshold tied to your SLO (for example, above the latency objective you already use for burn alerts—not a random round number).
- Keep a small probabilistic baseline (single-digit percent is common for high volume; OpenTelemetry notes that for very high volume, 1% or lower can still represent the population mathematically—verify against your traffic, do not cargo-cult a number).
- Optionally boost sample rates for a canary version attribute or a high-value tenant ID during releases.
All spans for a given trace_id must reach the same tail-sampling instance—the contrib processor warns about statefulness and sticky routing. That operational tax is why many SMBs eventually prefer a managed option.
Optional managed tail: Grafana Cloud Adaptive Traces. Grafana’s sampling strategies tip and Adaptive Traces introduction describe Adaptive Traces as managed tail sampling: policies decide keep/drop after considering the trace; policy types include probabilistic, latency, status code, string attribute, and drop; policies are OR’d by default (match any → keep). Adaptive Traces documents a two-stage wait (about 2 seconds after a root span, up to 30 seconds if no root arrives), recommendations that suggest error / slow / percentage policies, and extras such as diversity sampling and anomaly-triggered temporary policies. Treat product packaging and pricing as something to verify in your Grafana Cloud tenant—Softix is not claiming a universal bill outcome.
3) Alert — page on promises, debug with traces
Do not page humans on “span count dropped 12% after we enabled sampling.” Page on SLO burn, error-rate, and latency objectives—the same discipline as Softix’s error-budget model. Use traces for investigation:
- Prefer metrics with exemplars (or “example trace” links) so an alert opens a relevant sampled trace.
- Alert when the sampling pipeline itself is unhealthy (Collector/Alloy queue full, tail-sampler drops pending traces, Adaptive Traces / exporter errors)—those are reliability of the observability system, not of the product.
- After a release gated by feature flags, temporarily raise sample rates or add a string-attribute policy for the flag/version key—with an expiration—so you keep forensic depth without permanently paying for 100% forever.
If Kubernetes cost is already under Softix’s Measure–Match–Limit discipline, treat trace ingest the same way: measure volume by service, match retention to value, and set hard ceilings before the invoice surprises finance.
A 30-day SMB path
Week 1 — Instrument. Freeze a service inventory. Enable OTel on the edge + money path + top three dependencies only. Standardize service.name, deployment version, and HTTP route/operation. Ship to staging at 100% for a few days to validate attributes.
Week 2 — Observe cost and shape. Export to your managed backend (Grafana Cloud Tempo or equivalent) with Alloy or the Collector as the only hop. Chart spans/traces per service. Decide whether you are in “when not to sample” territory.
Week 3 — Sample. If volume warrants it, add either (a) probabilistic head/Collector sampling for a baseline, or (b) tail policies: errors + latency + probabilistic floor. Prefer Adaptive Traces if you are already on Grafana Cloud and do not want to operate sticky collectors. Document the policies in the same repo as your Alloy/Collector config.
Week 4 — Alert. Connect burn alerts to SLOs; attach exemplars. Add pipeline-health alerts. Tabletop one incident: confirm an injected 5xx still appears under the sampling policy. Retire vanity instrumentation that never helps debugging.
Gateway and traffic-routing changes (see Softix’s Gateway API migration guide) are a good moment to re-check edge span names so sampling policies keyed on routes stay accurate.
Limits and honesty
- Softix does not invent customer savings percentages or “typical” ingest reductions. Your reduction depends on traffic shape and policy design.
- Head sampling alone will miss some errors—by design. If that is unacceptable, you need tail sampling or 100% keep for error classes.
- Tail sampling done poorly can be more expensive than retaining more traces. Monitor the sampler.
- Vendor Adaptive Telemetry features evolve; confirm current Adaptive Traces behavior and entitlements in official Grafana docs for your stack.
FAQ
Is OpenTelemetry sampling only for huge enterprises?
No. OpenTelemetry’s own “when not to sample” guidance is aimed at low-volume systems. Small SaaS teams should instrument carefully first, then sample when cost or volume demands it—using the same head/tail concepts.
Will 5% head sampling keep all checkout errors?
Not reliably. Head sampling cannot see the finished trace. Use tail policies (or managed Adaptive Traces status/latency policies) if you need outcome-based keep rules.
Collector or Alloy?
Both appear in Grafana’s configuration references for sampling. Choose the agent you already operate; Softix cares more about sticky routing for tail sampling and clear policies than about brand loyalty.
Should sampling replace SLOs?
No. Sampling controls trace volume. SLOs and error budgets define the customer promise—see Softix’s SLO guide.
Build the pipeline once—then keep the traces that matter
Instrument–Sample–Alert is how small product teams get OpenTelemetry value without treating “100% forever” as a virtue. Softix helps US SMBs design SaaS platforms, collectors, and release workflows so observability stays operable as traffic grows—via custom software development.
Contact Softix with: (1) approximate traces/second in production, (2) whether you already run Alloy or the OTel Collector, and (3) which two user journeys define your latency SLO. We will not invent a sampling percentage or guarantee a backend invoice.
Share
