Key Takeaway
OpenTelemetry Collector pipelines need explicit processor ordering, not just chaining components: memory_limiter must run first to protect the process, volume-reducing steps like filter and tail_sampling must precede batch, and splitting agent and gateway deployments keeps tail-based sampling accurate while adding backends becomes a configuration change instead of a redeploy.
Most teams start their OpenTelemetry journey by pointing an SDK straight at a backend. That works for a demo, but it does not survive contact with production traffic, cost limits, or a second observability vendor. The fix is to put a Collector in the middle and treat telemetry as its own pipeline, with the same rigor you would apply to a data pipeline that feeds a warehouse.
This post walks through designing that pipeline end to end: how receivers, processors, and exporters compose, why processor order matters more than most teams realize, how tail-based sampling keeps the traces you actually care about, and how to fan the same stream out to more than one backend without duplicating instrumentation.
The anatomy of a Collector pipeline
A Collector pipeline is a named path that a signal, such as traces, metrics, or logs, follows through three kinds of components declared in the configuration file.
- Receivers accept data into the Collector, either by listening on a network port for a push protocol like OTLP, or by actively scraping a source on a schedule.
- Processors sit between receivers and exporters, transforming, filtering, enriching, or sampling the data as it flows through. Each pipeline gets its own independent processor instance, even if two pipelines reference the same processor name.
- Exporters forward the processed data to one or more destinations. If a pipeline lists more than one exporter, every exporter receives its own copy of the data through a fan-out step at the end of the chain.
Data moves through the pipeline in one direction only, from every configured receiver into the first processor, then sequentially through each remaining processor, and finally out to every listed exporter. A processor can also decide to drop data outright, which is exactly how sampling and filtering are implemented.
receivers:
otlp:
protocols:
grpc:
http:
processors:
memory_limiter:
check_interval: 1s
limit_mib: 1500
batch:
send_batch_size: 8192
timeout: 5s
exporters:
otlphttp:
endpoint: https://backend.example.com:4318
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp]Give every processor and exporter a named instance, such as batch or otlphttp with a slash suffix, when a signal needs different handling per pipeline. A single unnamed processor name is shared conceptually but instantiated separately per pipeline, so naming makes the intent explicit in the config file.
Why processor order is the whole game
Processors execute strictly in the order you list them, and that order changes both correctness and cost. A resource-protection processor has to run before anything expensive, and anything that reduces data volume has to run before the processor that batches for network efficiency, otherwise you are batching data you are about to throw away.
| Order | Processor | Why it goes here |
|---|---|---|
| 1 | memory_limiter | Protects the Collector process itself from out-of-memory crashes by rejecting or shedding data once memory pressure crosses a configured threshold, before any heavier processing runs. |
| 2 | resource / attributes | Attaches or corrects service, environment, and deployment attributes early so every downstream decision, including sampling policies, can rely on accurate metadata. |
| 3 | filter | Drops telemetry that never needed to leave the source, such as health-check spans or debug-level logs, before the more expensive sampling and batching steps touch it. |
| 4 | tail_sampling | Makes the keep-or-drop decision on whole traces after enough spans have arrived to evaluate policies like error status or high latency. |
| 5 | batch | Groups whatever survived sampling into larger batches immediately before export, which is where batching delivers the most benefit because nothing downstream will touch the data again. |
Tail-based sampling: keeping the traces that matter
Head-based sampling decides whether to keep a trace at the moment the first span is created, before anyone knows if the request will be slow or fail. Tail-based sampling waits until a trace, or enough of it, has arrived, and only then applies rules such as always keep traces with an error status, always keep traces slower than a latency threshold, and otherwise keep a small percentage of everything else.
The tradeoff is memory and topology. The processor holds spans in a buffer until a decision window elapses, and it can only make a correct decision if every span belonging to a given trace lands on the same Collector instance. In a load-balanced fleet that means routing by trace identifier at an earlier layer, not by round robin.
- A decision wait window, commonly around thirty seconds, controls how long the processor accumulates spans for a trace before evaluating its sampling policies.
- A fixed-size in-memory buffer, sized by a maximum number of traces, holds trace state between arrival and decision. Older traces are evicted once the buffer fills, which can drop a trace before it is ever evaluated.
- A layered deployment pattern is the standard fix for the routing requirement: a first layer of stateless Collectors load-balances spans by trace identifier, and a second layer running the tail sampling processor makes the final keep-or-drop call per trace.
If your Collectors run behind a generic load balancer that distributes spans round robin instead of by trace identifier, tail-based sampling will silently make wrong decisions because spans from the same trace scatter across instances that never see the full picture. Fix the routing layer before tuning sampling policies.
Agent versus gateway deployment
There are two common places to run Collector instances, and most production setups use both together rather than picking one.
- An agent Collector runs as a sidecar or a per-node daemon set, close to the application. It handles cheap, local work: receiving OTLP from the SDK, adding host and pod resource attributes, and applying a memory limiter.
- A gateway Collector runs as a separate, centrally deployed service that agents forward to. It is the right place for anything that needs a global view or heavier processing, such as tail-based sampling, deduplication, or fanning out to multiple backends.
- Connectors, a component type that acts as an exporter for one pipeline and a receiver for another inside the same Collector, let you split traffic between the two layers without leaving the Collector configuration, for example routing a subset of spans into a dedicated pipeline for a specific backend.
This split keeps per-node resource usage predictable while concentrating stateful, memory-hungry work like tail sampling in a smaller number of horizontally scaled gateway instances that you can size independently.
Shipping to any backend without touching application code
Because the SDK only ever talks OTLP to the Collector, switching or adding a backend is a configuration change, not a redeploy of every instrumented service. Add a second named exporter and list it alongside the first in the pipeline, and the Collector fans the same processed data out to both destinations.
exporters:
otlphttp/tempo:
endpoint: https://tempo.example.com:4318
otlphttp/honeycomb:
endpoint: https://api.honeycomb.io
headers:
x-honeycomb-team: ${env:HONEYCOMB_API_KEY}
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, tail_sampling, batch]
exporters: [otlphttp/tempo, otlphttp/honeycomb]This is also the cleanest way to run a proof of concept against a new backend, such as an open-source tracing store or a paid vendor, in parallel with your existing production destination, with zero risk of losing existing data if the trial does not pan out.
Because exporters run after sampling and batching, adding a second backend costs you configuration lines, not double the trace volume you already paid to sample down. The fan-out happens on the already-reduced stream.
A design checklist before you ship
Before treating a Collector pipeline as production-ready, walk through this short list.
- Is memory_limiter the first processor in every pipeline, with a limit set below the container or host memory ceiling?
- Does anything that reduces volume, such as filter or tail_sampling, run before batch, so you never batch data you are about to discard?
- If tail-based sampling is in use, is span routing by trace identifier guaranteed at the load balancer, not left to chance?
- Is there a gateway layer absorbing stateful processing, so agent Collectors on application nodes stay lightweight and easy to size?