DevOps
OpenTelemetry Collector: Designing a Telemetry Pipeline
July 202610 min read
Photo by woodleywonderworks on flickr
A Collector pipeline is a named path that a signal such as traces, metrics, or logs follows through three component types: receivers accept data, processors transform or sample it, and exporters send it to a backend. Data flows one way, from every receiver through each processor in order and finally out to every exporter. Pipelines are defined per signal in the service section of the config file.
Processors run strictly in the order you list them, so ordering affects both correctness and cost. The memory_limiter should run first to protect the process from out-of-memory crashes, and volume-reducing steps like filter and tail_sampling should run before batch. Otherwise you spend CPU and memory batching data you are about to discard.
Head-based sampling decides whether to keep a trace when the first span is created, before you know if the request was slow or failed. Tail-based sampling waits until the trace is complete, then applies policies such as keeping all errors and all slow requests. Tail sampling captures the traces you actually need to debug, at the cost of buffering spans in memory.
The tail sampling processor evaluates policies over a whole trace, so it needs to see every span of that trace on one instance to decide correctly. If a generic load balancer scatters spans round robin, each instance sees only a fragment and makes wrong decisions. The fix is a layered setup that routes by trace identifier to a dedicated sampling tier.
Because the SDK only speaks OTLP to the Collector, you add a second named exporter and list both exporters in the pipeline. The Collector fans the already-processed data out so each exporter gets its own copy. This makes it easy to trial a new backend in parallel with your existing one, and switching is a config change rather than an application redeploy.
Photo by woodleywonderworks on flickr
Key Takeaway
OpenTelemetry Collector pipelines need explicit processor ordering, not just chaining components: memory_limiter must run first to protect the process, volume-reducing steps like filter and tail_sampling must precede batch, and splitting agent and gateway deployments keeps tail-based sampling accurate while adding backends becomes a configuration change instead of a redeploy.
Most teams start their OpenTelemetry journey by pointing an SDK straight at a backend. That works for a demo, but it does not survive contact with production traffic, cost limits, or a second observability vendor. The fix is to put a Collector in the middle and treat telemetry as its own pipeline, with the same rigor you would apply to a data pipeline that feeds a warehouse.
This post walks through designing that pipeline end to end: how receivers, processors, and exporters compose, why processor order matters more than most teams realize, how tail-based sampling keeps the traces you actually care about, and how to fan the same stream out to more than one backend without duplicating instrumentation.
A Collector pipeline is a named path that a signal, such as traces, metrics, or logs, follows through three kinds of components declared in the configuration file.
Data moves through the pipeline in one direction only, from every configured receiver into the first processor, then sequentially through each remaining processor, and finally out to every listed exporter. A processor can also decide to drop data outright, which is exactly how sampling and filtering are implemented.
receivers:
otlp:
protocols:
grpc:
http:
processors:
memory_limiter:
check_interval: 1s
limit_mib: 1500
batch:
send_batch_size: 8192
timeout: 5s
exporters:
otlphttp:
endpoint: https://backend.example.com:4318
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp]Give every processor and exporter a named instance, such as batch or otlphttp with a slash suffix, when a signal needs different handling per pipeline. A single unnamed processor name is shared conceptually but instantiated separately per pipeline, so naming makes the intent explicit in the config file.
Processors execute strictly in the order you list them, and that order changes both correctness and cost. A resource-protection processor has to run before anything expensive, and anything that reduces data volume has to run before the processor that batches for network efficiency, otherwise you are batching data you are about to throw away.
| Order | Processor | Why it goes here |
|---|---|---|
| 1 | memory_limiter | Protects the Collector process itself from out-of-memory crashes by rejecting or shedding data once memory pressure crosses a configured threshold, before any heavier processing runs. |
| 2 | resource / attributes | Attaches or corrects service, environment, and deployment attributes early so every downstream decision, including sampling policies, can rely on accurate metadata. |
| 3 | filter | Drops telemetry that never needed to leave the source, such as health-check spans or debug-level logs, before the more expensive sampling and batching steps touch it. |
| 4 | tail_sampling | Makes the keep-or-drop decision on whole traces after enough spans have arrived to evaluate policies like error status or high latency. |
| 5 | batch | Groups whatever survived sampling into larger batches immediately before export, which is where batching delivers the most benefit because nothing downstream will touch the data again. |
Head-based sampling decides whether to keep a trace at the moment the first span is created, before anyone knows if the request will be slow or fail. Tail-based sampling waits until a trace, or enough of it, has arrived, and only then applies rules such as always keep traces with an error status, always keep traces slower than a latency threshold, and otherwise keep a small percentage of everything else.
The tradeoff is memory and topology. The processor holds spans in a buffer until a decision window elapses, and it can only make a correct decision if every span belonging to a given trace lands on the same Collector instance. In a load-balanced fleet that means routing by trace identifier at an earlier layer, not by round robin.
If your Collectors run behind a generic load balancer that distributes spans round robin instead of by trace identifier, tail-based sampling will silently make wrong decisions because spans from the same trace scatter across instances that never see the full picture. Fix the routing layer before tuning sampling policies.
There are two common places to run Collector instances, and most production setups use both together rather than picking one.
This split keeps per-node resource usage predictable while concentrating stateful, memory-hungry work like tail sampling in a smaller number of horizontally scaled gateway instances that you can size independently.
Because the SDK only ever talks OTLP to the Collector, switching or adding a backend is a configuration change, not a redeploy of every instrumented service. Add a second named exporter and list it alongside the first in the pipeline, and the Collector fans the same processed data out to both destinations.
exporters:
otlphttp/tempo:
endpoint: https://tempo.example.com:4318
otlphttp/honeycomb:
endpoint: https://api.honeycomb.io
headers:
x-honeycomb-team: ${env:HONEYCOMB_API_KEY}
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, tail_sampling, batch]
exporters: [otlphttp/tempo, otlphttp/honeycomb]This is also the cleanest way to run a proof of concept against a new backend, such as an open-source tracing store or a paid vendor, in parallel with your existing production destination, with zero risk of losing existing data if the trial does not pan out.
Because exporters run after sampling and batching, adding a second backend costs you configuration lines, not double the trace volume you already paid to sample down. The fan-out happens on the already-reduced stream.
Before treating a Collector pipeline as production-ready, walk through this short list.