How Mastodon Runs OpenTelemetry Collectors in Production

Mastodon's federation model creates an unusual telemetry challenge that makes their OpenTelemetry Collector deployment worth examining. Unlike typical monolithic or microservices architectures, Mastodon instances operate as independent servers that communicate via ActivityPub, meaning observability needs to work across organizational boundaries while respecting privacy constraints.

The Mastodon team runs OTel Collectors in agent mode on each application server rather than using a centralized gateway pattern. This decision stems directly from their federation architecture. When a user on mastodon.social interacts with content from another instance, the telemetry needs to capture both the local processing and the federated request characteristics without leaking sensitive information about remote instances. Running collectors locally gives them fine-grained control over what gets exported before data leaves the server.

Their collector configuration uses the filter processor extensively to strip personally identifiable information from spans before export. Specifically, they redact HTTP headers containing authentication tokens and user identifiers while preserving the request path structure and timing data. This matters because Mastodon's Rails application generates spans with full request context by default, and the collector acts as the enforcement point for their data retention policies.

For metrics, they're using the prometheus receiver to scrape the existing Puma and Sidekiq exporters rather than instrumenting everything through OTel SDKs. This hybrid approach reflects a pragmatic reality: Mastodon's Ruby codebase already had Prometheus instrumentation that worked, and rewriting it would have provided minimal value. The collector aggregates these metrics alongside OTLP traces from newer service components, then exports both to their Grafana Cloud backend using the otlphttp exporter with batch processing configured at 10 second intervals.

The most interesting technical detail is how they handle the bursty traffic patterns inherent to federated social media. When a popular post goes viral, federation causes cascading requests across instances. Their collector configuration uses the memory_limiter processor with a soft limit of 512 MiB and hard limit of 768 MiB per collector instance. They set check_interval to 1 second because the default 1 second proved too slow during traffic spikes, leading to OOM kills. The spike_limit_mib parameter is set to 128 MiB to accommodate sudden bursts without immediately triggering backpressure.

They also run the batch processor with a timeout of 5 seconds and send_batch_size of 8192, tuned specifically for their pattern of many small spans from ActivityPub requests rather than fewer large traces. The default batch settings caused excessive memory usage because federation generates thousands of short-lived HTTP client spans during peak activity.

One operational lesson: they initially tried running collectors as sidecars in their Kubernetes deployment but switched to DaemonSet mode after discovering that pod churn during deployments caused gaps in metrics collection. The DaemonSet approach means collectors persist across application pod restarts, maintaining continuous telemetry even during rolling updates.

For teams considering OTel Collector deployments in distributed systems, Mastodon's experience highlights that the agent versus gateway decision depends heavily on your data governance requirements. If you need to enforce privacy boundaries before telemetry leaves individual nodes, the agent pattern with aggressive filtering is worth the operational overhead of managing more collector instances.