From raw data to flame graphs: A deep dive into how the OpenTelemetry eBPF profiler symbolizes Go

When an eBPF profiler captures your Go application's stack trace in production, it sees nothing but hexadecimal addresses: 0x00000000000f0318, 0x00000000000f0478, 0x0000000000050c08. These raw program counters are useless for debugging until they're transformed into function names like main.computeResult or runtime.main. This transformation—symbolization—is what makes profiling actionable, and Go has a significant architectural advantage here that most engineers don't fully appreciate.

The core challenge is that eBPF profilers operate in kernel space. They can't call into your application's runtime, can't load debugging agents, and can't ask the running program what function corresponds to an address. They capture addresses and must figure out the rest by parsing binary files on disk, all while maintaining sub-1% CPU overhead across potentially hundreds of processes sampling at 20-100Hz.

When you compile a Go binary, the compiler embeds a symbol table mapping addresses to function names. Run nm on any Go binary and you'll see entries like "00000000000f0310 T main.processRequest". Given address 0xf0318, a profiler searches this table, finds it falls between processRequest's start (0xf0310) and the next function (0xf0370), and returns the correct symbol. Simple enough in theory.

The critical difference with Go is the gopclntab section. This is a compact data structure that maps every function's address range to its name and source location, and it persists even in stripped binaries. When you run strip on a Go binary to remove debug symbols—a common practice to reduce binary size in production—gopclntab remains intact. The OpenTelemetry eBPF profiler leverages this by parsing gopclntab directly from the binary, enabling on-target symbolization without requiring separate debug symbol files.

Contrast this with C or Rust programs. When you strip those binaries, you lose the symbol table entirely. The profiler captures addresses but has no mapping to function names. You're left with two options: ship unstripped binaries to production (bloating deployment size), or implement server-side symbolization where you maintain a symbol server with debug info for every binary version ever deployed. Both approaches add operational complexity that Go simply avoids.

The performance implications matter too. The profiler uses binary search over gopclntab entries, typically finding symbols in 10-15 comparisons even for binaries with thousands of functions. It caches recently symbolized frames in a hash table since hot paths get sampled repeatedly. These optimizations keep symbolization overhead negligible, but they only work because gopclntab provides a reliable, always-present data source.

You can inspect this yourself. Run "readelf -S your-go-binary | grep gopclntab" and you'll see the section. Run "file your-binary" before and after stripping—gopclntab survives. Try the same with a C binary and watch your symbols disappear with strip.

When symbolization fails in production, it's usually one of three issues: the binary on disk doesn't match what's running (deployment race condition), the profiler can't read the binary due to permissions, or you're dealing with a non-Go binary that actually was stripped. Understanding gopclntab helps you debug these scenarios quickly. If gopclntab is present and readable, symbolization should work. If you're seeing hex addresses for a Go program, the problem is elsewhere—likely a version mismatch or file access issue.

This architectural choice in Go's toolchain—embedding gopclntab by default and making it survive stripping—is why Go programs consistently produce better profiling data than other compiled languages. It's not magic, just a deliberate design decision that eBPF profilers can exploit. When you're evaluating languages for observable systems, this kind of operational detail matters more than benchmark numbers.