AI/ML Observability · Weekly Inspiration

Themes This Week

The agent observability space is converging on a hard truth: you can't eval your way out of operational chaos. Five of the eight articles this week come from LangChain's ecosystem, and they're all wrestling with the same fundamental tension—teams are deploying multi-agent systems into production before they've figured out how to tell if those systems are working. The emphasis on trace-level debugging, manual review, and root cause analysis before automation reveals that the industry massively underestimated how different agent observability is from traditional software monitoring. We're not adapting existing tools anymore; we're acknowledging we need entirely new primitives.

The second major thread is the professionalization of the LLM ops stack. The OpenTelemetry Profiles announcement matters because it signals that AI/ML observability is graduating from vendor-specific tooling to infrastructure-grade standards. When OTel adds a signal type, it's because the industry has reached consensus that this capability needs to exist everywhere, not just in premium observability platforms. The timing isn't coincidental—as agent deployments move from experiments to revenue-generating systems, the tolerance for proprietary lock-in drops to zero.

There's also a quiet acknowledgment that evaluation infrastructure has been built backwards. The repeated emphasis on manual trace review, production failure analysis, and domain expert involvement before building automated evals suggests that teams have been cargo-culting benchmark-driven evaluation from the research world without adapting it for production realities. The Kensho case study is instructive here—they built observability and evaluation as core architecture decisions, not afterthoughts. That's the maturity curve: early adopters bolt on evals after deployment, sophisticated teams architect for observability from day one.

What's Actionable Now

If you're running agents in production and you haven't implemented trace-level observability yet, that's your Q1 priority. The LangChain articles are remarkably consistent on this point: manual trace review is the foundation of everything else. LangGraph's embedded tracing capabilities and LangSmith's trace visualization aren't nice-to-haves—they're the minimum viable infrastructure for understanding what your agents are actually doing. The Kensho example demonstrates why: when you have multiple agents communicating through structured protocols, the only way to debug routing failures, data quality issues, or completeness problems is to see the full execution graph. Start instrumenting your agent workflows with structured tracing today, even if you're not using LangChain's stack. OpenTelemetry's semantic conventions for LLM operations are mature enough to use, and you'll thank yourself when you need to debug a production incident at 2am.

The evaluation readiness checklist from LangChain is immediately actionable and probably conflicts with whatever your team is doing right now. If you're spending more than 20-30% of your eval effort on automation and less than 60-80% on root cause analysis, you're optimizing the wrong thing. The core insight—that infrastructure issues and ambiguous success criteria masquerade as agent failures—means you need to build error taxonomies before you build eval pipelines. Concretely, this means creating a structured classification system for failures (infrastructure vs. model vs. task definition vs. data quality) and ensuring domain experts own the success criteria for each agent capability. Stop accumulating benchmark tasks and start curating behavior-focused evals from production failures. This is a people and process change more than a tooling change, which is why it's hard but also why it matters.

The OpenTelemetry Profiles alpha is production-ready enough to pilot if you're dealing with LLM inference cost problems. Continuous profiling has been available from vendors like Datadog and Grafana for years, but the OTel standardization means you can instrument now without vendor lock-in. For teams running self-hosted inference or trying to optimize prompt processing pipelines, having CPU and memory profiles correlated with traces and metrics gives you the data to actually understand where your inference costs are coming from. The overhead is low enough (sub-1% in most implementations) that you can run it in production, and the correlation with distributed traces means you can connect expensive operations back to specific user requests or agent workflows.

Worth Watching

The multi-agent evaluation patterns that Kensho describes—routing accuracy, data quality metrics, completeness checks—are still too application-specific to be productized, but watch for these to crystallize into reusable frameworks over the next two quarters. Right now every team building multi-agent systems is inventing their own evaluation taxonomy. When someone figures out the common abstractions (probably LangChain or someone in their ecosystem given their current momentum), that'll be the moment to adopt rather than continuing to build custom. The signal to watch: when you see multiple case studies using the same evaluation metric names and structures across different domains.

The broader OpenTelemetry semantic conventions for LLM operations are still evolving, but the trajectory is clear. OTel is becoming the standard instrumentation layer for AI/ML workloads the same way it became standard for microservices. The Kotlin SDK announcement is a small data point in a larger pattern—OTel is expanding to cover every runtime and platform where ML models execute. When the semantic conventions for agent-specific operations (tool calls, planning steps, memory operations) stabilize, that's when you'll want to migrate any custom instrumentation to the standard. Watch the OTel LLM working group's GitHub activity; when PRs start getting approved quickly rather than debated endlessly, the conventions are mature.

Monday Morning Ideas

Audit your agent failure taxonomy. Spend two hours this week reviewing your last 50 production agent failures and categorizing them into infrastructure issues, ambiguous success criteria, model capability gaps, and actual bugs. If you don't have 50 failures logged in a structured way, that's your real problem—you're flying blind. This exercise will immediately reveal whether you're spending eval effort on the right problems.

Instrument one agent workflow with full trace capture. Pick your most critical or most problematic agent workflow and add comprehensive tracing if it doesn't exist already. Capture every LLM call, tool invocation, and decision point with structured attributes. Use OpenTelemetry if you're not locked into a vendor stack, or LangSmith if you're already in that ecosystem. The goal is to be able to reconstruct the complete execution graph for any production run within 24 hours.

Schedule a manual trace review session with domain experts. Block two hours with the people who actually understand what your agents are supposed to do (not just the engineers who built them) and walk through 10-15 production traces together. You're looking for patterns in failures, ambiguity in success criteria, and gaps between what the system does and what users need. This is the 60-80% root cause analysis work that should happen before you automate anything.

Evaluate OpenTelemetry Profiles for your inference workloads. If you're running self-hosted models or spending more than $10K/month on inference, spin up the OTel Profiles alpha in a staging environment and correlate profiling data with your existing traces. You're looking for hotspots in prompt processing, tokenization, or model execution that might be optimizable. Even if you don't act on it immediately, having the instrumentation in place means you'll have data when the CFO asks why inference costs are growing faster than usage.