The OpenTelemetry Migration That Nearly Broke Everything
Last year, we decided to migrate our microservices platform from a mess of monitoring tools to OpenTelemetry. The decision made sense on paper. We had Prometheus for metrics, Jaeger for traces, and a custom logging solution that nobody really understood anymore. The promise of unified observability through a single framework was compelling.
The migration started well enough. The auto-instrumentation worked beautifully for our Java services, picking up HTTP requests and database calls without any code changes. But then we hit the first wall. Our legacy .NET services required manual instrumentation, and the documentation was terrible. Worse, the semantic conventions kept changing between versions, breaking our dashboards every few weeks.
Three months in, we were generating so much telemetry data that our storage costs tripled. The default sampling rates were completely wrong for our traffic patterns. High-frequency endpoints were drowning out the low-frequency but important payment flows we actually needed to monitor. We spent two weeks writing custom sampling rules that finally brought the data volume back to manageable levels.
Today, OpenTelemetry runs our entire observability stack. But getting there required patience and a willingness to iterate constantly. The framework is powerful, but it assumes you understand distributed systems deeply. If you’re just starting out, honestly consider a more opinionated solution first.
Prometheus in the Real World
Prometheus has been our metrics backbone for five years now. It’s reliable, well-documented, and the query language becomes second nature once you use it daily. But it’s not the silver bullet that conference talks make it seem.
The biggest surprise was cardinality explosion. Early on, we instrumented everything with user IDs as labels. That worked fine in development with a handful of test users. Production hit us with 50,000 unique user IDs, creating millions of time series overnight. Prometheus started consuming 32GB of RAM and query times stretched into minutes. We learned the hard way that labels need careful curation.
Storage retention became another challenge. Prometheus wasn’t designed for long-term storage, but business stakeholders wanted year-over-year comparisons. We eventually deployed Thanos to handle long-term storage and cross-cluster queries. The setup took three weeks to get right, mainly because the documentation assumes you’re already familiar with object storage and Kubernetes operators.
Prometheus remains our go-to for metrics. The ecosystem is mature, integrations are everywhere, and when something goes wrong at 3 AM, you want tools you can trust. Just respect its limitations and plan for scale from day one.
The Grafana Obsession That Went Too Far
Grafana dashboards became an obsession for our team. What started as a few simple charts for system health turned into 200+ dashboards covering every conceivable metric. Developers were spending more time crafting visualizations than writing code.
The real problem wasn’t the number of dashboards, it was that nobody knew which ones mattered. During incidents, we’d frantically click through dozens of charts looking for clues while our services burned. We had perfect visibility into irrelevant details and blind spots in important areas. The solution was painful but necessary: we deleted 80% of our dashboards and started over.
We settled on three dashboard categories: service health, business metrics, and incident response. Service health dashboards follow the RED method (Rate, Errors, Duration) for each service. Business metrics track actual user behavior like signup rates and payment processing. Incident response dashboards surface the specific metrics we need during outages, nothing more.
The new approach forced us to think about what we actually needed to monitor versus what we could monitor. Fewer dashboards meant faster incident response and less cognitive overhead during normal operations. Sometimes less really is more.
DataDog vs Build Your Own
Two years ago, we faced a choice: continue building our observability stack in-house or move to DataDog. The engineering team favored the DIY approach. We understood our tools, could customize everything, and weren’t locked into vendor pricing models. Management wanted the reliability and support that came with a commercial solution.
We ran both systems in parallel for six months to make an informed decision. DataDog’s agent automatically discovered and monitored services we had forgotten existed. Their anomaly detection caught a slow memory leak that our static thresholds missed completely. The unified interface meant less context switching during incidents.
But the cost was eye-watering. Our monthly DataDog bill exceeded what we spent on compute resources. The query language was less flexible than PromQL, and custom metrics required careful planning to avoid runaway costs. We also discovered vendor lock-in wasn’t just about data portability, it was about institutional knowledge. Team members became experts in DataDog’s specific features rather than general observability principles.
We ultimately chose DataDog for production and kept our open-source stack for development. It’s a compromise that gives us vendor reliability where it matters most while maintaining our technical skills and cost control for non-important environments. The decision wasn’t purely technical, it reflected our team’s capacity and business priorities.
What Actually Works in Practice
After years of trial and error, our observability strategy has stabilized around a few core principles. Start simple and add complexity only when justified. Instrument the user journey first, then system internals. Optimize for incident response, not perfect coverage.
The most valuable monitoring we’ve implemented tracks business metrics alongside technical ones. When checkout conversion rates drop, we correlate that with API latency, error rates, and deployment timing. This context transforms debugging from guesswork into methodical problem-solving.
No single framework solves every observability challenge. We use Prometheus for metrics, OpenTelemetry for traces, structured logging for event correlation, and DataDog for alerting and incident management. Each tool excels in its domain, and the integration overhead is manageable with proper planning.
The frameworks will keep changing, but the fundamental challenges remain the same: understanding your systems, building team expertise, and balancing cost with capability. I’d love to hear about your observability experiences and the lessons you’ve learned along the way.
