Building Observable Go Services

Observability is not something you bolt on after the fact. The most reliable production systems we have worked with treat metrics, logs, and traces as first-class concerns from day one.

In this post, we walk through practical patterns for instrumenting Go services with structured logging, Prometheus-compatible metrics exported to VictoriaMetrics, and distributed tracing via OpenTelemetry.

Start with Structured Logging

The foundation of any observable service is structured logging. Rather than free-form log lines, emit JSON logs with consistent fields:

logger.Info("request handled",
    "method", r.Method,
    "path", r.URL.Path,
    "status", status,
    "duration_ms", elapsed.Milliseconds(),
)

This makes logs searchable and aggregatable in systems like VictoriaLogs or Kibana.

Expose Prometheus Metrics

Use the prometheus/client_golang library to expose request counts, latencies, and error rates. VictoriaMetrics ingests these natively and provides long-term storage with efficient compression.

Key metrics to track for any HTTP service:

• Request rate by endpoint and status code
• Latency histograms (p50, p95, p99)
• In-flight request count
• Error rate

Add Distributed Tracing

OpenTelemetry provides a vendor-neutral way to propagate trace context across service boundaries. Combined with metrics and logs, traces give you the full picture when debugging production issues.

Conclusion

Investing in observability early pays dividends when things go wrong in production. The patterns described here are lightweight to implement and provide deep visibility into service behavior.