Epic AUD-OTEL-001: Observability & Telemetry Framework¶
Status: 🔄 In Progress (85% complete)
Owner: SRE Team
Target: Q4 2025
Epic Description¶
This epic establishes the Observability and Telemetry Framework for the Audit Trail Platform (ATP). It integrates OpenTelemetry (OTel) instrumentation across all microservices to provide unified traces, metrics, and logs. The framework ensures that every transaction—spanning ingestion, storage, query, and export—is fully traceable, measurable, and diagnosable within the platform's observability stack (Prometheus, Grafana, Jaeger, and Seq).
Epic Objectives¶
- Implement OpenTelemetry middleware across all ATP services
- Standardize structured logging with correlation IDs
- Collect metrics on performance, latency, error rates
- Integrate with Prometheus and Grafana
- Provide unified observability dashboards (90% complete)
- Ensure observability coverage ≥ 95% across all microservices
Features¶
Feature AUD-OTEL-TRC-001: Traces & Spans ✅¶
Status: Complete
Delivered: Q3 2025
Tasks: - ✅ Add OTel middleware to all services - ✅ Define trace naming & span conventions - ✅ Configure trace sampling & exporters
Achievements: - 100% trace continuity across service boundaries - Traces visible in Jaeger and Grafana Tempo - Average tracing overhead: 3.2% (target: <5%)
Feature AUD-OTEL-MET-001: Metrics (Latency, Throughput) ✅¶
Status: Complete
Delivered: Q3 2025
Tasks: - ✅ Instrument core metrics per service - ✅ Configure Prometheus/Grafana dashboards - ✅ Define SLOs and alerts
Deployed Metrics:
- http_requests_total, http_request_duration_seconds
- grpc_calls_total, bus_messages_published_total
- audit_records_ingested_total, audit_query_latency_seconds
Dashboards: - ✅ Service Health Dashboard (all services) - ✅ Business Metrics Dashboard (ingestion, query volumes) - ✅ SLO Dashboard (availability, error budget)
Feature AUD-OTEL-LOG-001: Structured Logs 🔄¶
Status: In Progress (90%)
Target: November 2025
Tasks: - ✅ Implement structured logging format (JSON) - ✅ Integrate centralized log storage (Seq, Application Insights) - 🔄 Add contextual logging middleware (final review)
Current Work: - Finalizing exception handling enrichment - Performance validation (CPU overhead < 3%) - Integration with correlation IDs from traces
Blockers: - None
Current Sprint Work¶
Sprint 2025-10-28 to 2025-11-08¶
Focus: Complete structured logging rollout to production
Tasks: 1. ✅ Deploy logging middleware to all services 2. 🔄 Validate log correlation with traces (testing) 3. ⏳ Create log aggregation dashboard in Grafana 4. ⏳ Configure log retention policies (30 days hot, 365 days archive)
Team Capacity: - 2 SRE engineers @ 100% allocation - 1 Platform engineer for dashboard creation
Dependencies¶
Upstream (Depends On)¶
- ✅ AUD-ARC-001: Platform architecture defined
- ✅ AUD-OPS-001: CI/CD pipelines operational
Downstream (Enables)¶
- All epics benefit from observability
- AUD-CHAOS-001: Chaos engineering needs metrics
- AUD-OPS-002: Operations monitoring requires dashboards
Metrics & KPIs¶
| Metric | Target | Current | Status |
|---|---|---|---|
| Observability Coverage | > 95% | 97% | ✅ |
| Trace Propagation | 100% | 100% | ✅ |
| Metrics Collection Rate | > 99% | 99.4% | ✅ |
| Log Ingestion Latency | < 5s | 3.2s | ✅ |
| Dashboard Load Time | < 2s | 1.8s | ✅ |
| Tracing Overhead | < 5% | 3.2% | ✅ |
Technology Stack¶
| Component | Technology | Purpose |
|---|---|---|
| Instrumentation | OpenTelemetry SDK | Traces, metrics, logs |
| Trace Backend | Jaeger + Grafana Tempo | Distributed tracing |
| Metrics | Prometheus | Time-series metrics |
| Visualization | Grafana | Dashboards and alerts |
| Logs (Dev) | Seq | Structured log search |
| Logs (Prod) | Azure Application Insights | Cloud-native logging |
Key Achievements¶
Q3 2025¶
- ✅ OpenTelemetry integrated across 8 microservices
- ✅ 15 Grafana dashboards deployed
- ✅ SLO dashboard with error budget tracking
- ✅ Distributed tracing validated end-to-end
Q4 2025 (Current)¶
- ✅ Structured logging format standardized
- 🔄 Log aggregation dashboard (90% complete)
- 📊 97% observability coverage achieved
Risks & Mitigation¶
| Risk | Severity | Mitigation | Status |
|---|---|---|---|
| Log volume overwhelming storage | Medium | Implement sampling + log retention policies | ✅ Mitigated |
| Grafana dashboard performance | Low | Optimize queries, add caching | ✅ Mitigated |
| Trace sampling misses critical errors | Medium | Always trace errors + 10% sampling | ✅ Mitigated |
Recent Updates¶
2025-10-30: - ✅ Logging middleware performance validated (<3% CPU overhead) - 🔄 Log correlation with traces tested in staging - 📝 Retention policy configuration documented
2025-10-15: - ✅ All services upgraded to OpenTelemetry 1.6.0 - ✅ Grafana alert rules published - 📊 SLO tracking dashboard deployed
2025-10-01: - ✅ Structured logging deployed to Ingestion and Storage services - 🎯 Started integration testing for log correlation
Next Steps¶
- Complete logging middleware - Final review and production deployment
- Create log aggregation dashboard - Grafana panels for error tracking
- Configure retention policies - 30 days hot, 365 days archive
- Document observability patterns - Best practices for service teams
Related Documentation¶
- Implementation: Observability
- Operations: Monitoring
- Operations: Alerts & SLOs
- Reference: Baseline Roadmap
Azure DevOps Links¶
- Epic: AUD-OTEL-001
- Current Sprint: Sprint 2025-10-28
- Dashboards: Grafana ATP
Next Review: 2025-11-07 (Sprint Planning)
Contact: #atp-sre-team on Slack