Skip to content

Key Rotation - Audit Trail Platform (ATP)

Continuous key renewal without disruption — ATP implements automated key rotation for all cryptographic keys and secrets with zero-downtime procedures, comprehensive audit trails, and compliance with security best practices.

📋 Documentation Generation Plan

This document will be generated in 6 cycles. Current progress:

Cycle Topics Estimated Lines Status
Cycle 1 Key Rotation Fundamentals & Strategy (1-2) ~2,500 ⏳ Not Started
Cycle 2 Encryption Key Rotation (3-4) ~3,000 ⏳ Not Started
Cycle 3 Signing Key Rotation (5-6) ~3,000 ⏳ Not Started
Cycle 4 Secret Rotation (7-8) ~2,500 ⏳ Not Started
Cycle 5 Automated Rotation & Monitoring (9-10) ~2,500 ⏳ Not Started
Cycle 6 Emergency Procedures & Compliance (11-12) ~2,500 ⏳ Not Started

Total Estimated Lines: ~16,000

Purpose & Scope

This document defines comprehensive key rotation procedures for the Audit Trail Platform (ATP), establishing strategies, automation, monitoring, and emergency procedures for rotating encryption keys, signing keys, certificates, and secrets to ensure continuous security posture, meet compliance requirements, and maintain zero-downtime operations during rotation events.

Key Rotation Principles - Regular Rotation: All keys rotated on schedule (daily, monthly, quarterly, annually) - Zero-Downtime: Rotation without service interruption or data unavailability - Automated: Rotation automated where possible to reduce human error - Audited: All rotation events logged with full audit trail - Dual-Running: New and old keys active simultaneously during transition - Backward Compatibility: Old keys retained for decrypting historical data - Emergency Rotation: Rapid rotation procedures for compromised keys - Compliance: Meet NIST, PCI-DSS, HIPAA rotation requirements

What this document covers

  • Establish key rotation fundamentals: Why rotate keys, rotation schedules, risk mitigation
  • Define encryption key rotation: KEK and DEK rotation, envelope encryption, backward compatibility
  • Specify signing key rotation: HSM signing keys, certificate rotation, historical signature validation
  • Document secret rotation: Connection strings, API keys, tokens, passwords
  • Detail certificate rotation: TLS certificates, service mesh certificates, client certificates
  • Describe automated rotation: Azure Key Vault automation, rotation triggers, monitoring
  • Outline zero-downtime procedures: Dual-running keys, graceful transition, rollback
  • Specify key versioning: Key versions, key metadata, key discovery
  • Document audit and compliance: Rotation logs, compliance evidence, attestation
  • Detail emergency rotation: Compromised key procedures, incident response, rapid rotation
  • Describe monitoring and alerting: Rotation status, expiration warnings, failure alerts
  • Outline testing and validation: Rotation testing, rollback testing, compliance validation

Out of scope (referenced elsewhere)

Readers & ownership

  • Security Engineering (owners): Key rotation strategy, procedures, compliance
  • Operations/SRE: Rotation execution, monitoring, incident response
  • Platform Engineering: Automation implementation, Key Vault configuration
  • Compliance/Audit: Rotation evidence, compliance validation, attestation
  • Backend Developers: Application integration with rotated keys
  • Incident Response: Emergency rotation procedures, compromised key handling

Artifacts produced

  • Rotation Schedule: Calendar of all key rotations (daily, monthly, quarterly, annually)
  • Rotation Procedures: Step-by-step runbooks for each key type
  • Automation Scripts: Azure Functions for automated rotation
  • Key Inventory: All keys with rotation status, next rotation date, owner
  • Rotation Audit Logs: Complete history of all rotations with timestamps and actors
  • Compliance Evidence: Rotation logs for SOC 2, HIPAA, PCI-DSS audits
  • Monitoring Dashboards: Key expiration warnings, rotation status, failure tracking
  • Alert Rules: Expiration warnings (30/7/1 days), rotation failures, usage of old keys
  • Emergency Runbooks: Compromised key rotation, incident response, rollback
  • Testing Framework: Rotation testing, rollback validation, zero-downtime verification
  • Validation Reports: Post-rotation validation, encryption/decryption tests
  • Key Metadata Schema: Key versions, rotation history, usage tracking

Acceptance (done when)

  • All key types have documented rotation procedures (encryption, signing, secrets, certificates)
  • Rotation schedules are defined for all keys with compliance-driven frequencies
  • Automated rotation is implemented for all keys where feasible
  • Zero-downtime procedures ensure no service interruption during rotation
  • Dual-running keys pattern allows graceful transition from old to new keys
  • Audit logging captures all rotation events with timestamps and actors
  • Monitoring and alerting warns of approaching expirations and rotation failures
  • Emergency procedures enable rapid rotation of compromised keys (<1 hour)
  • Testing framework validates rotation procedures in non-production environments
  • Compliance evidence supports SOC 2, HIPAA, PCI-DSS attestation
  • Rollback procedures allow reverting to previous keys if rotation causes issues
  • Documentation complete with runbooks, automation code, diagrams, and validation procedures

Detailed Cycle Plan

CYCLE 1: Key Rotation Fundamentals & Strategy (~2,500 lines)

Topic 1: Key Rotation Fundamentals

What will be covered:

  • What is Key Rotation?
  • Replacing cryptographic keys with new keys
  • Transitioning from old key to new key
  • Retiring old key after transition

  • Regular security practice

  • Why Rotate Keys?

  • Limit Exposure: Reduce impact of key compromise
  • Compliance: NIST, PCI-DSS, HIPAA require regular rotation
  • Best Practice: Industry standard security hygiene
  • Reduce Risk: Less data encrypted with single key
  • Crypto-Period: Limit cryptographic period per key

  • Post-Quantum Prep: Enable algorithm migration

  • Risks of Not Rotating Keys

  • Long-term key compromise affects all historical data
  • Compliance violations (fines, audit failures)
  • Increased cryptanalysis risk over time

  • Inability to respond to algorithm weaknesses

  • Key Rotation Challenges

  • Downtime: Rotation can cause service interruption
  • Data Re-encryption: Historical data encrypted with old keys
  • Coordination: Multiple services using same keys
  • Backward Compatibility: Old signatures must remain valid

  • Complexity: Many key types with different rotation procedures

  • ATP Key Inventory

Key Type Purpose Rotation Frequency Owner Storage
Master Encryption Key (MEK) Key Encryption Key Annual Security Azure Key Vault HSM
Data Encryption Keys (DEK) Per-tenant data encryption Quarterly Security Azure Key Vault
Signing Keys Digital signatures (integrity) Quarterly Security Azure Key Vault HSM
TLS Certificates HTTPS, mTLS 90 days (auto) Platform cert-manager
Service Bus Keys Azure Service Bus access Quarterly Platform Azure Key Vault
Database Credentials SQL, Cosmos DB Quarterly Platform Azure Key Vault
API Keys External service auth Monthly Integration Azure Key Vault
JWT Signing Keys OAuth token signing Quarterly Security Azure Key Vault
Webhook Secrets HMAC verification Quarterly Integration Azure Key Vault
  • Rotation Schedules
  • Daily: TLS certificates (automated via cert-manager)
  • Monthly: API keys, webhook secrets
  • Quarterly: DEK, signing keys, service credentials
  • Annually: Master keys (KEK)

  • Ad-hoc: Emergency rotation for compromised keys

  • Compliance Requirements

Standard Requirement ATP Implementation
NIST SP 800-57 Rotate keys based on crypto-period Quarterly DEK rotation
PCI-DSS 3.2 Rotate keys at least annually Quarterly (exceeds requirement)
HIPAA Encryption keys rotated regularly Quarterly with audit trail
SOC 2 Key management controls documented This document + evidence
GDPR Encryption key management Documented procedures

Code Examples: - Key inventory query (Azure Key Vault) - Rotation schedule configuration - Compliance mapping table

Diagrams: - Key hierarchy (MEK → KEK → DEK) - Rotation timeline - Key lifecycle states - Compliance requirements map

Deliverables: - Key rotation fundamentals guide - ATP key inventory - Rotation schedule - Compliance mapping

Topic 2: Key Rotation Strategy

What will be covered:

  • Rotation Strategies

1. Immediate Rotation (Risky) - Switch from old key to new key immediately - All services must update simultaneously - Requires downtime or careful coordination - Not used in ATP (too risky)

2. Dual-Running (ATP Approach) - New key created, old key remains active - New operations use new key - Old data remains encrypted with old key - Transition period: Both keys active - Retire old key after all data migrated - Zero-downtime rotation

3. Re-encryption (For Data-at-Rest) - Decrypt data with old key - Re-encrypt data with new key - Background job, gradual migration - Old key retired after all data re-encrypted

  • ATP Key Rotation Workflow (General) 

    1. Generate new key in Key Vault
2. Update key version pointer (new key becomes primary)
3. Dual-running period begins (new writes use new key, old reads use old key)
4. Background re-encryption (if applicable)
5. Monitor usage of old key (decreases over time)
6. Retire old key after transition complete
7. Archive old key for historical decryption only
8. Log rotation event for audit

  • Key States During Rotation

  • Active: Current key for new operations
  • Primary: Preferred key (after rotation, new key)
  • Secondary: Previous key (during dual-running)
  • Retired: No longer used for encryption, only decryption
  • Archived: Historical use only, access restricted

  • Revoked: Compromised, no longer trusted

  • Backward Compatibility Strategy

  • Old keys never deleted (archive only)
  • Old signatures remain valid indefinitely
  • Old encrypted data remains readable

  • Key versioning enables correct key selection

  • Zero-Downtime Rotation Pattern 

    public class KeyRotationService
{
    private readonly IKeyVaultClient _keyVault;

    public async Task RotateEncryptionKey(string keyName)
    {
        // 1. Create new key version
        var newKey = await _keyVault.CreateKeyVersionAsync(keyName);

        // 2. Update primary key pointer (atomic operation)
        await _keyVault.SetPrimaryKeyVersionAsync(keyName, newKey.Version);

        // 3. New encryptions use new key automatically
        // 4. Old decryptions use old key automatically (versioned)
        // 5. No downtime!

        // 6. Schedule background re-encryption (optional)
        await ScheduleReEncryptionJob(keyName, newKey.Version);

        // 7. Log rotation event
        await LogKeyRotation(keyName, newKey.Version);
    }
}

Code Examples: - Rotation workflow implementation - Dual-running key pattern - Key state machine - Zero-downtime rotation code

Diagrams: - Rotation strategies comparison - Dual-running timeline - Key state transitions - Zero-downtime flow

Deliverables: - Rotation strategy guide - Workflow specifications - Zero-downtime procedures - State management code

CYCLE 2: Encryption Key Rotation (~3,000 lines)

Topic 3: Data Encryption Key (DEK) Rotation

What will be covered:

  • DEK Rotation Overview
  • Per-tenant data encryption keys
  • Envelope encryption pattern (KEK wraps DEK)
  • DEK rotation frequency: Quarterly

  • Zero-downtime rotation essential

  • Envelope Encryption Refresher 

    Data Encryption:
1. Generate random DEK
2. Encrypt data with DEK (AES-256)
3. Encrypt DEK with KEK (from Key Vault)
4. Store encrypted DEK with encrypted data

Data Decryption:
1. Read encrypted DEK from storage
2. Decrypt DEK with KEK (from Key Vault)
3. Decrypt data with DEK
4. Return plaintext data

  • DEK Rotation Procedure 

    public async Task RotateTenantDEK(string tenantId)
{
    var keyName = $"dek-{tenantId}";

    // 1. Generate new DEK
    var newDEK = GenerateRandomDEK(); // 256-bit AES key

    // 2. Encrypt new DEK with current KEK
    var encryptedNewDEK = await _keyVault.EncryptAsync(
        kekName: "master-kek",
        plaintext: newDEK);

    // 3. Store new encrypted DEK with version
    await _dekStore.StoreAsync(
        tenantId: tenantId,
        dekVersion: DateTime.UtcNow.Ticks.ToString(),
        encryptedDEK: encryptedNewDEK,
        isPrimary: true);

    // 4. Mark old DEK as secondary (still valid for decryption)
    await _dekStore.MarkAsSecondaryAsync(tenantId, previousVersion);

    // 5. New encryptions automatically use new DEK
    // 6. Old data remains encrypted with old DEK (lazy re-encryption)

    // 7. Log rotation
    _logger.LogInformation(
        "DEK rotated for tenant {TenantId}, new version {Version}",
        tenantId, newVersion);
}

  • Data Re-encryption (Background Job)

  • Not required immediately (dual-running handles it)
  • Optional: Background job re-encrypts data
  • Gradual re-encryption over days/weeks
  • Prioritize hot data, defer cold data

  • Track re-encryption progress

  • DEK Versioning

  • Each DEK has version identifier
  • Encrypted data includes DEK version reference
  • Decryption selects correct DEK version

  • Old DEK versions retained indefinitely (for historical data)

  • Per-Tenant vs Shared DEK

  • ATP uses per-tenant DEK (tenant isolation)
  • Tenant deletion = DEK deletion (after retention)
  • Key compromise limited to single tenant
  • Compliance: tenant-scoped encryption

Code Examples: - Complete DEK rotation implementation - Background re-encryption job - DEK versioning and selection - Per-tenant key management

Diagrams: - DEK rotation workflow - Envelope encryption with rotation - Re-encryption process - DEK versioning timeline

Deliverables: - DEK rotation procedure - Re-encryption automation - Versioning implementation - Per-tenant key management

Topic 4: Key Encryption Key (KEK) Rotation

What will be covered:

  • KEK Rotation Overview
  • Master key that encrypts DEKs
  • KEK rotation frequency: Annually
  • More complex than DEK rotation

  • Requires re-wrapping all DEKs

  • KEK Rotation Procedure 

    public async Task RotateKEK()
{
    // 1. Create new KEK version in Azure Key Vault
    var newKEK = await _keyVault.CreateKeyVersionAsync("master-kek");

    // 2. Get all encrypted DEKs
    var allDEKs = await _dekStore.GetAllEncryptedDEKsAsync();

    // 3. Re-wrap each DEK (decrypt with old KEK, encrypt with new KEK)
    foreach (var dek in allDEKs)
    {
        // Decrypt DEK with old KEK
        var plaintextDEK = await _keyVault.DecryptAsync(
            keyName: "master-kek",
            keyVersion: oldKEKVersion,
            ciphertext: dek.EncryptedValue);

        // Encrypt DEK with new KEK
        var reWrappedDEK = await _keyVault.EncryptAsync(
            keyName: "master-kek",
            keyVersion: newKEK.Version,
            plaintext: plaintextDEK);

        // Update DEK storage
        await _dekStore.UpdateAsync(
            dek.Id,
            encryptedValue: reWrappedDEK,
            kekVersion: newKEK.Version);
    }

    // 4. Update primary KEK pointer
    await _keyVault.SetPrimaryKeyVersionAsync("master-kek", newKEK.Version);

    // 5. Mark old KEK as retired (keep for emergency)
    await _keyVault.DisableKeyVersionAsync("master-kek", oldKEKVersion);

    // 6. Log rotation
    _logger.LogWarning(
        "KEK rotated from {OldVersion} to {NewVersion}, {Count} DEKs re-wrapped",
        oldKEKVersion, newKEK.Version, allDEKs.Count);
}

  • KEK Rotation Challenges

  • Must re-wrap thousands/millions of DEKs
  • Long-running operation (hours)
  • Must not impact production

  • Requires careful coordination

  • Re-wrapping Strategy

  • Batch re-wrapping (1000 DEKs at a time)
  • Gradual migration over maintenance window
  • Progress tracking and checkpointing
  • Rollback capability if issues
  • Zero-downtime (old KEK still works during re-wrap)

Code Examples: - KEK rotation implementation - DEK re-wrapping code - Batch processing - Progress tracking

Diagrams: - KEK rotation workflow - DEK re-wrapping process - Batch processing strategy - Zero-downtime timeline

Deliverables: - KEK rotation procedure - Re-wrapping automation - Progress monitoring - Rollback procedures

CYCLE 3: Signing Key Rotation (~3,000 lines)

Topic 5: Digital Signing Key Rotation

What will be covered:

  • Signing Key Rotation Overview
  • HSM signing keys for integrity proofs
  • Rotation frequency: Quarterly
  • Challenge: Old signatures must remain valid

  • Solution: Key versioning with historical keys

  • Signing Key Rotation Workflow 

    public async Task RotateSigningKey()
{
    // 1. Generate new signing key in HSM
    var newKey = await _keyVault.CreateKeyAsync(
        keyName: "atp-signing-key",
        keyType: KeyType.RSA,
        keySize: 2048,
        keyOptions: new KeyCreateOptions 
        { 
            Enabled = true,
            KeyOps = new[] { KeyOperation.Sign, KeyOperation.Verify }
        });

    // 2. Export public key certificate
    var publicCert = await _keyVault.GetCertificateAsync(newKey.Id);

    // 3. Publish public certificate for verifiers
    await PublishPublicCertificate(publicCert);

    // 4. Update primary signing key pointer
    await _configService.UpdateAsync(
        "IntegrityService:SigningKeyId",
        newKey.Id);

    // 5. New signatures use new key immediately
    // 6. Old signatures verified with old public keys (historical)

    // 7. Mark old key as retired (archive)
    await _keyVault.UpdateKeyPropertiesAsync(
        keyId: oldKeyId,
        enabled: false, // Can't sign
        tags: new Dictionary<string, string> 
        {
            ["status"] = "retired",
            ["rotatedAt"] = DateTime.UtcNow.ToString("O"),
            ["replacedBy"] = newKey.Id
        });

    // 8. Log rotation event
    _logger.LogWarning(
        "Signing key rotated from {OldKey} to {NewKey}",
        oldKeyId, newKey.Id);
}

  • Historical Signature Validation

  • Signatures include key ID and version
  • Verification uses corresponding public key
  • Old signatures remain valid forever

  • Public key archive for verification

  • Public Key Distribution

  • Publish public certificates to well-known URL
  • JWKS (JSON Web Key Set) endpoint
  • Certificate chain included

  • Automatic distribution to verifiers

  • Key Ceremony for Critical Keys

  • Multi-person approval for KEK generation
  • Secure key generation environment
  • Split knowledge for master secrets
  • Documented and witnessed procedure

Code Examples: - Signing key rotation code - Historical signature verification - Public key publishing - JWKS endpoint implementation

Diagrams: - Signing key rotation flow - Historical signature validation - Public key distribution - Key ceremony process

Deliverables: - Signing key rotation runbook - Verification with historical keys - Public key management - Key ceremony procedures

Topic 6: Certificate Rotation

What will be covered:

  • TLS Certificate Rotation
  • Let's Encrypt certificates (90-day validity)
  • Automatic renewal with cert-manager
  • 30 days before expiration trigger

  • Zero-downtime certificate replacement

  • Service Mesh Certificate Rotation

  • mTLS certificates for pod-to-pod
  • Automatic rotation (daily)
  • Managed by service mesh (Linkerd/Istio)

  • No application code changes needed

  • Client Certificate Rotation

  • Certificates for external clients
  • Manual renewal process
  • Grace period for old certificates
  • Revocation procedures

Complete certificate rotation guide

CYCLE 4: Secret Rotation (~2,500 lines)

Topic 7: Connection String Rotation

What will be covered:

  • Database Connection String Rotation
  • Azure SQL, Cosmos DB credentials
  • Rotation frequency: Quarterly

  • Dual-running credentials during transition

  • Service Bus Connection String Rotation

  • SAS tokens or managed identity (preferred)
  • If using connection strings: Quarterly rotation

  • Primary and secondary keys

  • Storage Account Key Rotation

  • Azure Storage account keys
  • Rotation frequency: Quarterly
  • Use managed identity (no rotation needed)

Complete connection string rotation procedures

Topic 8: API Key and Token Rotation

What will be covered:

  • API Key Rotation
  • External service API keys
  • Rotation frequency: Monthly

  • Dual-key pattern (key1 and key2)

  • Webhook Secret Rotation

  • HMAC secrets for webhooks
  • Rotation frequency: Quarterly
  • Notify consumers of new secret

  • Transition period with both secrets valid

  • OAuth Token Rotation

  • JWT signing keys
  • Rotation frequency: Quarterly
  • JWKS rotation
  • Token validation with key discovery

Complete API key and token rotation procedures

CYCLE 5: Automated Rotation & Monitoring (~2,500 lines)

Topic 9: Automated Key Rotation

What will be covered:

  • Azure Key Vault Automatic Rotation
  • Built-in rotation for Key Vault keys
  • Rotation policy configuration
  • Notification on rotation

  • Integration with applications

  • cert-manager Automation

  • Automatic TLS certificate renewal
  • Let's Encrypt integration
  • Certificate lifecycle management

  • Renewal webhook notifications

  • Custom Rotation Automation 

    // Azure Function timer trigger for rotation
[FunctionName("RotateEncryptionKeys")]
public async Task Run(
    [TimerTrigger("0 0 2 1 */3 *")] TimerInfo timer, // Quarterly at 2 AM
    ILogger log)
{
    log.LogInformation("Starting quarterly DEK rotation");

    var tenants = await _tenantRepo.GetAllActiveTenants();

    foreach (var tenant in tenants)
    {
        try
        {
            await _keyRotationService.RotateTenantDEK(tenant.Id);
            log.LogInformation("DEK rotated for tenant {TenantId}", tenant.Id);
        }
        catch (Exception ex)
        {
            log.LogError(ex, 
                "DEK rotation failed for tenant {TenantId}", 
                tenant.Id);
            // Alert on-call team
            await _alertService.SendAlert(
                $"DEK rotation failed: {tenant.Id}");
        }
    }

    log.LogInformation("Quarterly DEK rotation completed");
}

  • Rotation Triggers

  • Time-Based: Scheduled (daily, monthly, quarterly)
  • Event-Based: Key compromise detected
  • Manual: Operator-initiated
  • Automatic: Key expiration approaching

Code Examples: - Azure Function for rotation - Key Vault rotation policy - cert-manager configuration - Rotation trigger implementations

Diagrams: - Automation architecture - Rotation trigger flow - Azure Function deployment

Deliverables: - Automation implementation - Azure Function code - Rotation policies - Trigger configuration

Topic 10: Monitoring and Alerting

What will be covered:

  • Key Expiration Monitoring
  • Monitor all key expiration dates
  • Alerts: 30 days, 7 days, 1 day before expiration

  • Dashboard showing upcoming expirations

  • Rotation Status Monitoring

  • Track rotation completion status
  • Monitor dual-running period
  • Alert on rotation failures

  • Track old key usage (should decrease)

  • Key Usage Metrics

  • Operations per key version
  • Key age and rotation lag
  • Failed rotation attempts

  • Time to complete rotation

  • Alerting Rules

  • Critical: Key expires in 7 days
  • Warning: Key expires in 30 days
  • Error: Rotation failed
  • Info: Rotation completed successfully

Code Examples: - Monitoring queries (Azure Monitor KQL) - Alert rule definitions - Dashboard JSON (Grafana) - Metrics collection code

Diagrams: - Monitoring architecture - Alert escalation flow - Dashboard mockup

Deliverables: - Monitoring setup guide - Alert configurations - Dashboards - Metrics catalog

CYCLE 6: Emergency Procedures & Compliance (~2,500 lines)

Topic 11: Emergency Key Rotation

What will be covered:

  • Compromised Key Scenario
  • Key leaked or suspected compromise
  • Immediate rotation required (<1 hour)
  • Potential data exposure

  • Incident response coordination

  • Emergency Rotation Procedure 

    1. Detect compromise (alert, report, investigation)
2. Assess scope (which key, what data, exposure window)
3. Emergency approval (security lead + on-call manager)
4. Execute rapid rotation (skip normal procedures)
5. Revoke old key immediately (no dual-running)
6. Force all services to new key
7. Audit data encrypted with compromised key
8. Notify affected parties (if required)
9. Post-incident review
10. Update rotation procedures (lessons learned)

  • Rapid Rotation vs Normal Rotation

Aspect Normal Rotation Emergency Rotation
Approval Automated or routine Security lead + manager
Timeline Gradual (days/weeks) Immediate (<1 hour)
Dual-Running Yes (smooth transition) No (revoke immediately)
Downtime Zero downtime Accept brief downtime
Re-encryption Background, lazy Immediate, forced
Notification Internal team Potentially external
  • Post-Compromise Actions
  • Audit all operations with compromised key
  • Identify potentially affected data
  • Re-encrypt all affected data immediately
  • Notify affected tenants (if required by law)
  • Root cause analysis
  • Update security controls

Code Examples: - Emergency rotation script - Forced key update - Compromise assessment queries - Notification templates

Diagrams: - Emergency rotation workflow - Incident response flow - Post-compromise actions - Decision tree (rotate immediately vs gradual)

Deliverables: - Emergency rotation runbook - Incident response procedures - Compromise assessment guide - Notification templates

Topic 12: Compliance and Audit Evidence

What will be covered:

  • Rotation Audit Trail
  • All rotations logged to Azure Monitor
  • Log includes: Timestamp, key name, old version, new version, actor, reason
  • Immutable audit log (7-year retention)

  • Queryable for compliance audits

  • Compliance Evidence for Rotation 

    Evidence Package for Key Rotation:
- Rotation schedule (documented policy)
- Rotation logs (proof of execution)
- Expiration monitoring (proactive management)
- Rotation test results (validated procedures)
- Incident reports (emergency rotations)
- Access logs (who rotated keys)

  • SOC 2 Controls

  • CC6.1: Logical and physical access (key rotation procedures)
  • CC6.6: Encryption (key rotation frequency)
  • CC6.7: Encryption key management (documented procedures)

  • CC7.2: System monitoring (rotation monitoring)

  • HIPAA Controls

  • §164.312(a)(2)(iv): Encryption and decryption (key management)

  • §164.308(a)(1)(ii)(D): Information system activity review (rotation logs)

  • PCI-DSS Controls

  • Requirement 3.6: Cryptographic key management procedures

  • Requirement 3.6.4: Key rotation at least annually

  • Compliance Reporting

  • Quarterly rotation summary
  • Annual key management review
  • Auditor access to rotation logs
  • Evidence export for assessments

Code Examples: - Audit log queries (compliance reports) - Evidence export scripts - Compliance dashboards

Diagrams: - Audit trail flow - Compliance evidence structure - Control mapping

Deliverables: - Compliance documentation - Audit trail specifications - Evidence collection procedures - Reporting templates

Key Rotation Quick Reference

Rotation Schedules

Key Type Frequency Automation Downtime Re-encryption
TLS Certificates 90 days ✅ Auto (cert-manager) ❌ None N/A
mTLS Certificates Daily ✅ Auto (service mesh) ❌ None N/A
Data Encryption Keys (DEK) Quarterly ✅ Auto (Azure Function) ❌ None ⚠️ Background
Key Encryption Keys (KEK) Annually ⚠️ Semi-Auto ❌ None ✅ Required
Signing Keys Quarterly ⚠️ Manual ❌ None N/A
Service Bus Keys Quarterly ⚠️ Manual ❌ None N/A
Database Credentials Quarterly ⚠️ Manual ❌ None N/A
API Keys Monthly ⚠️ Manual ❌ None N/A
Webhook Secrets Quarterly ⚠️ Manual ❌ None N/A

Rotation Checklist

Pre-Rotation: - ✅ Backup current key metadata - ✅ Verify rotation procedure in non-prod - ✅ Schedule maintenance window (if needed) - ✅ Notify stakeholders - ✅ Prepare rollback plan

During Rotation: - ✅ Generate new key - ✅ Configure dual-running - ✅ Update primary key pointer - ✅ Monitor for errors - ✅ Validate new key working

Post-Rotation: - ✅ Verify new key in use - ✅ Monitor old key usage (should decrease) - ✅ Schedule re-encryption (if needed) - ✅ Log rotation event - ✅ Update key inventory - ✅ Mark old key as retired

Emergency Rotation (Compromised Key)

Timeline: <1 hour from detection to completion

Steps: 1. 0-5 min: Detect and confirm compromise 2. 5-15 min: Emergency approval (security lead) 3. 15-30 min: Generate and activate new key 4. 30-40 min: Revoke compromised key 5. 40-60 min: Force update all services 6. 60+ min: Audit and notify

Summary & Implementation Plan

Implementation Phases

Phase 1: Foundations (Cycle 1) - 2 weeks - Fundamentals and strategy

Phase 2: Encryption (Cycle 2) - 3 weeks - DEK and KEK rotation

Phase 3: Signing (Cycle 3) - 3 weeks - Signing keys and certificates

Phase 4: Secrets (Cycle 4) - 2 weeks - Connection strings, API keys, tokens

Phase 5: Automation (Cycle 5) - 2 weeks - Automated rotation and monitoring

Phase 6: Emergency (Cycle 6) - 2 weeks - Emergency procedures and compliance

Success Metrics

  • Rotation Compliance: 100% keys rotated on schedule
  • Zero Downtime: 0 service interruptions during rotation
  • Automation Rate: >80% rotations automated
  • Detection Time: Expired keys detected within 24 hours
  • Emergency Response: <1 hour for compromised key rotation
  • Audit Coverage: 100% rotations logged

Ownership & Maintenance

  • Security Engineering: Cycles 1-3, 6 (strategy, encryption, signing, emergency)
  • Platform Engineering: Cycles 2, 4-5 (KEK, secrets, automation)
  • Operations: Cycle 5-6 (monitoring, incident response)
  • Compliance: Cycle 6 (evidence, reporting)

Document Status: ✅ Plan Approved - Ready for Content Generation

Target Start Date: Q4 2025

Expected Completion: Q1 2026 (14 weeks)

Owner: Security Engineering Team

Last Updated: 2024-10-30