Skip to content

Pulumi Infrastructure as Code - Audit Trail Platform (ATP)

Code-first Infrastructure as Code with C# — ATP's Pulumi implementation provisions Azure resources (AKS, Key Vault, Storage, Service Bus) with strong typing, stack references, and GitOps integration for complete auditability and compliance.

📋 Documentation Generation Plan

This document will be generated in 12 cycles. Current progress:

Cycle Topics Estimated Lines Status
Cycle 1 Pulumi Overview & ATP Foundation (1-2) ~2,500 ⏳ Not Started
Cycle 2 Project Structure & Stack Architecture (3-4) ~2,500 ⏳ Not Started
Cycle 3 Azure Resources: AKS & Networking (5-6) ~2,500 ⏳ Not Started
Cycle 4 Azure Resources: Storage & Databases (7-8) ~2,500 ⏳ Not Started
Cycle 5 Azure Resources: Security & Keys (9-10) ~2,500 ⏳ Not Started
Cycle 6 Azure Resources: Messaging & Observability (11-12) ~2,500 ⏳ Not Started
Cycle 7 Configuration & Secrets Management (13-14) ~2,000 ⏳ Not Started
Cycle 8 Multi-Environment & Stack References (15-16) ~2,000 ⏳ Not Started
Cycle 9 Azure Pipeline Integration (17-18) ~2,000 ⏳ Not Started
Cycle 10 Testing & Validation (19-20) ~2,000 ⏳ Not Started
Cycle 11 Disaster Recovery & Rollbacks (21-22) ~2,000 ⏳ Not Started
Cycle 12 Operations & Best Practices (23-25) ~2,000 ⏳ Not Started

Total Estimated Lines: ~27,000

Purpose & Scope

This document defines ATP's Pulumi-based Infrastructure as Code (IaC) implementation using C# and Azure Native provider for provisioning and managing all Azure resources required by the Audit Trail Platform across multiple environments, regions, and tenants.

Key Technologies & Patterns - Pulumi C# Runtime: Strongly-typed infrastructure code with full .NET ecosystem - Azure Native Provider: Direct Azure Resource Manager (ARM) API access - Stack References: Cross-stack output consumption for layered architecture - Azure DevOps Pipelines: CI/CD integration for automated deployments - Secrets Management: Azure Key Vault integration with Pulumi secrets provider - State Backend: Azure Blob Storage for Pulumi state with locking - GitOps Integration: Declarative manifests generated via Pulumi - Policy as Code: Azure Policy and OPA/Conftest validation

Detailed Cycle Plan

CYCLE 1: Pulumi Overview & ATP Foundation (~2,500 lines)

Topic 1: Pulumi Fundamentals for ATP

What will be covered: - What is Pulumi? - Infrastructure as code with real programming languages - Pulumi vs. Terraform, Bicep, ARM templates - Declarative outcomes with imperative logic - State management and concurrency control

  • Why C# for ATP?
  • Type safety and compile-time validation
  • Seamless integration with ConnectSoft ecosystem
  • Rich .NET ecosystem (NuGet, async/await, LINQ)
  • Developer familiarity within ATP team

  • Code reuse across application and infrastructure

  • Pulumi Core Concepts

  • Resources: Immutable infrastructure objects
  • Stacks: Environment/region-specific instances
  • Outputs: Cross-stack data sharing
  • Inputs: Strongly-typed configuration
  • Providers: Azure Native, Kubernetes, Random

  • Components: Reusable infrastructure modules

  • Pulumi in ATP Architecture

  • ATP's layered infrastructure model
  • Foundation stacks vs. application stacks
  • Stack dependency graph
  • Multi-region deployment strategy
  • Environment promotion flow

Code Examples: - Simple Azure resource group in C# - Resource with outputs - Stack reference consumption - Configuration reading pattern - Basic Pulumi program structure

Diagrams: - Pulumi architecture overview - ATP stack topology - Resource provisioning lifecycle - State management flow

Deliverables: - Pulumi overview for ATP - C# rationale and benefits - Core concepts glossary - Architecture context

Topic 2: ATP Infrastructure Requirements

What will be covered: - Azure Resources by Category - Compute: AKS clusters, node pools, autoscaling - Networking: VNets, subnets, NSGs, Private Link - Storage: Blob (WORM), Azure SQL, Cosmos DB - Messaging: Service Bus (topics, queues, subscriptions) - Security: Key Vault, Managed HSM, secrets, keys - Identity: Managed Identity, Workload Identity - Observability: Log Analytics, Application Insights, Monitor - Governance: Azure Policy, resource tagging

  • Resource Dependencies & Ordering
  • Foundation resources (RG, VNet, Key Vault)
  • Infrastructure resources (AKS, SQL, Storage)
  • Application configuration (K8s manifests via Pulumi Kubernetes)

  • Cross-stack references for outputs

  • Multi-Environment Strategy

  • Shared foundation stack (dev, test, staging, prod)
  • Environment-specific configurations
  • Resource naming conventions

  • Tag-based cost allocation

  • Compliance & Security Requirements

  • Encryption at rest (all storage)
  • Private endpoints (no public access)
  • Network security groups
  • Diagnostic settings (audit logs)
  • Resource locks (prod protection)
  • RBAC assignments
  • Azure Policy assignments

Code Examples: - Resource naming helper function - Tag generation based on environment - Compliance policy application - Conditional resource creation by environment

Diagrams: - ATP infrastructure topology (Azure services) - Resource dependency graph - Multi-environment stack architecture - Network topology with Private Link

Deliverables: - Complete ATP infrastructure inventory - Dependency mapping - Environment strategy - Compliance checklist

CYCLE 2: Project Structure & Stack Architecture (~2,500 lines)

Topic 3: Pulumi Project Organization

What will be covered: - Repository Structure - ConnectSoft.Audit.InfrastructureModel/ project layout - Stack definitions (Foundation, AKS, Data, Messaging) - Shared modules and components - Configuration files (Pulumi.yaml, Pulumi.{env}.yaml)

  • Project Configuration (Pulumi.yaml)
  • Runtime: dotnet
  • Project metadata
  • Backend configuration (Azure Blob)

  • Plugin versions

  • C# Project Structure 

    ConnectSoft.Audit.InfrastructureModel/
├── Pulumi.yaml
├── Pulumi.dev.yaml
├── Pulumi.test.yaml
├── Pulumi.staging.yaml
├── Pulumi.prod.yaml
├── Program.cs                    # Entry point
├── Stacks/
│   ├── FoundationStack.cs        # RG, VNet, Key Vault
│   ├── AksStack.cs               # AKS cluster
│   ├── DataStack.cs              # SQL, Cosmos, Storage
│   ├── MessagingStack.cs         # Service Bus
│   └── ObservabilityStack.cs    # Monitor, Log Analytics
├── Components/
│   ├── AksClusterComponent.cs
│   ├── BlobStorageComponent.cs
│   ├── KeyVaultComponent.cs
│   └── ServiceBusComponent.cs
├── Config/
│   ├── AksConfig.cs
│   ├── StorageConfig.cs
│   └── NetworkConfig.cs
├── Helpers/
│   ├── NamingHelper.cs
│   ├── TaggingHelper.cs
│   └── SecurityHelper.cs
└── ConnectSoft.Audit.InfrastructureModel.csproj

  • Stack Selection Pattern

  • Dynamic stack selection in Program.cs
  • Environment-based routing

  • Stack instantiation with configuration

  • NuGet Package as Reusable IaC

  • Packaging infrastructure code
  • Versioning strategy
  • Consumption from Azure Pipelines

Code Examples: - Complete Pulumi.yaml configuration - Program.cs with stack routing - Sample stack class structure - Component class pattern - Configuration reading with strong types - Helper utilities implementation

Diagrams: - Project structure tree - Stack selection flow - Component composition model - NuGet packaging workflow

Deliverables: - Project structure template - Configuration patterns - Stack routing implementation - Packaging guidelines

Topic 4: Stack Architecture & Layering

What will be covered: - Layered Stack Strategy - Foundation Stack (Layer 0) - Resource groups - Virtual networks and subnets - Key Vault (foundation) - Managed Identity - Diagnostic settings - Infrastructure Stack (Layer 1) - AKS cluster and node pools - Azure SQL / Cosmos DB - Blob Storage (WORM) - Service Bus namespace - Application Stack (Layer 2) - Kubernetes manifests via Pulumi Kubernetes provider - Helm charts deployment - Service accounts and RBAC - ConfigMaps and Secrets references

  • Stack References
  • Consuming outputs from other stacks
  • StackReference class usage
  • Strong typing of outputs

  • Cross-environment references

  • Stack Naming Convention

  • Pattern: atp-{environment}-{region}-{layer}

  • Examples: atp-dev-westeurope-foundation, atp-prod-eastus-data

  • Stack Dependencies

  • Explicit vs. implicit dependencies
  • Stack update ordering
  • Dependency graph visualization

  • Circular dependency prevention

  • Per-Environment Stack Configuration

  • Pulumi.{env}.yaml files
  • Environment-specific values
  • Secrets encryption
  • Config inheritance patterns

Code Examples: - FoundationStack.cs complete implementation - AksStack.cs with StackReference - Output export pattern - Config class with validation - Stack dependency declaration

Diagrams: - Layered stack architecture - Stack dependency graph - Stack reference flow - Multi-environment topology

Deliverables: - Stack layering guide - Reference pattern documentation - Naming conventions - Configuration templates

CYCLE 3: Azure Resources - AKS & Networking (~2,500 lines)

Topic 5: Azure Kubernetes Service (AKS) Provisioning

What will be covered: - AKS Cluster Configuration - Kubernetes version selection - System node pool (dedicated for system pods) - User node pools (workload-specific) - Availability zones for HA - Autoscaling (cluster and HPA) - Network profile (Azure CNI) - Azure AD integration - Workload Identity enablement - Container Insights integration

  • Node Pool Strategy
  • System node pool: 3 nodes, Standard_D4s_v5
  • Ingestion pool: autoscale 3-10, Standard_D8s_v5
  • Query pool: autoscale 3-20, Standard_D8s_v5
  • Projection pool: autoscale 2-10, Standard_D4s_v5

  • Export pool: on-demand scale

  • AKS Security Hardening

  • Private cluster (no public API server)
  • Azure Policy for Kubernetes
  • Pod Security Standards enforcement
  • Network policies (Calico)
  • Managed Identity for control plane

  • Key Vault CSI driver integration

  • AKS Add-ons & Extensions

  • Azure Monitor Container Insights
  • Azure Key Vault CSI driver
  • Azure Policy add-on
  • Azure Defender for Containers
  • Kured (node reboot daemon)

Code Examples: - Complete AksStack.cs implementation - ManagedCluster resource with full configuration - AgentPool resources for node pools - Private cluster configuration - Workload Identity setup - AKS component class

Diagrams: - AKS architecture with node pools - Network topology (private cluster) - Workload Identity flow - Pod security architecture

Deliverables: - AKS provisioning guide - Node pool strategy - Security hardening checklist - Add-on configuration reference

Topic 6: Network Architecture & Security

What will be covered: - Virtual Network Design - VNet CIDR planning (RFC1918) - Subnet segmentation strategy - AKS system subnet (/24) - AKS workload subnets (/23 each) - Private Link subnet (/27) - Gateway subnet (/27) - Azure SQL subnet (/27) - Service Bus subnet (/27)

  • Network Security Groups (NSGs)
  • Default deny-all approach
  • Allow rules for required flows
  • Service tags usage

  • ASG (Application Security Groups)

  • Private Link & Private Endpoints

  • Azure SQL private endpoint
  • Blob Storage private endpoint
  • Key Vault private endpoint
  • Service Bus private endpoint
  • Cosmos DB private endpoint (if used)

  • Azure Container Registry private endpoint

  • Azure Firewall / Application Gateway (optional)

  • Ingress traffic filtering
  • WAF rules

  • TLS termination

  • DNS Configuration

  • Private DNS zones for Private Link
  • Zone linking to VNet
  • DNS record management

Code Examples: - VirtualNetwork resource with subnets - NetworkSecurityGroup with rules - PrivateEndpoint for Azure SQL - PrivateDnsZone and VirtualNetworkLink - Network component class

Diagrams: - VNet topology with subnets - NSG rules visualization - Private Link architecture - DNS resolution flow

Deliverables: - Network design document - Subnet allocation table - NSG rule matrix - Private Link configuration guide

CYCLE 4: Azure Resources - Storage & Databases (~2,500 lines)

Topic 7: Azure Storage (Blob with WORM)

What will be covered: - Storage Account Configuration - Account kind: StorageV2 - Performance: Premium Block Blobs for hot tier - Replication: ZRS (zone-redundant) or GZRS (geo-zone-redundant) - Access tier: Hot for append store - Encryption: Microsoft-managed keys (or customer-managed via Key Vault) - Network rules: Private endpoint only, deny public access

  • WORM (Write-Once-Read-Many) Implementation
  • Immutability policy configuration
  • Time-based retention policy
  • Legal hold support
  • Version-level immutability

  • Compliance vs. governance mode

  • Container Structure

  • atp-{tenantId}-hot: Append-only segments
  • atp-{tenantId}-exports: Export packages

  • atp-{tenantId}-audit-logs: Operational logs

  • Lifecycle Management

  • Transition to Cool tier (90 days)
  • Transition to Archive tier (365 days)

  • Deletion policies (if allowed)

  • Access Control

  • Managed Identity (no SAS tokens)
  • Azure RBAC (Storage Blob Data Contributor)
  • Private endpoint connectivity

Code Examples: - StorageAccount resource with WORM - BlobContainer with immutability policy - ManagementPolicy for lifecycle - RoleAssignment for Managed Identity - Blob storage component class

Diagrams: - Storage account architecture - WORM immutability model - Lifecycle tier transitions - Access control flow

Deliverables: - Storage configuration guide - WORM implementation spec - Lifecycle policy templates - Access control matrix

Topic 8: Databases (Azure SQL & Cosmos DB)

What will be covered: - Azure SQL Configuration - SQL Server with AAD authentication only - Elastic Pool for multi-tenant databases - Database per tenant or shared with RLS - Performance tier selection (General Purpose, Business Critical) - Backup configuration (PITR, LTR) - Transparent Data Encryption (TDE) - Advanced Threat Protection - Private endpoint connectivity - Failover groups for DR

  • Cosmos DB Configuration (optional for global scale)
  • API selection: SQL API
  • Consistency level: Session or Bounded Staleness
  • Multi-region write configuration
  • Partition key strategy: /tenantId
  • Throughput: Autoscale (400-4000 RU/s)
  • Backup: Continuous (PITR)

  • Private endpoint connectivity

  • Connection String Management

  • Store in Azure Key Vault
  • Retrieve via Managed Identity

  • Connection string rotation strategy

  • Database Initialization

  • FluentMigrator integration (see database-migrations.md)
  • Schema versioning
  • Seed data for dev/test environments

Code Examples: - SqlServer resource with AAD admin - SqlDatabase with elastic pool - CosmosDBAccount resource - SqlDatabase resource with RLS - KeyVault secret for connection string - Database component class

Diagrams: - Azure SQL topology with elastic pool - Cosmos DB multi-region setup - Connection flow with Key Vault - Backup and DR architecture

Deliverables: - Database provisioning guide - Multi-tenancy data isolation strategy - Backup and recovery procedures - Connection management patterns

CYCLE 5: Azure Resources - Security & Key Management (~2,500 lines)

Topic 9: Azure Key Vault Configuration

What will be covered: - Key Vault Provisioning - Standard vs. Premium tier (HSM-backed) - Soft-delete and purge protection (mandatory) - RBAC vs. Access Policies (use RBAC) - Private endpoint connectivity - Diagnostic settings for audit logs - Network rules (deny public, allow trusted services)

  • Key Vault Secrets
  • Connection strings (SQL, Service Bus, Cosmos)
  • API keys for external services
  • Webhook signing secrets
  • Certificate private keys

  • OAuth client secrets

  • Key Vault Keys

  • Data encryption keys (DEK) for application-level encryption
  • Key Encryption Keys (KEK) for envelope encryption
  • Signing keys for tamper-evidence (see tamper-evidence.md)

  • Key versioning and rotation (see key-rotation.md)

  • Key Vault Certificates

  • TLS certificates for ingress
  • mTLS certificates for service-to-service

  • Certificate auto-renewal with Let's Encrypt or DigiCert

  • Managed HSM (for production signing keys)

  • Security domain configuration
  • FIPS 140-2 Level 3 compliance
  • Key backup and restore

  • Multi-admin approval for operations

  • Access Control

  • Managed Identity for workloads
  • Key Vault Administrator role for operators
  • Key Vault Secrets User for application reads
  • Certificate-specific RBAC
  • Audit logging for all access

Code Examples: - KeyVault resource with RBAC - Secret creation from configuration - Key creation for encryption - ManagedIdentity with Key Vault access - HSM provisioning (Premium tier) - KeyVault component class with helper methods

Diagrams: - Key Vault architecture - Key hierarchy (KEK wrapping DEK) - Access control flow - Managed HSM security model

Deliverables: - Key Vault provisioning guide - Secret/key/certificate taxonomy - Access control matrix - HSM operational procedures

Topic 10: Identity & Access Management

What will be covered: - Managed Identity Strategy - System-assigned vs. User-assigned - Per-service identity pattern - Identity lifecycle management

  • Workload Identity for AKS
  • Azure AD Workload Identity setup
  • Federated identity credentials
  • Service Account annotations

  • Token exchange flow

  • Azure AD Integration

  • AKS Azure AD integration
  • Admin group configuration
  • Kubernetes RBAC with Azure AD

  • Conditional Access policies

  • RBAC Assignments

  • Subscription-level roles
  • Resource group-level roles
  • Resource-level roles
  • Custom role definitions (if needed)
  • Just-In-Time (JIT) access for operators

Code Examples: - UserAssignedIdentity resource - FederatedIdentityCredential for Workload Identity - RoleAssignment to Managed Identity - Custom role definition - AKS Azure AD integration config

Diagrams: - Workload Identity architecture - Azure AD integration flow - RBAC hierarchy - Identity lifecycle

Deliverables: - Identity strategy document - Workload Identity setup guide - RBAC assignment matrix - Access governance procedures

CYCLE 6: Azure Resources - Messaging & Observability (~2,500 lines)

Topic 11: Azure Service Bus Configuration

What will be covered: - Service Bus Namespace - Tier selection: Standard or Premium - Premium: dedicated capacity, VNet integration, zones - Zone redundancy for HA - Private endpoint connectivity - Managed Identity authentication (no connection strings in app code)

  • Messaging Topology
  • Topics & Subscriptions (Event-Driven Architecture)
    • audit-events topic: All domain events
    • audit-events-ingestion subscription: Ingestion service
    • audit-events-projection subscription: Projection service
    • audit-events-integrity subscription: Integrity service
    • Subscription filters for routing
  • Queues (Command/Request patterns)
    • export-requests queue: Export command queue
    • policy-decisions queue: Policy evaluation requests

  • Dead-Letter Queues (DLQ)

    • Automatic DLQ per subscription
    • DLQ monitoring and replay strategy

  • Message Delivery Guarantees

  • At-least-once delivery
  • Duplicate detection windows
  • Sessions for ordered processing

  • Message TTL configuration

  • Security & Access Control

  • Managed Identity for send/receive
  • Role assignments (Azure Service Bus Data Sender/Receiver)

  • No Shared Access Signatures (SAS)

  • Monitoring & Diagnostics

  • Diagnostic settings to Log Analytics
  • Metrics: message count, DLQ depth, throttling
  • Alerts for DLQ thresholds

Code Examples: - ServiceBusNamespace resource (Premium tier) - ServiceBusTopic with duplicate detection - ServiceBusSubscription with filters - ServiceBusQueue resource - RoleAssignment for Managed Identity access - Service Bus component class

Diagrams: - Service Bus topology (topics, subscriptions, queues) - Message flow for domain events - DLQ handling architecture - Security model (Managed Identity auth)

Deliverables: - Service Bus design document - Messaging topology specification - Subscription filter rules - DLQ monitoring and replay procedures

Topic 12: Observability Infrastructure

What will be covered: - Log Analytics Workspace - Workspace creation per environment - Retention policy configuration (90-365 days) - Data export for long-term archival - Private Link connectivity

  • Application Insights
  • Application Insights per environment
  • Link to Log Analytics workspace
  • Sampling configuration

  • Instrumentation key storage in Key Vault

  • Azure Monitor

  • Diagnostic settings for all resources
  • Metrics collection
  • Log forwarding to Log Analytics

  • Action groups for alerting

  • Alert Rules

  • AKS node CPU/memory thresholds
  • Storage account capacity alerts
  • Service Bus DLQ depth alerts
  • SQL DTU/CPU alerts

  • Key Vault access anomalies

  • Dashboards

  • Azure Dashboard JSON templates
  • Environment-specific dashboards
  • SLO/SLA tracking dashboards

Code Examples: - LogAnalyticsWorkspace resource - ApplicationInsights resource - DiagnosticSettings for resources - MetricAlert resource - ActionGroup for notifications - Dashboard JSON template

Diagrams: - Observability architecture - Log flow from resources to Log Analytics - Alerting workflow - Dashboard visualization examples

Deliverables: - Observability infrastructure guide - Alert rule catalog - Dashboard templates - Log retention policies

CYCLE 7: Configuration & Secrets Management (~2,000 lines)

Topic 13: Pulumi Configuration Management

What will be covered: - Configuration Files (Pulumi.{env}.yaml) - Structure and syntax - Environment-specific values - Encrypted secrets with pulumi config set --secret - Configuration inheritance patterns

  • Strongly-Typed Configuration in C#
  • Config class pattern
  • Required vs. optional values
  • Validation logic
  • Default values

  • Type conversions

  • Configuration by Environment

  • Dev: minimal resources, relaxed policies
  • Test: production parity, synthetic data
  • Staging: full production parity

  • Production: maximum security, compliance

  • Secret Management Strategy

  • Secrets in Pulumi state (encrypted backend)
  • Secrets in Azure Key Vault (runtime)
  • Secret rotation automation
  • Break-glass secret access procedures

Code Examples: - Pulumi.dev.yaml with configuration - Config class with validation - Reading encrypted secrets - Conditional logic based on environment - Configuration helper utilities

Diagrams: - Configuration flow - Secret lifecycle - Environment-specific variations

Deliverables: - Configuration management guide - Environment configuration templates - Secret management procedures - Validation patterns

Topic 14: Secrets Provider Integration

What will be covered: - Pulumi Secrets Provider - Default: Pulumi Service - Azure Key Vault as secrets backend - State encryption at rest

  • Integration with Azure Key Vault
  • Store Pulumi secrets in Key Vault
  • Retrieve secrets during deployment

  • Managed Identity for Pulumi → Key Vault

  • Secret Injection Patterns

  • Environment variables from Key Vault
  • Kubernetes Secrets from Pulumi

  • CSI driver for runtime secret mounting

  • Secret Rotation Coordination

  • Pulumi stack update triggers
  • Webhook-based rotation notifications
  • Zero-downtime rotation strategies

Code Examples: - Azure Key Vault secrets provider configuration - Secret retrieval during stack update - Kubernetes Secret resource from Key Vault - Secret rotation automation script

Diagrams: - Secrets provider architecture - Secret injection flow - Rotation workflow

Deliverables: - Secrets provider setup guide - Injection pattern catalog - Rotation automation playbook

CYCLE 8: Multi-Environment & Stack References (~2,000 lines)

Topic 15: Multi-Environment Deployment Strategy

What will be covered: - Environment Isolation - Separate Azure subscriptions (or resource groups) - Separate Pulumi stacks - Separate AKS clusters - Separate Key Vaults

  • Promotion Workflow
  • Dev → Test → Staging → Prod
  • Approval gates in Azure Pipelines
  • Configuration diff reviews

  • Drift detection before promotion

  • Environment Parity

  • Resource SKU mapping (dev: small, prod: large)
  • Feature flags per environment
  • Data volume differences

  • Cost optimization strategies

  • Disaster Recovery Environments

  • DR region provisioning
  • Data replication setup
  • Failover testing procedures
  • RTO/RPO targets

Code Examples: - Environment-specific resource sizing logic - Conditional resource creation by environment - Stack selection and routing - DR region stack configuration

Diagrams: - Multi-environment topology - Promotion pipeline - DR architecture

Deliverables: - Environment strategy document - Promotion workflow procedures - DR failover playbook

Topic 16: Stack References & Output Sharing

What will be covered: - Exporting Outputs - Output class usage - Naming conventions for outputs - Output documentation - Sensitive outputs handling

  • Consuming Stack References
  • StackReference class
  • GetOutput method with type casting
  • Cross-environment references (foundation → app)

  • Cross-region references (primary → DR)

  • Output Dependency Management

  • Implicit dependencies via Outputs
  • Explicit DependsOn for ordering

  • Dependency graph visualization

  • Stack Reference Patterns

  • Foundation stack outputs (VNet ID, Key Vault URI)
  • AKS stack outputs (cluster name, kubeconfig)

  • Data stack outputs (SQL connection string reference)

  • Troubleshooting Stack References

  • Missing output errors
  • Type mismatch issues
  • Circular dependency detection

Code Examples: - Export output pattern - StackReference consumption - Cross-stack resource creation - Output dependency chaining

Diagrams: - Stack reference architecture - Output dependency graph - Cross-environment references

Deliverables: - Stack reference guide - Output naming conventions - Dependency management best practices

CYCLE 9: Azure Pipeline Integration (~2,000 lines)

Topic 17: CI/CD Pipeline for Pulumi

What will be covered: - Azure Pipeline Structure - Build stage: compile, test, package - Deploy stage: Pulumi preview/up - Validation stage: policy checks, drift detection

  • Pulumi CLI in Azure Pipelines
  • Pulumi CLI installation task
  • Authentication (Azure Service Principal)
  • Stack selection
  • pulumi preview for validation
  • pulumi up --yes for deployment

  • pulumi refresh for drift detection

  • Pipeline Variables & Secrets

  • Azure Key Vault task for secret retrieval
  • Variable groups per environment

  • Runtime variable substitution

  • Approval Gates

  • Manual approval before production
  • Change Advisory Board (CAB) integration

  • Automated compliance checks

  • Rollback Procedures

  • pulumi stack history for audit
  • pulumi stack rollback for revert
  • Automated rollback on failure

Code Examples: - Azure Pipeline YAML for Pulumi deployment - Pulumi CLI task definitions - Approval gate configuration - Rollback automation script

Diagrams: - CI/CD pipeline stages - Pulumi deployment flow - Approval workflow - Rollback sequence

Deliverables: - Azure Pipeline templates - Deployment procedures - Approval gate configuration - Rollback playbook

Topic 18: GitOps Integration

What will be covered: - Pulumi & GitOps Complementary Roles - Pulumi: Azure infrastructure (VMs, DBs, networks) - GitOps (FluxCD): Kubernetes manifests, Helm charts - Handoff: Pulumi provisions AKS, GitOps deploys apps

  • Pulumi Kubernetes Provider
  • Deploy initial K8s resources (namespaces, RBAC)
  • Bootstrap FluxCD components via Pulumi

  • Configure FluxCD to watch GitOps repo

  • Output Sharing to GitOps

  • AKS kubeconfig export
  • Cluster identity (Workload Identity)
  • Key Vault references for secrets

  • Service endpoints for applications

  • Declarative Infrastructure Workflow

  • Git commit → Azure Pipeline → Pulumi up → AKS updated
  • FluxCD detects AKS changes → reconciles app manifests
  • Complete auditability via Git history

Code Examples: - Pulumi deploying Kubernetes namespace - FluxCD bootstrap via Pulumi - Helm chart deployment via Pulumi (initial setup) - Output generation for GitOps consumption

Diagrams: - Pulumi + GitOps architecture - Handoff workflow - End-to-end deployment flow

Deliverables: - GitOps integration guide - Pulumi-to-FluxCD handoff procedures - Combined audit trail strategy

CYCLE 10: Testing & Validation (~2,000 lines)

Topic 19: Pulumi Testing Strategies

What will be covered: - Unit Testing Infrastructure Code - Pulumi testing framework - Mocking resources - Testing output values - Testing resource properties

  • Integration Testing
  • Ephemeral stack creation
  • Real resource provisioning in test subscription
  • Validation of connectivity and configuration

  • Teardown after tests

  • Policy as Code

  • OPA/Rego policies for Azure resources
  • Conftest integration
  • Policy enforcement in CI pipeline

  • Custom policy rules for ATP compliance

  • Pulumi Policy Packs

  • TypeScript policy packs
  • Mandatory tags validation
  • Encryption enforcement
  • Private endpoint requirement
  • Cost guardrails

Code Examples: - Unit test for stack with Pulumi test framework - Mock resource creation - Integration test script - OPA policy for encryption validation - Pulumi Policy Pack example

Diagrams: - Testing pyramid for IaC - Policy enforcement flow - CI/CD integration with testing

Deliverables: - Testing strategy document - Test framework setup guide - Policy pack library - Compliance validation procedures

Topic 20: Validation & Drift Detection

What will be covered: - Pre-Deployment Validation - pulumi preview output review - Resource change diff analysis - Approval workflows for destructive changes

  • Post-Deployment Validation
  • Resource existence checks
  • Connectivity tests (private endpoints)
  • Configuration validation

  • Security posture verification

  • Drift Detection

  • pulumi refresh scheduled runs
  • Comparison of desired vs. actual state
  • Drift alerting mechanisms

  • Automated drift correction (optional)

  • Continuous Compliance Scanning

  • Azure Policy evaluation
  • Defender for Cloud recommendations
  • Cost anomaly detection
  • Security Center alerts

Code Examples: - Pulumi preview automation script - Post-deployment validation script (Az CLI) - Drift detection scheduled pipeline - Compliance scanning integration

Diagrams: - Validation workflow - Drift detection architecture - Compliance scanning flow

Deliverables: - Validation checklist - Drift detection procedures - Compliance scanning setup

CYCLE 11: Disaster Recovery & Rollbacks (~2,000 lines)

Topic 21: Disaster Recovery Strategy

What will be covered: - Multi-Region Architecture - Primary region: full stack - DR region: standby stack - Data replication (SQL, Blob Storage) - Service Bus geo-replication

  • Pulumi Stack per Region
  • atp-prod-westeurope-* stacks
  • atp-prod-eastus-dr-* stacks
  • Configuration sharing between regions

  • Conditional resource creation (active/standby)

  • Failover Procedures

  • DNS/Traffic Manager switchover
  • Activate DR stack
  • Data synchronization verification

  • Rollback to primary after recovery

  • Backup & Restore

  • Azure SQL automated backups
  • Blob Storage snapshots
  • Key Vault backup

  • Pulumi state backup

  • DR Drills

  • Quarterly failover tests
  • RTO/RPO measurement
  • Runbook validation
  • Post-drill retrospective

Code Examples: - DR region stack with replication - Traffic Manager configuration - Failover automation script - Backup restoration script

Diagrams: - Multi-region DR architecture - Failover sequence diagram - Backup and restore flow

Deliverables: - DR strategy document - Failover runbook - Backup procedures - DR drill schedule

Topic 22: Rollback & Recovery Procedures

What will be covered: - Pulumi Stack Rollback - pulumi stack history to view past updates - pulumi stack rollback to revert changes - Limitations and risks of rollback - Manual vs. automated rollback

  • Blue-Green Infrastructure Deployments
  • Parallel stack deployment
  • Traffic cutover after validation

  • Rollback by switching traffic back

  • Immutable Infrastructure Pattern

  • Replace rather than update
  • Minimize in-place modifications

  • Reduce rollback complexity

  • Recovery from Failed Deployments

  • Partial failure handling
  • Resource cleanup
  • State file corruption recovery
  • Support escalation procedures

Code Examples: - Pulumi rollback command usage - Blue-green deployment script - Failed deployment cleanup script - State file recovery procedure

Diagrams: - Rollback decision tree - Blue-green deployment flow - Recovery workflow

Deliverables: - Rollback playbook - Blue-green deployment guide - Recovery procedures - Incident escalation matrix

CYCLE 12: Operations & Best Practices (~2,000 lines)

Topic 23: Operational Procedures

What will be covered: - Day-to-Day Operations - Stack update frequency - Change management process - Approval workflows - Communication protocols

  • Monitoring Pulumi Deployments
  • Deployment success/failure metrics
  • Duration tracking
  • Resource provisioning errors

  • Alert configuration

  • State Management

  • State backend (Azure Blob Storage)
  • State locking mechanism
  • State backup strategy

  • State recovery procedures

  • Cost Management

  • Resource tagging for cost allocation
  • Cost anomaly detection
  • Budget alerts

  • Cost optimization recommendations

  • Security Operations

  • Secret rotation (see key-rotation.md)
  • Access reviews (Managed Identity permissions)
  • Compliance audits
  • Security incident response

Code Examples: - Cost allocation tag automation - State backup script - Monitoring dashboard query (Kusto/KQL) - Security audit checklist

Diagrams: - Operational workflow - State management architecture - Cost tracking dashboard

Deliverables: - Operations manual - State management guide - Cost management procedures - Security operations checklist

Topic 24: Troubleshooting & Common Issues

What will be covered: - Common Errors - Resource name conflicts - Insufficient permissions (RBAC) - State locking conflicts - Network connectivity issues (Private Link) - Quota limits exceeded

  • Debugging Techniques
  • Enable verbose logging (--logtostderr -v=9)
  • Pulumi state inspection
  • Azure Resource Manager (ARM) deployment logs

  • Azure Activity Log analysis

  • Performance Optimization

  • Parallel resource creation
  • Output caching
  • Stack reference optimization

  • Provider plugin updates

  • Breaking Changes & Upgrades

  • Pulumi CLI version management
  • Azure Native provider updates
  • Breaking change mitigation
  • Testing upgrades in non-prod

Code Examples: - Verbose logging configuration - State inspection commands - Performance profiling script - Provider upgrade procedure

Diagrams: - Troubleshooting flowchart - Debugging workflow - Performance analysis

Deliverables: - Troubleshooting guide - Common errors catalog - Performance tuning checklist - Upgrade procedures

Topic 25: Best Practices & Governance

What will be covered: - Code Organization Best Practices - Modularity and reusability - Component-based architecture - Avoid hardcoding values - Comprehensive documentation

  • Version Control
  • Git branching strategy
  • Commit message conventions
  • Pull request reviews

  • CI checks before merge

  • Security Best Practices

  • Least privilege for Managed Identities
  • No credentials in code or config
  • Secrets encryption in state

  • Regular security audits

  • Compliance & Governance

  • Azure Policy enforcement
  • Resource tagging standards
  • Naming conventions adherence

  • Change documentation (ADRs)

  • Team Collaboration

  • Stack ownership model
  • Cross-team communication
  • Knowledge sharing (runbooks, wikis)

  • On-call procedures

  • Continuous Improvement

  • Retrospectives after incidents
  • Metric-driven optimization
  • Technology updates (Pulumi, Azure)
  • Community engagement

Code Examples: - Component library structure - Git pre-commit hooks for validation - ADR (Architectural Decision Record) template - Team collaboration workflow

Diagrams: - Governance framework - Collaboration model - Continuous improvement cycle

Deliverables: - Best practices handbook - Governance policy - Collaboration guidelines - Continuous improvement plan

Summary of Deliverables

Across all 12 cycles, this documentation will provide:

  1. Foundational Knowledge
  2. Pulumi overview and C# rationale
  3. ATP infrastructure requirements

  4. Stack architecture patterns

  5. Implementation Guides

  6. Complete stack implementations (Foundation, AKS, Data, Messaging, Observability)
  7. Azure resource provisioning (30+ resource types)

  8. Configuration and secrets management

  9. Integration Patterns

  10. Azure Pipeline CI/CD integration
  11. GitOps complementary workflow

  12. Stack references and cross-stack dependencies

  13. Operational Excellence

  14. Testing strategies (unit, integration, policy)
  15. Monitoring and drift detection
  16. Disaster recovery and rollback procedures

  17. Troubleshooting and best practices

  18. Security & Compliance

  19. Zero-trust security implementation
  20. Key Vault and Managed Identity patterns
  21. Azure Policy enforcement

  22. Audit trail and governance

  23. Code Repository Structure

  24. Complete C# project templates
  25. Helper libraries and components

  26. Reusable NuGet package

  27. Operational Runbooks

  28. Day-to-day operations
  29. Incident response
  30. Change management
  31. Cost optimization

Next Steps

  1. Review & Approval: Validate cycle plan with architecture and platform teams
  2. Cycle 1 Generation: Begin content generation for Pulumi fundamentals and ATP foundation
  3. Code Repository Setup: Create ConnectSoft.Audit.InfrastructureModel project structure
  4. Tool Prerequisites: Install Pulumi CLI, Azure CLI, configure state backend
  5. Pilot Stack: Implement Foundation Stack for dev environment as reference

This documentation plan covers the complete Pulumi IaC implementation for ATP, from foundational concepts to advanced operational patterns, ensuring secure, auditable, and compliant infrastructure provisioning across all Azure environments.