Skip to content

Split access bootstrap from core infrastructure

Context and Problem Statement

The original infra/ state mixed Contributor-safe resource creation with privileged Azure authorization operations such as:

  • Microsoft.Authorization/roleAssignments/write
  • Microsoft.Authorization/policyExemptions/write

This forced the deploying identity to hold Owner or equivalent elevated rights even when provisioning otherwise ordinary infrastructure. CI service principals and evaluation users commonly have Contributor, not Owner. We need a deployment model that lets core AKS infrastructure provision cleanly first, then applies privileged access bootstrap later with a different identity.

Decision Drivers

  • Contributor-only azd provision must succeed for core infrastructure
  • Privileged authorization changes should not be coupled to whoever happened to run Terraform
  • Grafana, ExternalDNS, and AKS bootstrap access are all optional add-ons around the core cluster
  • A failed role assignment or policy exemption must not poison the main infrastructure state
  • Group-based bootstrap is safer than assigning roles to the current caller implicitly

Considered Options

  • Keep a single infra/ state and gate privileged resources behind booleans
  • Require Owner for all deployments
  • Split privileged authorization resources into a separate Terraform root and bootstrap step

Decision Outcome

Chosen option: "Split privileged authorization resources into a separate Terraform root and bootstrap step", because it gives a clean state boundary between creating infrastructure and granting access to it.

Core Infrastructure (infra/)

infra/ now owns only contributor-safe resources:

  • AKS cluster
  • Log Analytics workspace
  • Azure Monitor workspace
  • Optional Grafana workspace
  • Optional ExternalDNS managed identity + federated credential

Access Bootstrap (infra-access/)

infra-access/ owns privileged authorization resources:

  • AKS Cluster Admin role assignments
  • CNPG probe policy exemption
  • Grafana Monitoring Reader / Monitoring Data Reader / Log Analytics Reader role assignments
  • Grafana Admin role assignments
  • ExternalDNS DNS Zone Contributor role assignment

Operational Model

  1. Run azd provision to create core infrastructure.
  2. If post-provision detects missing Kubernetes bootstrap access, it defers cleanly instead of failing the infra state.
  3. Run ./scripts/bootstrap-access.ps1 with a privileged identity to apply infra-access/.
  4. Rerun azd hooks run postprovision, then continue with azd deploy.

Consequences

  • Good, because Contributor-only pipelines can validate AKS infrastructure without Owner rights
  • Good, because privileged access changes live in a separate Terraform state with separate failure handling
  • Good, because access is granted to explicit user/group object IDs instead of the implicit current caller
  • Neutral, because full solution deployment is now a two-step process when bootstrap access is not pre-granted
  • Bad, because azd up is no longer sufficient for low-privilege identities unless bootstrap access already exists

Migration Notes

  • The enable_external_dns_identity variable defaults to null. When unset, the ExternalDNS identity is created automatically if all three dns_zone_* variables are configured (preserving backward compatibility with dev). Set enable_external_dns_identity = true or false explicitly to override.
  • The infra-access/ Terraform root uses a local filesystem backend. State is stored at infra-access/.tfstate/<cluster-name>.tfstate. Re-running bootstrap-access.ps1 from a different machine is safe — existing role assignments are detected and treated as success.