# Azure Key Vault Rebuild Guide
**Cold Bore Ballistics LLC — Confidential**
Last updated: 2026-06-06

> Full from-scratch rebuild procedure for both CBB Key Vaults. Use this when recovering from a catastrophic loss of the Azure subscription state, or when standing up a net-new environment. For day-to-day operations see `Azure_Key_Vault_Operating_Runbook.md`.

---

## Before You Start

You will need:

1. **Azure CLI** installed and authenticated — see Operating Runbook Prerequisites section for install instructions.
2. **Terraform** >= 1.8.0 installed:

   **Debian / Ubuntu:**
   ```bash
   sudo apt-get install -y gnupg software-properties-common
   wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
   echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
   sudo apt-get update && sudo apt-get install terraform
   ```

   **RHEL / Fedora:**
   ```bash
   sudo dnf install -y dnf-plugins-core
   sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo
   sudo dnf install terraform
   ```

   **Windows:**
   ```powershell
   winget install HashiCorp.Terraform
   ```

   **macOS:**
   ```bash
   brew tap hashicorp/tap
   brew install hashicorp/tap/terraform
   ```

3. **All secret values** — retrieve from the break-glass record in `OneDrive/CBB_Corporate/04_-_Highly_Confidential/` (printed copy) or from a trusted operator who has them. You must have every secret value in the inventory before starting — do not proceed with partial secrets.

4. **Access to the `cbb-infra` repository** and the Terraform remote state storage account (`stcbbtfstate`).

---

## Step 1 — Rebuild `kv-cbb-shared-prod` via Terraform

The shared vault is created by the `services/shared` Terraform service.

1. Navigate to the repo directory, then into the shared service:
   ```bash
   cd cbb-infra/services/shared
   ```

2. Copy the example vars file and fill in your subscription ID:
   ```bash
   cp terraform.tfvars.example terraform.tfvars
   # Edit terraform.tfvars — set subscription_id to the CBB Azure subscription ID
   ```

3. Initialize and apply:
   ```bash
   terraform init
   terraform apply
   ```
   This creates `kv-cbb-shared-prod` and the networking resources. Review the plan before confirming.

4. Populate the secrets. For each secret in the `kv-cbb-shared-prod` inventory (see Architecture Reference), run:
   ```bash
   az keyvault secret set \
     --vault-name kv-cbb-shared-prod \
     --name <secret-name> \
     --value "<secret-value>"
   ```

   Secrets to populate:
   - `ipsec-psk`
   - `vm-db-password`
   - `vm-ssh-public-key`

---

## Step 2 — Rebuild `kv-scope-dope-prod` via Terraform

The ScopeDOPE vault is created by the `services/scope-dope` Terraform service.

1. Navigate to the scope-dope service:
   ```bash
   cd cbb-infra/services/scope-dope
   ```

2. Copy the example vars file. For a rebuild you only need `subscription_id` — secret values are no longer passed as Terraform variables:
   ```bash
   cp terraform.tfvars.example terraform.tfvars
   # Edit terraform.tfvars — set subscription_id only
   ```

3. Initialize and apply:
   ```bash
   terraform init
   terraform apply
   ```
   This creates `kv-scope-dope-prod`, the Container App environment, the VM, storage, ACR, and static web app. Review the plan before confirming.

4. Populate the secrets:
   ```bash
   az keyvault secret set --vault-name kv-scope-dope-prod --name database-url              --value "<postgresql://...>"
   az keyvault secret set --vault-name kv-scope-dope-prod --name storage-connection-string --value "<DefaultEndpointsProtocol=https;...>"
   az keyvault secret set --vault-name kv-scope-dope-prod --name github-token              --value "<ghp_...>"
   az keyvault secret set --vault-name kv-scope-dope-prod --name flask-secret-key          --value "<random-32-byte-value>"
   ```

   To generate a new Flask secret key if none exists:
   ```bash
   openssl rand -base64 32
   ```

---

## Step 3 — Grant the Container App Managed Identity Access

After the Container App is created by Terraform, grant its managed identity the `Key Vault Secrets User` role on `kv-scope-dope-prod`.

1. Get the Container App's managed identity object ID:
   ```bash
   az containerapp show \
     --resource-group rg-cbb-prod \
     --name feedback-service \
     --query "identity.principalId" \
     --output tsv
   ```

2. Get your subscription ID:
   ```bash
   az account show --query id --output tsv
   ```

3. Assign the role:
   ```bash
   az role assignment create \
     --role "Key Vault Secrets User" \
     --assignee <principal-id-from-step-1> \
     --scope "/subscriptions/<subscription-id>/resourceGroups/rg-cbb-prod/providers/Microsoft.KeyVault/vaults/kv-scope-dope-prod"
   ```

---

## Step 4 — Verify

1. Confirm both vaults are visible in the Azure portal under `rg-cbb-prod`.
2. Confirm all secrets are present in each vault:
   ```bash
   az keyvault secret list --vault-name kv-cbb-shared-prod --query "[].name" --output tsv
   az keyvault secret list --vault-name kv-scope-dope-prod --query "[].name" --output tsv
   ```
3. Restart the `feedback-service` Container App and confirm it starts healthy — a startup failure usually means a missing or misconfigured secret.
4. Update `Azure_Key_Vault_Architecture_Reference.md` — change Status from "Planned" to "Active" and update the Last Updated date.

---

## Document Control

| Field | Value |
|---|---|
| **Document owner** | `jschwefel@coldboreballisticsllc.com` |
| **Classification** | Confidential (03_-_Confidential/) |
| **Location** | `OneDrive/CBB_Corporate/03_-_Confidential/Infrastructure/Azure_Key_Vault/Azure_Key_Vault_Rebuild_Guide.md` |
| **Cross-references** | `Azure_Key_Vault_Architecture_Reference.md`; `Azure_Key_Vault_Operating_Runbook.md` |
| **Created** | 2026-06-07 |
| **Last updated** | 2026-06-07 |
| **Next full review due** | 2027-06-07 |

---

## Revision Control Block

| Rev | Date | Changed By | Reviewed By | Approved By | Summary |
|-----|------|------------|-------------|-------------|---------|
| 0.9 | 2026-06-07 | jschwefel | — | — | Initial release |
