# Zoho Receipts Pipeline — Operating Runbook

**Status:** Active
**Last updated:** 2026-06-13
**Owns:** Step-by-step procedures for day-to-day operation.
**Does NOT own:** Resource IDs / as-built state (Architecture Reference), rebuild from scratch (Rebuild Guide), cost analysis (Cost-Impact Analysis), script flag reference (`cbb-utils/zoho/README.md` and `cbb-utils/asset-register/README.md`).

This runbook assumes zero prior platform experience per CLAUDE.md "Instructions must assume zero prior platform experience — MANDATORY". Every step is written for a reader who has never used Zoho Books, Azure Container Apps, or Microsoft Graph before.

---

## 1. Audience

You are operating the Zoho Books receipts pipeline. You may or may not be the original implementer. You have:

- Read access to the OneDrive `Receipts/` folder (via M365 sign-in as a CBB user)
- Owner role on the Azure subscription containing the `rg-cbb-receipts-prod` resource group
- KV Secrets Officer role on both `kv-cbb-receipts-prod` and `kv-cbb-asset-register-prod`
- Admin access to the Zoho Books org
- Edit access to the CBB Asset Register SharePoint list

If any of those are missing, stop and request them before proceeding.

## 2. Daily operating posture

The pipeline is **fully automated**. You do not need to take any daily action when it is healthy.

You receive notifications via:

- **Slack DM** to your `jschwefel@coldboreballisticsllc.com` Slack user
- **Email** from `noreply@coldboreballisticsllc.com` to your mailbox

Three notification types arrive automatically:

| Type | When | Action required |
|---|---|---|
| `needs_review` | A receipt failed validation or extraction | **Yes — see §4** |
| `zero_dollar_receipt` | A $0.00 receipt arrived (free trial, $0 statement) and was filtered out of Zoho per R-15 | **Yes — see §4b** |
| `import_success` (toggleable) | A receipt was successfully imported to Zoho Books | None unless toggled off |

To toggle `import_success` notifications: edit `Receipts/_config/pipeline-config.json` in OneDrive and set `notify_on_success: false`. Change takes effect on the next pipeline run.

## 3. Verifying the pipeline is alive

Run this monthly, or any time you suspect the pipeline has stopped:

### 3.1 Check recent run logs

1. Open OneDrive in a web browser.
2. Navigate to: `CBB_Corporate/03_-_Confidential/Financial_Records/Receipts/_runs/`
3. The most recent date folder should be **today** or **yesterday** (the pipeline runs hourly).
4. Open the latest `run-<HHMMSS>.json` file. It should contain:
   - `status: "completed"` (not `"failed"` or `"interrupted"`)
   - `messages_processed`: a number (0 is OK if no mail arrived)
   - `errors`: an empty array
   - `zero_dollar_filtered`: a number (typically 0; non-zero means free trials were filtered)

If the latest date folder is more than 24 hours old, **see §6: Pipeline appears stopped**.

### 3.2 Check Azure Portal job status (optional)

1. Sign in to `https://portal.azure.com` with your `jschwefel@coldboreballisticsllc.com` account.
2. In the search bar, type `caj-cbb-receipts-ingest-prod` and click the result.
3. Click **Job execution history** in the left sidebar.
4. The latest row should show **Status: Succeeded** with a timestamp within the last hour.

### 3.3 Check Zoho Books expense count

1. Sign in to `https://books.zoho.com` with the Zoho admin account.
2. Click **Purchases** → **Expenses** in the left navigation.
3. Sort by Date descending. The most recent expense should be from within the last 24 hours if any non-zero-dollar vendor receipt arrived during that period.

### 3.4 Check asset event queue

1. Open Azure Portal → `stcbbreceiptsprod` → **Queues** → `asset-candidate-events`.
2. Queue depth should normally be 0 or very small (events are consumed within seconds of arrival).
3. If queue depth is growing over time, the asset service has stopped consuming — see §10b.3.

## 4. Handling a `needs_review` notification

When you receive a `needs_review` Slack DM or email, follow these steps:

### 4.1 Identify the receipt

The notification contains:
- A message-ID
- The OneDrive path to the failed normalization
- A short reason (e.g., "vendor not recognized," "totals do not balance," "no attachment", "PDF render failed", "Zoho rejected expense (5015 amount invalid)")

### 4.2 Open the failed receipt

1. Open OneDrive in a web browser.
2. Navigate to the path provided in the notification.
3. Read the partial Markdown and the embedded error log section.
4. Open the raw email at `Receipts/_raw/<YYYY>/<MM>/<message-id>/original.eml` to see the original mail.
5. If `canonical.pdf` was successfully generated but Zoho rejected the upload, it lives at `Receipts/_raw/<YYYY>/<MM>/<message-id>/canonical.pdf` for manual upload.

### 4.3 Decide the disposition

**Case A — It is a real receipt the pipeline could not parse:**

1. Manually normalize the Markdown to the schema documented in `project_normalized_receipts.md` (Memory).
2. Save to the correct line directory: `Receipts/CBB/<YYYY>/<MM>/VENDOR-REF-YYYYMMDD.md` or `Receipts/ScopeDOPE/<YYYY>/<MM>/...`
3. Manually create the expense in Zoho Books:
   - Click **Purchases** → **Expenses** → **Record Expense**
   - Fill in Date, Vendor, Category (using the doubled-COA convention), Amount, Reference Number = filename stem
   - Attach `canonical.pdf` (or the original source) — drag-drop into the Attachments panel
   - Click **Save**
4. Delete the `needs_review` Markdown.
5. If the vendor's extractor should have worked but didn't, file a Jira issue in project `CBB`.

**Case B — It is not a receipt (e.g., a vendor newsletter, a phishing attempt, a refund):**

1. Move the source email manually within the `finance@` mailbox to a `Not_Receipt/` folder.
2. Delete the `needs_review` Markdown and the `_raw/` directory.
3. Document in `pipeline-config.json` if this vendor regularly sends non-receipt mail.

**Case C — It is a duplicate of an already-imported receipt:**

1. Verify the existing Zoho expense has the same reference number.
2. Move the source email manually to a `Duplicates/` folder.
3. Delete the `needs_review` Markdown.

### 4.4 Confirm resolution

Run §3.1 again to confirm the next pipeline run completes cleanly.

## 4b. Handling a `zero_dollar_receipt` notification

Per R-15, the pipeline filters $0.00 receipts out of Zoho entirely. These are most commonly **free trial signups** and **$0 statement-only receipts** from existing free-tier subscriptions. The notification surfaces them so you can decide whether to track the trial.

### 4b.1 Read the notification

The Slack DM and email contain:
- Vendor
- Date
- Subject line of the source email
- A best-guess **convert date** if the receipt mentions one (e.g., "Your free trial expires on 2026-07-13")
- Link to the source email at `Receipts/_raw/<YYYY>/<MM>/<message-id>/original.eml`
- Link to `canonical.pdf` (rendered for the audit trail even though it's not in Zoho)

### 4b.2 Decide whether to track

**If you intend to convert to a paid subscription:** schedule a calendar reminder for a date a few days before the convert-date so you can decide whether to keep, downgrade, or cancel. Stopgap until R-15 picks a permanent mechanism — use `m365cal` or create the event in Outlook directly. Example:

```bash
m365cal add "Decide on <Vendor> free trial conversion" "<YYYY-MM-DD HH:MM>" \
  --duration 15 \
  --reminder 60 \
  --body "Free trial started <start-date>; converts to paid <convert-date>. Decide: keep / downgrade / cancel. Source: <OneDrive path>"
```

**If you don't intend to convert or the trial is informational only:** no action needed. The source remains preserved in `_raw/` for audit; nothing is in Zoho.

### 4b.3 Confirm no Zoho expense exists for the zero-dollar receipt

Per the design, the pipeline did NOT create a Zoho expense for this receipt. To confirm:

1. Sign in to Zoho Books → Purchases → Expenses.
2. Search for the vendor name; the most recent matching expense should be older than the zero-dollar receipt's date.

If a Zoho expense was created anyway (would be a defect), it's safe to delete via the Zoho UI.

## 5. Adding a new vendor to the recognized list

When a recurring new vendor appears in `needs_review` more than once, add them to the recognized vendors list:

1. Open `infra/cbb-utils/zoho/extractors/` in the repo.
2. Find the closest existing extractor (e.g., `amazon.py`, `namecheap.py`) and copy it as a starting template.
3. Modify the extractor's sender-domain match, the field-extraction regexes, and the category mapping.
4. Add the new vendor's expense category (per the doubled-COA convention) to Zoho Books first:
   - In Zoho: **Settings** → **Chart of Accounts** → **+ New Account**
   - Account Type: Expense
   - Account Name: `<Category Base> - CBB` or `<Category Base> - ScopeDOPE`
5. Test the extractor locally against a saved sample email.
6. Commit and push (cbb-utils commits go directly to `main` per CLAUDE.md).
7. The next hourly run will pick up the new extractor and handle the vendor automatically.

## 6. Pipeline appears stopped

When the most recent `_runs/` folder is more than 24 hours old, or when `caj-cbb-receipts-ingest-prod` shows **Failed** status in the Azure Portal, follow these steps in order:

### 6.1 Check Container Apps Job logs

1. In Azure Portal, open `caj-cbb-receipts-ingest-prod`.
2. Click **Job execution history**.
3. Click the most recent failed row.
4. Click **Console logs**.
5. Read the last 50 lines for the error.

### 6.2 Diagnose by error class

**`Authentication failed` against Microsoft Graph:**
- The managed identity's Graph permission was revoked, or the ApplicationAccessPolicy was deleted.
- Verify in Azure Portal → Enterprise Applications → search for the managed identity name → Permissions tab → confirm `Mail.ReadWrite` is still granted with admin consent.
- Re-create the ApplicationAccessPolicy via Exchange Online PowerShell (see Rebuild Guide §7.3).

**`Authentication failed` against Zoho:**
- The Zoho refresh token was revoked or expired.
- Sign in to Zoho Developer Console (`https://api-console.zoho.com/`) as the Zoho admin user.
- Select the **`Self Client`** for CBB receipts.
- Click **Generate Code** → select the scopes listed in Architecture Reference §5.2 → set duration to 10 minutes → click **Create**.
- Run the OAuth code-exchange script in `cbb-utils/zoho/` (see that README) to convert the grant to a refresh token.
- Update the `zoho-books-refresh-token` secret in Key Vault.
- Trigger a manual job run (see §6.3).

**`Key Vault access denied`:**
- The managed identity lost its RBAC role assignment.
- In Azure Portal → `kv-cbb-receipts-prod` → Access control (IAM) → confirm each pipeline job's managed identity has `Key Vault Secrets User`.
- If missing, re-add via the Add role assignment button.

**`PDF render failed` (weasyprint error in logs):**
- HTML body was malformed in a way `weasyprint` cannot handle. This is rare.
- The receipt lands in `_needs_review/` with the `weasyprint` error captured.
- Manual disposition per §4.3 Case A — open the source, save as PDF manually, attach in the Zoho UI.

**`Zoho rejected expense — code 5015`:**
- The receipt's extracted amount was zero or invalid. The zero-dollar filter should have caught this — investigate the extractor.
- Look at the extracted JSON in `Receipts/_raw/.../extracted.json`. If `total == 0`, the filter logic has a bug.
- File a Jira issue in project `CBB`.

**`Zoho rejected receipt attachment — code 40`:**
- The PDF rendering stage produced an output Zoho doesn't accept (extremely rare; should only happen if `weasyprint` produced a corrupt PDF).
- The expense is in Zoho without the attachment. Manual fix: download `canonical.pdf` from `_raw/`, attach via Zoho UI.

**`Network timeout` / `connection refused`:**
- Transient. Wait for the next scheduled run. If two consecutive runs fail with the same error, escalate.

**`No such mailbox` / `404 on finance@...`:**
- The shared mailbox was deleted, renamed, or moved. Verify in M365 admin center.

**`Quota exceeded` against Zoho:**
- Zoho Standard caps at 2,000 API requests/day. At normal volume this is unreachable.
- If this fires, something is in a retry loop. Read the run logs from the last several runs to identify the loop and pause the job:
  - In Azure Portal → the job → Configuration → set **Trigger mode** to Manual temporarily.
- After the loop is identified and fixed in code, restore Schedule trigger.

### 6.3 Trigger a manual job run

1. In Azure Portal, open the job.
2. Click **Run now** at the top.
3. Watch **Job execution history** for the new row.
4. Verify Console logs show successful completion.

## 7. Rotating credentials

### 7.1 Zoho refresh token (on revoke or annual policy)

See §6.2 "Authentication failed against Zoho" above.

### 7.2 Zoho client ID and client secret

1. Sign in to Zoho Developer Console as the Zoho admin user.
2. Find the **Self Client** for CBB receipts.
3. There is **no in-place secret regeneration** for Self Clients — to rotate, create a new Self Client with the same scopes, transition the refresh token to it, then delete the old.
4. Update `zoho-books-client-id` and `zoho-books-client-secret` in Key Vault.
5. Re-run the OAuth code-exchange flow to mint a new refresh token using the new client.
6. Update `zoho-books-refresh-token` in Key Vault.
7. Trigger a manual job run to verify.

### 7.3 Managed identity / Graph permission

Managed identity credentials rotate automatically — no operator action needed. If the Graph permission grant is accidentally revoked, re-grant via Azure Portal → Enterprise Applications → the managed identity → Permissions → Microsoft Graph → `Mail.ReadWrite` → Grant admin consent.

## 8. Running a backup restore test

Quarterly, verify the backups in `stcbbreceiptsprod` are actually restorable. **Do not restore into production Zoho** — restore into a sandbox project to verify integrity.

1. Azure Portal → `stcbbreceiptsprod` → `zoho-backup-weekly` container.
2. Find the most recent week's ZIP.
3. Download it (note: if older than 365 days, it is in Archive tier and requires a rehydration to Hot first — see §8.1).
4. Inspect the contents: expenses, contacts, journals, chart of accounts should all be present.
5. Optional: import a small subset into a Zoho Books sandbox to verify the format is still importable.

### 8.1 Rehydrating an Archive-tier blob

1. Right-click the blob → **Change tier** → select **Hot** → click **Save**.
2. Wait 1–15 hours (Standard priority) or pay extra for High priority (rehydration in 1 hour).
3. Once status is Hot, download as normal.
4. Set tier back to Archive after use if no longer needed.

## 9. Monthly P&L per-product-