Fail-Closed Data Operations

Outcome

An automated data product loads on schedule, verifies its source state, and publishes only a current verified result or an explicit retained verified value.

Who this is for

Anyone who makes a Monday decision off an automated number. A covers or sales dashboard, an inventory or par level report, a weekly digest, an account activity summary, a program recap going to a national team. If a fresh-looking falsehood would cause you to order wrong, staff wrong, or tell a partner something untrue, this is for you. The urgent trigger is a stale source, a failed pull, a scheduler drifting late, or an ambiguous payment event.

Why it is worth money

Automated reports create damage when they confidently summarize yesterday’s data as current. The documented repair protected a digest and revenue surface from false freshness, while preserving the last verified number when retrieval failed.

Proof

BuildoutFeed repaired a stale-data path and produced first fresh data with 1,331 venues and 162 stage-5 venues. It passed 4/4 verification tests and had 11/12 deliverability with one bounce suppressed. The Operator Command Center used the same stale-but-correct rule for Stripe metrics.

Certainty: 9/10. This pattern is supported by a real stale-data incident, concrete repair work, test results, and a second application in the command center.

Stack

Build sequence

1. Set the freshness contract. Define the freshness SLA and a hard stale threshold. BuildoutFeed used an 89-day stale condition for ingestion repair and a 48-hour freshness breaker for digest publication.
   • Checkable artifact: A documented SLA and thresholds.
2. Schedule the load. Run at 04:00 ET without relying on a brittle exact-hour guard.
   • Checkable artifact: A nightly scheduled run.
3. Load and normalize. Load source records, normalize them, and write a freshness stamp.
   • Checkable artifact: A normalized load and freshness record.
4. Verify post-load state. Check row counts, dates, key fields, and expected summary keys.
   • Checkable artifact: A verification result set.
5. Block on failure. Block the digest and notify the operator if verification fails or freshness is outside the window.
   • Checkable artifact: A blocked-publication event and operator notification.
6. Publish the digest. Publish the 7:00 AM digest only after freshness passes. Treat quiet days and zero alerts as deliberate product states.
   • Checkable artifact: A verified digest or an explicit quiet state.
7. Isolate billing events. Accept only allowlisted price IDs and guarded subscription or invoice events.
   • Checkable artifact: A billing event acceptance rule.
8. Retain last verified values. Preserve the last verified dashboard values on refresh failure.
   • Checkable artifact: A dashboard displaying its retained verified value.
9. Retire migration leftovers. Identify temporary loaders or stubs left behind during source changes.
   • Checkable artifact: An explicit retirement decision for each leftover path.

Verification gates

• Freshness gate
  • Block when: The source is beyond the documented 48-hour digest breaker.
  • Warn when: The source is approaching the allowed window or needs operator attention.
  • Proceed when: The source is within the allowed window and has a freshness stamp.
• Post-load verification
  • Block when: Row counts, dates, key fields, or expected summary keys fail.
  • Warn when: A source succeeds with degraded behavior that needs follow-up.
  • Proceed when: All expected verification checks pass.
• Digest gate
  • Block when: Freshness or verification is not confirmed.
  • Warn when: The day is quiet or has zero alerts.
  • Proceed when: Only a verified current digest or explicit quiet state proceeds.
• Billing event gate
  • Block when: The event price ID is not allowlisted or fails subscription or invoice guards.
  • Warn when: An event is accepted but requires operator review.
  • Proceed when: The event passes BOF_STRIPE_PRICE_IDS and applicable guards.
• Refresh failure gate
  • Block when: Retrieval fails.
  • Warn when: One refresh has failed.
  • Proceed when: The last verified value remains visible. After two failures, notify the operator.

Guardrails and failure modes

• Exact-hour scheduler drift
  • What broke: The former exact-hour guard was dead.
  • Why it broke: GitHub runners can start late, so a narrow hour check rejects valid runs.
  • Fix: Run nightly without the exact-hour test and rely on freshness verification.
  • Operating check: Test the schedule for delayed starts and confirm verification, not wall-clock equality, controls publication.
• Stale digest publication
  • What broke: A digest could be sent from frozen data.
  • Why it broke: A delivery schedule alone does not prove source freshness.
  • Fix: Block publication outside the freshness window.
  • Operating check: Require a fresh verification result before the 7:00 AM digest.
• Broken source replacement
  • What broke: The old FDNY source returned 404.
  • Why it broke: The dependency changed or disappeared.
  • Fix: Replace it with DOB NOW LAA.
  • Operating check: Record the source change and verify row and date behavior after the switch.
• GeoSearch instability
  • What broke: GeoSearch was unstable.
  • Why it broke: Address resolution depended on an unreliable source.
  • Fix: Add retry and backoff plus address-to-BBL caching, and warn when failure exceeds 50%.
  • Operating check: Surface a warning when GeoSearch failure is over 50%.
• Shared-account Stripe adoption
  • What broke: Shared Stripe account events could be adopted by the wrong product.
  • Why it broke: A broad webhook listener did not isolate product-specific events.
  • Fix: Use BOF_STRIPE_PRICE_IDS plus subscription and invoice guards, and fail closed.
  • Operating check: Reject an event unless its price ID and applicable event guards pass.
• Last verified replacement
  • What broke: A failed refresh could overwrite a correct stale value with partial or guessed data.
  • Why it broke: Retrieval failure does not establish a replacement value.
  • Fix: Keep the last verified value.
  • Operating check: Do not commit, deploy, or publish any new value on failed retrieval.
• Migration leftovers
  • What broke: A temporary Edge Function loader and later a 410 stub remained after an incident.
  • Why it broke: Source migrations can leave live or confusing paths behind.
  • Fix: Explicitly retire leftovers.
  • Operating check: Maintain a checklist that marks each temporary path retained, disabled, or removed.
• False zero-alert state
  • What broke: A failure could look like an empty result set.
  • Why it broke: The system might synthesize content or silently treat failed collection as no news.
  • Fix: Treat quiet-day and zero-alert state as deliberate only after verification.
  • Operating check: Block an empty digest unless source retrieval and verification succeeded.

Reusable assets

• BuildoutFeed cadence

  04:00 ET load
  freshness verification
  07:00 ET digest

  • Use: Use as the documented daily load and publication order.
• Command Center cadence

  weekdays at 8:00 AM
  1:00 PM
  6:00 PM ET

  • Use: Use for the documented revenue-refresh schedule.
• Publication rule

  commit/deploy/publish only on successful retrieval and verification
  after two refresh failures, notify the operator while retaining the last verified value

  • Use: Use as the fail-closed decision rule.
• Verification checks

  row counts
  dates
  key fields
  expected summary keys

  • Use: Use as the documented post-load check set.
• Failure posture

  fail closed, not fail open

  • Use: Use wherever a source state controls external publication.
• GeoSearch warning

  warn when GeoSearch failure exceeds 50%

  • Use: Use with retry, backoff, and address-to-BBL caching.

Handoff and boundaries

• In scope: freshness contract, scheduled load, normalization, verification, blocked publication, notification, billing event isolation, retained verified values, and temporary-path retirement.
• Out of scope: a promise that the documented 89-day or 48-hour thresholds apply to every product. Those are evidence-backed BuildoutFeed settings.
• Human approval: source replacement, new accepted price IDs, verification exceptions, and any decision to publish after a degraded source event.
• Operator ownership: freshness SLA, schedule health, secret configuration, verification expectations, alert response, and retirement of temporary migration paths.

Commercial frame

• Setup price: Not established in the source record.
• Recurring price: Not established in the source record.
• Delivery window: Not established in the source record.
• Change-control boundary: source replacement, freshness thresholds, verification keys, webhook allowlists, publication exceptions, and schedule changes require operator approval.

Sources

• /home/user/workspace/memory/knowledge/projects/buildoutfeed.md:L18-L29
• /home/user/workspace/memory/knowledge/concepts/operator-command-center.md:L12-L23
• /home/user/workspace/memory/sessions/2026-08-03_2026-08-09/31aaded4/conversation.md:L32-L259
• /home/user/workspace/memory/sessions/2026-08-10_2026-08-16/3fb7b2bf/conversation.md:L176-L202