Fail-Closed Data Operations

Outcome

An automated data product loads on schedule, verifies its source state, and publishes only a current verified result or an explicit retained verified value.

Who this is for

Operators of dashboards, digests, or data-fed revenue views where a fresh-looking falsehood would cause a bad decision. The urgent trigger is a stale source, failed retrieval, drifting scheduler, or ambiguous billing event.

Why it is worth money

Automated reports create damage when they confidently summarize yesterday’s data as current. The documented repair protected a digest and revenue surface from false freshness, while preserving the last verified number when retrieval failed.

Proof

BuildoutFeed repaired a stale-data path and produced first fresh data with 1,331 venues and 162 stage-5 venues. It passed 4/4 verification tests and had 11/12 deliverability with one bounce suppressed. The Operator Command Center used the same stale-but-correct rule for Stripe metrics.

Certainty: 9/10. This pattern is supported by a real stale-data incident, concrete repair work, test results, and a second application in the command center.

Stack

Build sequence

1. Set the freshness contract. Define the freshness SLA and a hard stale threshold. BuildoutFeed used an 89-day stale condition for ingestion repair and a 48-hour freshness breaker for digest publication.
   • Checkable artifact: A documented SLA and thresholds.
2. Schedule the load. Run at 04:00 ET without relying on a brittle exact-hour guard.
   • Checkable artifact: A nightly scheduled run.
3. Load and normalize. Load source records, normalize them, and write a freshness stamp.
   • Checkable artifact: A normalized load and freshness record.
4. Verify post-load state. Check row counts, dates, key fields, and expected summary keys.
   • Checkable artifact: A verification result set.
5. Block on failure. Block the digest and notify the operator if verification fails or freshness is outside the window.
   • Checkable artifact: A blocked-publication event and operator notification.
6. Publish the digest. Publish the 7:00 AM digest only after freshness passes. Treat quiet days and zero alerts as deliberate product states.
   • Checkable artifact: A verified digest or an explicit quiet state.
7. Isolate billing events. Accept only allowlisted price IDs and guarded subscription or invoice events.
   • Checkable artifact: A billing event acceptance rule.
8. Retain last verified values. Preserve the last verified dashboard values on refresh failure.
   • Checkable artifact: A dashboard displaying its retained verified value.
9. Retire migration leftovers. Identify temporary loaders or stubs left behind during source changes.
   • Checkable artifact: An explicit retirement decision for each leftover path.

Verification gates

• Freshness gate
  • Block when: The source is beyond the documented 48-hour digest breaker.
  • Warn when: The source is approaching the allowed window or needs operator attention.
  • Proceed when: The source is within the allowed window and has a freshness stamp.
• Post-load verification
  • Block when: Row counts, dates, key fields, or expected summary keys fail.
  • Warn when: A source succeeds with degraded behavior that needs follow-up.
  • Proceed when: All expected verification checks pass.
• Digest gate
  • Block when: Freshness or verification is not confirmed.
  • Warn when: The day is quiet or has zero alerts.
  • Proceed when: Only a verified current digest or explicit quiet state proceeds.
• Billing event gate
  • Block when: The event price ID is not allowlisted or fails subscription or invoice guards.
  • Warn when: An event is accepted but requires operator review.
  • Proceed when: The event passes BOF_STRIPE_PRICE_IDS and applicable guards.
• Refresh failure gate
  • Block when: Retrieval fails.
  • Warn when: One refresh has failed.
  • Proceed when: The last verified value remains visible. After two failures, notify the operator.

Guardrails and failure modes

• Exact-hour scheduler drift
  • What broke: The former exact-hour guard was dead.
  • Why it broke: GitHub runners can start late, so a narrow hour check rejects valid runs.
  • Fix: Run nightly without the exact-hour test and rely on freshness verification.
  • Operating check: Test the schedule for delayed starts and confirm verification, not wall-clock equality, controls publication.
• Stale digest publication
  • What broke: A digest could be sent from frozen data.
  • Why it broke: A delivery schedule alone does not prove source freshness.
  • Fix: Block publication outside the freshness window.
  • Operating check: Require a fresh verification result before the 7:00 AM digest.
• Broken source replacement
  • What broke: The old FDNY source returned 404.
  • Why it broke: The dependency changed or disappeared.
  • Fix: Replace it with DOB NOW LAA.
  • Operating check: Record the source change and verify row and date behavior after the switch.
• GeoSearch instability
  • What broke: GeoSearch was unstable.
  • Why it broke: Address resolution depended on an unreliable source.
  • Fix: Add retry and backoff plus address-to-BBL caching, and warn when failure exceeds 50%.
  • Operating check: Surface a warning when GeoSearch failure is over 50%.
• Shared-account Stripe adoption
  • What broke: Shared Stripe account events could be adopted by the wrong product.
  • Why it broke: A broad webhook listener did not isolate product-specific events.
  • Fix: Use BOF_STRIPE_PRICE_IDS plus subscription and invoice guards, and fail closed.
  • Operating check: Reject an event unless its price ID and applicable event guards pass.
• Last verified replacement
  • What broke: A failed refresh could overwrite a correct stale value with partial or guessed data.
  • Why it broke: Retrieval failure does not establish a replacement value.
  • Fix: Keep the last verified value.
  • Operating check: Do not commit, deploy, or publish any new value on failed retrieval.
• Migration leftovers
  • What broke: A temporary Edge Function loader and later a 410 stub remained after an incident.
  • Why it broke: Source migrations can leave live or confusing paths behind.
  • Fix: Explicitly retire leftovers.
  • Operating check: Maintain a checklist that marks each temporary path retained, disabled, or removed.
• False zero-alert state
  • What broke: A failure could look like an empty result set.
  • Why it broke: The system might synthesize content or silently treat failed collection as no news.
  • Fix: Treat quiet-day and zero-alert state as deliberate only after verification.
  • Operating check: Block an empty digest unless source retrieval and verification succeeded.

Reusable assets

• BuildoutFeed cadence

  04:00 ET load
  freshness verification
  07:00 ET digest

  • Use: Use as the documented daily load and publication order.
• Command Center cadence

  weekdays at 8:00 AM
  1:00 PM
  6:00 PM ET

  • Use: Use for the documented revenue-refresh schedule.
• Publication rule

  commit/deploy/publish only on successful retrieval and verification
  after two refresh failures, notify the operator while retaining the last verified value

  • Use: Use as the fail-closed decision rule.
• Verification checks

  row counts
  dates
  key fields
  expected summary keys

  • Use: Use as the documented post-load check set.
• Failure posture

  fail closed, not fail open

  • Use: Use wherever a source state controls external publication.
• GeoSearch warning

  warn when GeoSearch failure exceeds 50%

  • Use: Use with retry, backoff, and address-to-BBL caching.

Handoff and boundaries

• In scope: freshness contract, scheduled load, normalization, verification, blocked publication, notification, billing event isolation, retained verified values, and temporary-path retirement.
• Out of scope: a promise that the documented 89-day or 48-hour thresholds apply to every product. Those are evidence-backed BuildoutFeed settings.
• Human approval: source replacement, new accepted price IDs, verification exceptions, and any decision to publish after a degraded source event.
• Operator ownership: freshness SLA, schedule health, secret configuration, verification expectations, alert response, and retirement of temporary migration paths.

Commercial frame

• Setup price: Not established in the source record.
• Recurring price: Not established in the source record.
• Delivery window: Not established in the source record.
• Change-control boundary: source replacement, freshness thresholds, verification keys, webhook allowlists, publication exceptions, and schedule changes require operator approval.

Sources

• /home/user/workspace/memory/knowledge/projects/buildoutfeed.md:L18-L29
• /home/user/workspace/memory/knowledge/concepts/operator-command-center.md:L12-L23
• /home/user/workspace/memory/sessions/2026-08-03_2026-08-09/31aaded4/conversation.md:L32-L259
• /home/user/workspace/memory/sessions/2026-08-10_2026-08-16/3fb7b2bf/conversation.md:L176-L202