Changelog

A running log of decisions.

Planning-phase entries record architectural choices, not commits. Once the build phase begins, this becomes a conventional release log.

2026-05-21 · stack decided by external review · empirical bake-off deferred

Decide the stack on the page: the review brief becomes a full decision dossier

  • Reframed how the stack gets chosen. Under cutover time pressure we are not building the slice twice now. The stack will be decided by external analytical review of a deepened brief; the empirical bake-off is deferred — retained as the eventual first-build acceptance spec + fixtures plan for whichever stack wins.
  • docs/stack-review-brief.md rewritten from a brief into a self-contained decision dossier: full per-candidate composition tables (A Elixir / B Go+SvelteKit / C TS end-to-end); a divergence matrix isolating the ~13 surfaces where the stacks actually differ; a shared / swap-path table showing most of the system is already settled and every choice reversible; the representative slice reframed as reason-about-not-build; and explicit decision dimensions (build-effort, change-cost, problem-fit, agent velocity, cutover risk, reversibility) to score against.
  • Settled: the live disagreement is A (Elixir/LiveView) vs B (Go+SvelteKit); C is retained as the documented alternative, not a third build.
  • docs/decision-bakeoff.md reconciled with a status banner + resolved open questions so the planning hub isn't self-contradictory.
2026-05-21 · build-repo strategy + external stack review

The build lands in a separate repo; the stack goes out for external review

  • Decided: the actual build is a new, private repo that references this planning hub — not built inside it. The planning repo stays the source of decisions and specs; once the stack is locked we stand up the build repo and the environment there.
  • New doc (docs/stack-review-brief.md): a self-contained briefing to hand to external models for an independent stack recommendation. Inlines the project sketch, the locked decisions, the constraints, the three candidates (A Elixir / B Go+SvelteKit / C TS end-to-end), the "UI strategy decides the backend" lens, the bake-off plan, and six pointed questions.
  • All 2026-05-21 planning work committed to main and pushed to origin (private).
2026-05-21 · import contract decision

The ingestion front door: one pipeline, provenance-safe merge

  • New doc (docs/decision-import.md) settles how data gets in and becomes the identity graph — the first of the three decision docs hanging off the API-first reframe, and the path the cutover runbook calls "normal ingestion."
  • One pipeline, two channels (operator upload + API POST /api/v1/imports) and two payload classes (universe imports that create the graph vs enrichment imports that augment existing entities only). Import is an API endpoint first; the UI uploader is a client of it (dogfood rule).
  • Five-stage contract: LAND (verbatim to file_rows.payload) → PARSE (pure, versioned parser → extraction record) → RESOLVE (natural-key hashes) → MERGE (idempotent upsert with provenance) → RECORD. Re-ingestion re-runs over preserved payload, so adding a column later is a re-extraction, not a re-upload.
  • 400-column map: preserve everything verbatim, promote a curated subset by field family; phone1..phoneN fan out to one phone_numbers row each — never phone1/2/3 columns.
  • Hash recipes decided (entity-model open #2/#3): property_hash from CASS-normalized address (not lat/lng); identity_hash = name + property_hash, deliberately biased to under-merge — safe because send-dedup is keyed on phone_e164, not identity. Over-merging two real people is the dangerous error and is avoided.
  • Compliance imports stay append-only events (consent_events / suppression_list), never flags. Scalar conflicts resolve last-writer-wins by (confidence, recency) with payload as the verbatim audit fallback.
  • Cutover backfill is the same family: legacy history, suppression, and the suppression seed are registered import jobs, not bespoke scripts — which is what lets the cutover runbook treat C1 as ordinary ingestion.
2026-05-21 · minimum-viable-cutover scope

Define the smallest system that can replace legacy — build it, cut over, then finish v1

  • New doc (docs/decision-mvp-cutover.md) defines the MVC: the minimum that satisfies cutover phase C0 (compliant send) plus the legacy bridge. Strategy: build MVC → cut over → build the rest of v1 on a live system, rather than finishing all eleven phases before touching legacy.
  • Inclusion test: a capability is in scope only if it serves C0 (compliant send) or the two-way bridge — everything else is explicitly deferred, "no exceptions for while we're in there."
  • One initial message per lead per campaign — the retext/nurture/direct-contact engine, sequence builder, and reply UX are entirely out of MVC, removing a large surface cleanly.
  • Compliance + the three-layer dedup are entirely in scope (the cardinal rule). Grading, geo/maps, MMS, multiplayer, theming, and the node builder are deferred to named later phases.
  • Maps to thin slices of Phases 1/2/3/6 + the Platform API slice + a sliver of 8, then Phase M.
2026-05-21 · legacy cutover runbook

A reversible, strangler-style migration off the legacy SMS systems

  • New doc (docs/decision-cutover.md) fills the roadmap's biggest gap — there was no migration phase at all. Strangler/phased, never big-bang: C0 shadow → C1 backfill → C2 shadow-run → C3 canary → C4 ramp → C5 full → C6 sunset; only C6 is irreversible.
  • Cardinal rule: compliance continuity beats every other goal — the new system must never text an opted-out / DNC / litigator / in-cooldown number, including numbers the legacy systems contacted.
  • Designs the recent-contact suppression seed: seed MAX(cooldown) (~90d — not 45; "45 days" is the cross-vertical cooldown, the same-vertical default is 90) of legacy (from,to) pairs into global_from_to_history + the dedup store; idempotent, re-runnable, run as the last pre-send step.
  • Parallel-run bridge: shared global_from_to_history dual-write + bidirectional near-real-time opt-out (most-restrictive-wins); temporary by design, severed at C6.
  • New Phase M added to the roadmap, gated on the minimum-viable-cutover engine, not the full feature set.
2026-05-21 · stack bake-off spec

Choose the stack on evidence, not on paper

  • New doc (docs/decision-bakeoff.md): build one thin vertical slice twice — once per candidate stack — against identical fixtures and measurements, so the only variable is the stack itself.
  • The slice: synthetic 100k-row file → COPY ingest → enqueue → simulated send (with retries + a STOP) → live grid update → self-documenting API read. It settles both open forks (UI strategy and engine language) at once.
  • Primary metric is build-effort + change-cost (time three small extensions after a 2–3 day gap) + cold-read clarity (a fresh agent locates where send / contract / realtime live) — not throughput, which Postgres dominates regardless of stack.
  • Hard constraints honored even in the spike: no direct-table write UI, the simulated sender behind one interface, a generated OpenAPI contract, service boundaries respected.
  • The shared fixtures + OpenTelemetry harness are kept — they become the permanent regression benchmark and the first version of the sandbox-as-product surface.
2026-05-21 · stack comparison (not locked)

Candidate stacks weighed; recommendation made, decision deferred to the bake-off

  • New doc (docs/decision-stack.md), the phase-0 exit criterion, walks the candidates against the real constraints — now including API-first and an AI-agent-built team with no incumbent language.
  • Decision lens: your UI strategy decides your backend. LiveView-first ⇒ Elixir; a rich JS UI ⇒ the backend is free (it talks to the UI only through the OpenAPI contract). The stated requirements (great UI fast, node builder, multiplayer, agent velocity) pull toward a rich JS UI — which means giving up LiveView, Elixir's biggest differentiator here.
  • Candidates: A Elixir/Phoenix+LiveView; B Go (Huma+sqlc+River) + SvelteKit — the lean for this team, lowest cutover risk; C TypeScript end-to-end + a Rust/Go worker for maximum agent velocity.
  • Directus ruling: acceptable only as a read-only explorer over an api schema of views — never a write path on base tables, which would bypass append-only consent, event emission, audit, and dedup. Reaffirms decision-api's "DB-as-API over curated views only."
  • The "100M-row" worry is stack-independent (COPY, partitions, matviews, pg_trgm, ClickHouse later); language matters for concurrent send rate and presence, not row count. Includes a component-by-component "optimal choice" table with a swap path for every pick. Not locked — owner's call after the bake-off.
2026-05-21 · API-first reframe

esd.quest reframed as an API-first platform; the UI becomes one client

  • New doc (docs/decision-api.md) reframes esd.quest from "the UI is the product" to a headless hub: external systems push leads in, pull lead/message state out, and receive funneled leads — bidirectionally — over a documented API, and the operator UI is one client of that same contract (the dogfood rule).
  • This amends vision principle #6; the wording is proposed in the doc, pending owner sign-off — vision.md is not yet edited.
  • Primary contract: REST + OpenAPI (generated, versioned, /api/v1, additive-only within a major). Optional GraphQL / DB-as-API read layer over curated views only, never base tables.
  • Machine auth defined separately from passkeys: scoped, hashed API keys (leads:read, import:write, …) + HMAC-signed outbound webhooks; every call audited; per-key rate limits.
  • Two-way sync rules set before the dependent docs: idempotency keys, an external_refs mapping table, per-field ownership (esd.quest owns consent/message-derived state), event-sourced leads, webhooks-over-polling. Promotes the Platform API to a Phase 1 track and constrains the stack to auto-OpenAPI or spec-first codegen.
2026-05-14 · message store decision

Native Postgres partitions confirmed as the v1 storage engine

  • New doc (docs/decision-message-store.md) walks the four candidates — native partitions, TimescaleDB, ClickHouse, hot/cold hybrid — against the seven actual queries the system issues, and recommends keeping the v0.2 lock.
  • Confirmed: messages PARTITION BY RANGE(created_at) monthly, PK (id, created_at).
  • Locked-in discipline: PK shape kept Timescale-compatible; FKs into messages are forbidden so a future create_hypertable() is a same-day migration, not a rewrite.
  • Added to migration 001 scope: a campaign_send_stats rollup table maintained from the send pipeline, so Q4 deliverability analytics never degenerate into ad-hoc aggregation over the partitioned log.
  • Named revisit triggers for adopting TimescaleDB later (≥100M rows/install, OR compression need, OR continuous-aggregate need) and for ClickHouse (aggregate-across-installs analytics, OR volume that breaks Timescale).
2026-05-14 · critique of prior art

Honest accounting of where our reference pattern falls short

  • New doc (docs/critique-prior-art.md) catalogues what we keep from prior art, where it compromised due to its history, and where esd.quest can do strictly better.
  • Six v0.2 locks flagged for re-opening: message store (TimescaleDB / ClickHouse vs native partitions), carrier abstraction, free-text join columns → enums, identity-hash naivety, denormed timezone strings, shared cross-system history sunset.
  • Twelve greenfield wins identified: real multi-tenancy, multi-carrier abstraction, lead-event sourcing (not just consent), JSON Schema validation on every JSONB column, named materialized views as first-class read models, GDPR/CCPA tombstone pattern, bloom-filter dedup front, structured number warming schedules, event bus over HTTP callbacks, single-VM-default deploy, PII column encryption at rest, audit-log-via-middleware.
2026-05-14 · v0.2 entity model + diagrams

Entity model locked at v0.2; visual planning surface live

  • Read-only audits of two prior systems completed; reports retained for the project but kept off the public surface.
  • Locked: identity-first graph (identities × identity_phones × phone_numbers × email_addresses × properties) with provenance, confidence, and source on every join row.
  • Locked: leads is a junction UNIQUE(identity_id, vertical_id, property_id) — same person, multiple verticals, no duplication.
  • Locked: conversations keyed by (account_id, from_phone_e164, to_phone_e164).
  • Locked: messages PARTITION BY RANGE(created_at) monthly (reopened, see critique).
  • Locked: consent is event-sourced (consent_events append-only); no boolean DNC flags.
  • Locked: dedup is a service with three layers — hot box → FROM-TO cache → authoritative store.
  • Locked: single environment enum (prod/sandbox/test) replaces orthogonal flags.
  • Entity model rewritten as v0.2 with column-level detail (docs/entities.md).
  • New /diagrams.html page added: ER diagrams, send pipeline, dedup cascade, inbound + opt-out, multiplayer flow, secrets envelope, deployment topology — rendered via Mermaid.
  • Removed references to specific lead-source vendors from public docs and roadmap.
2026-05-13 · planning hub initialized

Project established

  • esd.quest domain carved into its own document root
  • Splash page, roadmap, changelog, and docs scaffold deployed
  • Vision, principles, and anti-patterns drafted (v0.1)
  • Initial entity sketch published: files, leads, phones, addresses, campaigns, templates, numbers, crusades, schedules, messages, conversations, opt_outs
  • Integration inventory drafted: Twilio (multi-account), Postmark, DNC.com, Google, ImageRouter, OpenRouter, GoHighLevel
  • Stack still open — backend, frontend, realtime, and ORM are unresolved