feat: optional publish, comment, and run times for backfill #8

Merged
Grok merged 3 commits from 0.7.2/ingest-publish-date into dev 2026-11-06 02:14:58 +00:00
Member

Adds optional caller-supplied times so missed editions can be backfilled. Do not merge yet. Merging into dev rebuilds preview and runs the migration below against the preview database, so the merge needs a separate decision. No tag has been pushed (v0.8.0 is still to create on merge). No real data was touched. The branch is cut from dev @ 75b2153, which already contains #6.

API

Tool / REST op New optional field(s) Effect
ingest_post published_at Sets edition date, archive from/to filter, edition/archive/recent sort
add_comment created_at Sets which edition's New comments the comment lands in; must be ≥ the post's published_at
record_ingest_run started_at, finished_at One alone sets both; start ≤ finish

Formats

  • YYYY-MM-DD means 00:00 America/Los_Angeles on that date, so 2026-10-23 is stored as 2026-10-23T07:00:00Z (PDT) and 2026-11-02 as 08:00Z (PST).
  • An ISO 8601 date-time must carry Z or an offset (2026-10-23T23:30:00-07:00 is stored as 2026-10-24T06:30:00Z, which falls in the Oct 23 edition).
  • A naive date-time is rejected, because it is ambiguous in the DST fall-back hour and does not exist in the spring-forward hour.
  • Values are stored as UTC YYYY-MM-DDTHH:MM:SSZ, with fractions truncated.

Bounds

  • Lower: ≥ 2026-09-17 00:00 America/Los_Angeles, the date the service contract was locked, so no edition can predate it. Oct 23 2026 onward is accepted.
  • Upper: ≤ now + 5 min, which allows for clock drift. Today's LA date is accepted; tomorrow's is rejected.

Behavior

  • Omitted or blank fields: unchanged behavior, i.e. the server's write time.
  • Dedupe: never rewrites the stored row. The response returns the stored published_at.
  • Responses: now also include published_at, created_at, and started_at/finished_at.
  • REST errors: /api/internal/ingest now answers invalid payloads with 422 field: reason. It used to return an unhandled 500, and the input is not echoed. MCP still returns {"error":"invalid_argument","detail":...}.

Synthetic examples:

{"op":"ingest_post","payload":{"headline":"Example headline","article":"...","read_more_url":"https://example.com/a","category":"Weather","type":"News","published_at":"2026-10-23"}}
{"op":"add_comment","payload":{"post_id":"<id>","kind":"update","body":"...","created_at":"2026-10-24T08:00:00-07:00"}}
{"op":"record_ingest_run","payload":{"notes":"backfill 2026-10-23","started_at":"2026-10-23T07:00:00-07:00","finished_at":"2026-10-23T07:04:30-07:00"}}

MCP: the same names are keyword arguments on ingest_post, add_comment, and record_ingest_run.

Storage and migration

  • posts.published_at is new. Every reader path (edition, latest edition date, archive filter and sort, list_recent_posts, the post and archive pages) uses it. ingested_at remains the true write time.
  • comments.recorded_at and ingest_runs.recorded_at are new and hold the true write time.
  • migrate() runs at connect inside BEGIN IMMEDIATE. It adds the missing columns, then backfills them from the column that held the same instant (published_at←ingested_at, recorded_at←created_at/finished_at), then adds idx_posts_published_at. It is additive and idempotent, so existing rows keep their date and order. This runs on the real DB at the first startup after deploy.

Tests (tests/test_backdate.py, 40 new; full suite 82 passed ×3; ruff check clean; new or edited files formatted)

  • Parser: PDT/PST date-only; the fall-back day (midnight still PDT); both 01:30s on fall-back via offsets; the spring-forward date; fraction truncation. Rejections cover naive times, impossible dates and times, non-ISO input, below the lower bound (including 06:59:59Z on Sep 17), beyond the skew, and LA tomorrow.
  • REST: the omitted field keeps the old behavior; a backdated post lands on its LA date; the 23:30 PDT boundary goes to the previous day; /editions HTML; archive filter, sort and search plus the HTML timestamp; dedupe keeps the stored date; 422 messages; blank means omitted.
  • Comments: a backdated comment appears in that date's New comments, archive last-update uses it, and recorded_at keeps the real write time; a comment before its post, or in the future, is rejected.
  • Runs: both bounds; one bound; the default; reversed bounds are rejected.
  • MCP over HTTP with a real bearer: the schema exposes the optional params; ingest, edition, search, comment and run are applied; future and naive values return clear errors.
  • Migration: run against a pre-0.8.0 schema, twice.
  • The existing test_app.py backdating helpers now set published_at as well. A test-local fixture resets the process-global rate limiter so the many ingests don't starve other tests.
  • Local smoke run: uvicorn on a temp DB with synthetic values. A date-only ingest returned 2026-10-23T07:00:00Z; a future date returned 422; /api/edition?date=2026-10-23 and /editions?date=2026-10-23 showed the post.

Limitations and dependencies

  • ntfy is unchanged and keys on ingested_at. If ntfy.json has enabled: true during the backfill, one "News Posted" notice fires for the burst, labeled with the backfill day.
  • list_recent_posts now windows on published_at, so backfilled posts older than days do not appear there.
  • A backfilled comment must be ≥ its post's published_at, so date-only posts (00:00 LA) accept any same-day comment.
  • design.md §4.1–4.4/§4.8 should name published_at and recorded_at. That belongs to the docs owner; this PR updates only docs/README.md.
  • Chief's backfill depends on this merging into dev (preview) and later main.
Adds optional caller-supplied times so missed editions can be backfilled. **Do not merge yet.** Merging into `dev` rebuilds preview and runs the migration below against the preview database, so the merge needs a separate decision. No tag has been pushed (`v0.8.0` is still to create on merge). No real data was touched. The branch is cut from `dev` @ 75b2153, which already contains #6. ## API | Tool / REST op | New optional field(s) | Effect | |---|---|---| | `ingest_post` | `published_at` | Sets edition date, archive `from`/`to` filter, edition/archive/recent sort | | `add_comment` | `created_at` | Sets which edition's New comments the comment lands in; must be ≥ the post's `published_at` | | `record_ingest_run` | `started_at`, `finished_at` | One alone sets both; start ≤ finish | **Formats** - `YYYY-MM-DD` means 00:00 America/Los_Angeles on that date, so `2026-10-23` is stored as `2026-10-23T07:00:00Z` (PDT) and `2026-11-02` as `08:00Z` (PST). - An ISO 8601 date-time must carry `Z` or an offset (`2026-10-23T23:30:00-07:00` is stored as `2026-10-24T06:30:00Z`, which falls in the **Oct 23** edition). - A **naive date-time is rejected**, because it is ambiguous in the DST fall-back hour and does not exist in the spring-forward hour. - Values are stored as UTC `YYYY-MM-DDTHH:MM:SSZ`, with fractions truncated. **Bounds** - **Lower:** ≥ 2026-09-17 00:00 America/Los_Angeles, the date the service contract was locked, so no edition can predate it. Oct 23 2026 onward is accepted. - **Upper:** ≤ now + 5 min, which allows for clock drift. Today's LA date is accepted; tomorrow's is rejected. **Behavior** - **Omitted or blank fields:** unchanged behavior, i.e. the server's write time. - **Dedupe:** never rewrites the stored row. The response returns the stored `published_at`. - **Responses:** now also include `published_at`, `created_at`, and `started_at`/`finished_at`. - **REST errors:** `/api/internal/ingest` now answers invalid payloads with **422 `field: reason`**. It used to return an unhandled 500, and the input is not echoed. MCP still returns `{"error":"invalid_argument","detail":...}`. Synthetic examples: ```json {"op":"ingest_post","payload":{"headline":"Example headline","article":"...","read_more_url":"https://example.com/a","category":"Weather","type":"News","published_at":"2026-10-23"}} {"op":"add_comment","payload":{"post_id":"<id>","kind":"update","body":"...","created_at":"2026-10-24T08:00:00-07:00"}} {"op":"record_ingest_run","payload":{"notes":"backfill 2026-10-23","started_at":"2026-10-23T07:00:00-07:00","finished_at":"2026-10-23T07:04:30-07:00"}} ``` MCP: the same names are keyword arguments on `ingest_post`, `add_comment`, and `record_ingest_run`. ## Storage and migration - `posts.published_at` is new. Every reader path (edition, latest edition date, archive filter and sort, `list_recent_posts`, the post and archive pages) uses it. `ingested_at` remains the true write time. - `comments.recorded_at` and `ingest_runs.recorded_at` are new and hold the true write time. - `migrate()` runs at connect inside `BEGIN IMMEDIATE`. It adds the missing columns, then backfills them from the column that held the same instant (`published_at`←`ingested_at`, `recorded_at`←`created_at`/`finished_at`), then adds `idx_posts_published_at`. It is additive and idempotent, so existing rows keep their date and order. **This runs on the real DB at the first startup after deploy.** ## Tests (`tests/test_backdate.py`, 40 new; full suite 82 passed ×3; ruff check clean; new or edited files formatted) - **Parser:** PDT/PST date-only; the fall-back day (midnight still PDT); both 01:30s on fall-back via offsets; the spring-forward date; fraction truncation. Rejections cover naive times, impossible dates and times, non-ISO input, below the lower bound (including 06:59:59Z on Sep 17), beyond the skew, and LA tomorrow. - **REST:** the omitted field keeps the old behavior; a backdated post lands on its LA date; the 23:30 PDT boundary goes to the previous day; `/editions` HTML; archive filter, sort and search plus the HTML timestamp; dedupe keeps the stored date; 422 messages; blank means omitted. - **Comments:** a backdated comment appears in that date's New comments, archive last-update uses it, and `recorded_at` keeps the real write time; a comment before its post, or in the future, is rejected. - **Runs:** both bounds; one bound; the default; reversed bounds are rejected. - **MCP over HTTP with a real bearer:** the schema exposes the optional params; ingest, edition, search, comment and run are applied; future and naive values return clear errors. - **Migration:** run against a pre-0.8.0 schema, twice. - The existing `test_app.py` backdating helpers now set `published_at` as well. A test-local fixture resets the process-global rate limiter so the many ingests don't starve other tests. - **Local smoke run:** uvicorn on a temp DB with synthetic values. A date-only ingest returned `2026-10-23T07:00:00Z`; a future date returned 422; `/api/edition?date=2026-10-23` and `/editions?date=2026-10-23` showed the post. ## Limitations and dependencies - **ntfy is unchanged** and keys on `ingested_at`. If `ntfy.json` has `enabled: true` during the backfill, one "News Posted" notice fires for the burst, labeled with the backfill day. - `list_recent_posts` now windows on `published_at`, so backfilled posts older than `days` do not appear there. - A backfilled comment must be ≥ its post's `published_at`, so date-only posts (00:00 LA) accept any same-day comment. - `design.md` §4.1–4.4/§4.8 should name `published_at` and `recorded_at`. That belongs to the docs owner; this PR updates only `docs/README.md`. - Chief's backfill depends on this merging into `dev` (preview) and later `main`.
ingest_post (MCP and the /api/internal/ingest fallback) takes an
optional published_at. add_comment takes an optional created_at, and
record_ingest_run takes optional started_at and finished_at. When the
new fields are omitted, behavior is unchanged: every time is the
server's write time.

Accepted values: YYYY-MM-DD, read as 00:00 on that date in
America/Los_Angeles, or an ISO 8601 date-time with Z or a UTC offset.
Midnight exists on every Los Angeles date (DST switches at 02:00), so
a date-only value is never ambiguous. A date-time without an offset is
rejected rather than guessed: in America/Los_Angeles the 01:00-02:00
hour on the fall-back date happens twice and the 02:00-03:00 hour on
the spring-forward date never happens. Stored values are UTC
YYYY-MM-DDTHH:MM:SSZ with fractional seconds truncated, the same shape
as server times, so text order stays chronological order.

Bounds: no earlier than 2026-09-17 00:00 America/Los_Angeles, the date
the service contract was locked, since no edition can predate the
service; no later than now plus 5 minutes, which absorbs caller clock
drift. A date-only value for today's Los Angeles date is accepted and
tomorrow's is rejected. A comment may not be timed before its post's
published_at, because archive "last update" assumes a comment is never
older than its post. started_at may not follow finished_at; when only
one is sent, both take that value (v1 rows already had the two equal).

Storage: posts gain published_at. Edition grouping, the most recent
edition date, archive from/to, archive and edition sort,
list_recent_posts, and the post and archive pages now read
published_at instead of ingested_at. ingested_at stays the true server
write time. Comments and ingest_runs gain recorded_at for the true
write time while created_at / started_at / finished_at carry the
supplied time. migrate() runs at connect inside BEGIN IMMEDIATE. It
adds the columns when they are missing, fills them from the column
that held the same instant (published_at from ingested_at, recorded_at
from created_at or finished_at), and creates idx_posts_published_at.
Existing rows keep their edition date and their order. The migration
only adds data and can run any number of times.

Dedupe never rewrites a stored row, so a duplicate ingest_post returns
the stored published_at and ignores the supplied one. Responses now
include published_at (ingest_post, posts, archive, list_recent_posts),
created_at (add_comment), and started_at / finished_at
(record_ingest_run).

/api/internal/ingest used to return 500 on a payload that failed
validation. It now returns 422 with "field: reason" and does not echo
the input. MCP keeps its invalid_argument shape.

ntfy is unchanged. It still keys its quiet clock, burst, and labels on
ingested_at, so a backfill burst with ntfy enabled sends one notice
labeled with the backfill day.

Rejected alternatives:
- Overwriting ingested_at with the supplied time. That loses the real
  write time and would make ntfy treat backfilled rows as long quiet.
- Reading naive date-times as Los Angeles time. That is ambiguous
  across DST.
- Noon for date-only values. Midnight is the edition-day start and is
  always unambiguous.
- An open lower bound. That allows typo years like 2016.

Version 0.8.0.
The README MCP section gets a Backfill times table covering
published_at, created_at, and started_at / finished_at. It gives the
accepted formats, the America/Los_Angeles reading of date-only values,
the bounds, the separate server write times, the dedupe rule, and the
422 REST error shape. design.md is not edited. The schema and edition
definitions in design.md 4.1-4.4 and 4.8 need a docs-owner update to
name published_at and recorded_at.
Brandon confirmed the new time fields are optional and an omitted one
means the current day. This pins that down: with the server clock at
2026-11-06T01:30:00Z (17:30 PST on Nov 5), a post, comment, and run
filed without times store that instant and appear in the 2026-11-05
edition, not the UTC date 2026-11-06. No application change was needed;
edition dates already convert stored UTC to America/Los_Angeles.
Grok merged commit 04c2b91f34 into dev 2026-11-06 02:14:58 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
ai-collab/agent-news-aggregator!8
No description provided.