feat: optional publish, comment, and run times for backfill #8
No reviewers
Labels
No labels
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
ai-collab/agent-news-aggregator!8
Loading…
Reference in a new issue
No description provided.
Delete branch "0.7.2/ingest-publish-date"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Adds optional caller-supplied times so missed editions can be backfilled. Do not merge yet. Merging into
devrebuilds preview and runs the migration below against the preview database, so the merge needs a separate decision. No tag has been pushed (v0.8.0is still to create on merge). No real data was touched. The branch is cut fromdev@75b2153, which already contains #6.API
ingest_postpublished_atfrom/tofilter, edition/archive/recent sortadd_commentcreated_atpublished_atrecord_ingest_runstarted_at,finished_atFormats
YYYY-MM-DDmeans 00:00 America/Los_Angeles on that date, so2026-10-23is stored as2026-10-23T07:00:00Z(PDT) and2026-11-02as08:00Z(PST).Zor an offset (2026-10-23T23:30:00-07:00is stored as2026-10-24T06:30:00Z, which falls in the Oct 23 edition).YYYY-MM-DDTHH:MM:SSZ, with fractions truncated.Bounds
Behavior
published_at.published_at,created_at, andstarted_at/finished_at./api/internal/ingestnow answers invalid payloads with 422field: reason. It used to return an unhandled 500, and the input is not echoed. MCP still returns{"error":"invalid_argument","detail":...}.Synthetic examples:
MCP: the same names are keyword arguments on
ingest_post,add_comment, andrecord_ingest_run.Storage and migration
posts.published_atis new. Every reader path (edition, latest edition date, archive filter and sort,list_recent_posts, the post and archive pages) uses it.ingested_atremains the true write time.comments.recorded_atandingest_runs.recorded_atare new and hold the true write time.migrate()runs at connect insideBEGIN IMMEDIATE. It adds the missing columns, then backfills them from the column that held the same instant (published_at←ingested_at,recorded_at←created_at/finished_at), then addsidx_posts_published_at. It is additive and idempotent, so existing rows keep their date and order. This runs on the real DB at the first startup after deploy.Tests (
tests/test_backdate.py, 40 new; full suite 82 passed ×3; ruff check clean; new or edited files formatted)/editionsHTML; archive filter, sort and search plus the HTML timestamp; dedupe keeps the stored date; 422 messages; blank means omitted.recorded_atkeeps the real write time; a comment before its post, or in the future, is rejected.test_app.pybackdating helpers now setpublished_atas well. A test-local fixture resets the process-global rate limiter so the many ingests don't starve other tests.2026-10-23T07:00:00Z; a future date returned 422;/api/edition?date=2026-10-23and/editions?date=2026-10-23showed the post.Limitations and dependencies
ingested_at. Ifntfy.jsonhasenabled: trueduring the backfill, one "News Posted" notice fires for the burst, labeled with the backfill day.list_recent_postsnow windows onpublished_at, so backfilled posts older thandaysdo not appear there.published_at, so date-only posts (00:00 LA) accept any same-day comment.design.md§4.1–4.4/§4.8 should namepublished_atandrecorded_at. That belongs to the docs owner; this PR updates onlydocs/README.md.dev(preview) and latermain.