ai-collab/bulk-image-organizer
ai-collab/bulk-image-organizer
3
0
Fork 0
Code Issues 2 Pull requests Projects Releases 75 Packages Wiki Activity Actions

fix: network workspace cache full sync and scan performance (fixes #36) #39

Merged
ValleyGeek merged 12 commits from 0.1.17/issue-36-network-db-sync into main 2026-06-11 05:30:20 +00:00
Conversation 0 Commits 12 Files changed 20 +1257 -103
Grok commented 2026-06-11 05:13:54 +00:00
Member
Copy link

Summary

Fixes #36 — network share .bulk_image_organizer.db stayed too small and reopening forced full thumbnail regeneration.

Changes

  • Full DB file-copy sync (DR-015 extended): WAL checkpoint + chunked copy of the local workspace cache to the share on close, directory switch, post-scan, and 5-minute background intervals while a network folder is open.
  • Hydrate on open: copy newer share DB to local cache; merge legacy metadata-only share files when no thumbnail BLOBs exist.
  • Thumbnail cache preservation: placeholder upserts use COALESCE so discovery does not wipe cached BLOBs; grid rehydrates thumbs after preload.
  • Changed-file detection: discovery compares filesize/mtime before upsert to queue stale thumbnail regen without re-statting the entire share.
  • Scan progress UX: rate/ETA in status bar, network sync progress bar, discovery every 20 images.
  • NAS discovery perf: entry.stat() and relpath without per-file resolve().
  • Warm-cache enrich UX: pending work total instead of misleading 0 of N.

Release

  • Version bump: 0.1.18
  • Docs: CHANGELOG.md, USER_GUIDE.md, DECISIONS.md (DR-015), DESIGN.md, TECHNICAL_OUTLINE.md

Threading notes

Scan and thumbnail workers unchanged — plain-data signals only; sync file-copy runs on worker/main thread with processEvents for progress UI.

Verification

  • ruff check . && ruff format --check .
  • pytest (workspace cache, image scanner, scan progress, working_db)
  • ./scripts/check-version-bump.sh
  • ./scripts/smoke-ui-exit.sh
## Summary Fixes #36 — network share `.bulk_image_organizer.db` stayed too small and reopening forced full thumbnail regeneration. ## Changes - **Full DB file-copy sync** (DR-015 extended): WAL checkpoint + chunked copy of the local workspace cache to the share on close, directory switch, post-scan, and 5-minute background intervals while a network folder is open. - **Hydrate on open**: copy newer share DB to local cache; merge legacy metadata-only share files when no thumbnail BLOBs exist. - **Thumbnail cache preservation**: placeholder upserts use `COALESCE` so discovery does not wipe cached BLOBs; grid rehydrates thumbs after preload. - **Changed-file detection**: discovery compares filesize/mtime before upsert to queue stale thumbnail regen without re-statting the entire share. - **Scan progress UX**: rate/ETA in status bar, network sync progress bar, discovery every 20 images. - **NAS discovery perf**: `entry.stat()` and relpath without per-file `resolve()`. - **Warm-cache enrich UX**: pending work total instead of misleading `0 of N`. ## Release - Version bump: **0.1.18** - Docs: `CHANGELOG.md`, `USER_GUIDE.md`, `DECISIONS.md` (DR-015), `DESIGN.md`, `TECHNICAL_OUTLINE.md` ## Threading notes Scan and thumbnail workers unchanged — plain-data signals only; sync file-copy runs on worker/main thread with `processEvents` for progress UI. ## Verification - `ruff check . && ruff format --check .` - `pytest` (workspace cache, image scanner, scan progress, working_db) - `./scripts/check-version-bump.sh` - `./scripts/smoke-ui-exit.sh`
Grok added 11 commits 2026-06-11 05:13:54 +00:00 
Network workspaces now checkpoint the local SQLite cache and copy the
complete database (including thumbnail BLOBs) to the share on close,
directory switch, and periodic background intervals. On reopen, a newer
share-side database hydrates the local staging cache before SQLite open;
legacy metadata-only share files still merge via import.

Scan enrichment skips thumbnail regeneration when cached mod_date and
filesize match the source file. Close/shutdown releases worker DB locks
before publishing the cache so WAL contents flush reliably.
Placeholder discovery upserts were overwriting thumbnail_blob (and
metadata columns) with NULL via unconditional ON CONFLICT updates.
Upserts now COALESCE nullable cache fields so discovery refreshes stat
fields without erasing stored thumbnails.

On open, cached thumbnail BLOBs are loaded into the grid model immediately
after metadata preload. mod_date staleness checks normalize Z/offset forms.
Publish the local SQLite cache to the remote share when a metadata scan
finishes successfully, so thumbnail BLOBs and post-scan DB updates are
on the share without waiting for close or the background interval.
The metadata enrichment loop queried every cached thumbnail and stat'd
each file on every iteration, yielding O(n^2) work as thumbnails were
written. Stale rows are now discovered once when discovery completes and
reused for the remainder of the scan.
File-copy hydrate and publish operations report byte progress through
chunked copies. Network workspace open, directory switch, workspace close,
and application shutdown display status text and a determinate progress bar
when the database file size is known; metadata-only merges use an
indeterminate bar.
Directory scans now update the status label and center count with
current/total during discovery indexing and metadata enrichment. The
progress bar shows a numeric fraction with visible text. Enrichment and
thumbnail generation include images-per-second and M:SS ETA estimates.

Placeholder seeding emits progress callbacks. Any thumbnails still
missing after the metadata pass start ThumbnailWorker with the same UI.
The placeholder filesystem walk emits progress while the total is still
unknown. Status and count labels show "N found" at 20-image intervals,
then switch to indexing once the final count is known.
Remove the centered scan counter during discovery, indexing, and
thumbnail generation. Progress counts remain in the status label as
"n of total" (or "N found" during discovery).
Extend img/s and remaining-time hints to discovery indexing and
filesystem walk counts, not only metadata and thumbnail phases.
Use extra decimal precision below 1 img/s so slow scans do not
round to 0.0.
Use scandir-cached entry.stat and relpath without per-file resolve
during discovery walks. Detect changed files during placeholder upsert
and queue stale thumbnail regen from that diff instead of re-statting
the entire share. Enrichment progress now uses pending work total so
warm-cache rescans no longer sit at 0 of N.
docs: PR-prep for issue 36 — release 0.1.18
Some checks failed
CI / lint (pull_request) Successful in 7s
Details
CI / test (pull_request) Failing after 2m5s
Details
CI / nuitka-smoke (pull_request) Has been cancelled
Details
7682f3469f 
Bump VERSION to 0.1.18. Document full network workspace cache file-copy
sync, discovery/enrichment improvements, and scan progress UX in
CHANGELOG, USER_GUIDE, and design decision records.
test: expect initial enrich progress work-total callback
All checks were successful
CI / lint (pull_request) Successful in 7s
Details
CI / test (pull_request) Successful in 1m17s
Details
CI / nuitka-smoke (pull_request) Successful in 8m10s
Details
e766eeee40 
Metadata scan now emits on_progress(0, work_total, "") before
processing items; update PR9 assertions accordingly.
ValleyGeek deleted branch 0.1.17/issue-36-network-db-sync 2026-06-11 05:30:20 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
Kind/Bug
Something isn't working as expected

  
Kind/Feature
New capability or user-requested feature

  
Priority/High
High priority

  
Priority/Medium
Medium priority

  
Reviewed/Confirmed
Bug or feature report confirmed valid

  
Compat/Breaking
Breaking change that won't be backward compatible

  
Kind/Bug
Something is not working

  
Kind/Documentation
Documentation changes

  
Kind/Enhancement
Improve existing functionality

  
Kind/Feature
New functionality

  
Kind/Security
This is security issue

  
Kind/Testing
Issue or pull request related to testing

  
Priority
Critical
 The priority is critical

  
Priority
High
 The priority is high

  
Priority
Low
 The priority is low

  
Priority
Medium
 The priority is medium

  
Reviewed
Confirmed
 Issue has been confirmed

  
Reviewed
Duplicate
 This issue or pull request already exists

  
Reviewed
Invalid
 Invalid issue

  
Reviewed
Won't Fix
 This issue won't be fixed

  
Status
Abandoned
 Somebody has started to work on this but abandoned work

  
Status
Blocked
 Something is blocking this issue or pull request

  
Status
Need More Info
 Feedback is required to reproduce issue or to continue work

No labels
Kind/Bug
Kind/Feature
Priority/High
Priority/Medium
Reviewed/Confirmed
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ai-collab/bulk-image-organizer!39
No description provided.