ai-collab/bulk-image-organizer

Fork 0

[Bug]: Database on network share isn't being fully updated #36

New issue

Closed

opened 2026-06-10 04:39:49 +00:00 by ValleyGeek · 2 comments

ValleyGeek commented

2026-06-10 04:39:49 +00:00

Owner

App version

0.1.7

Platform

Windows

Steps to reproduce

Open a remote path on windows. i.e. \server\directory\pictures
Allow the scan to complete
Close the application
View the remote DB size (clearly too small to hold the thumbnails)
Open the application and directory again

Expected behavior

The remote database to be much larger, indicating that image blobs exist
Thumbnails to be cached when the application reopens
The remote database is occasionally synced in the background at low priority, through system file-copy from local (faster than using native sqlite on the share)
When the directory is changed or the application closes the local cache is synced, through standard file-copy instead of sqlite, before switching directory or terminating the application process
When opening the directory a second time, the database is copied, via system file copy mechanisms, to the local temp staging area before opened as a sqlite file
Thumbnails should only be regenerated if the file has changed from what's cached

Actual behavior

Using a thumbnail size of 384px on a directory with 3,500+ images, the saved database is less than 2mb.
Opening the application and directory a second time results in effectively a full initial scan and thumbnail generation

Additional context

Wasn't tested on Linux, can't confirm it's limited Windows only.

### App version 0.1.7 ### Platform Windows ### Steps to reproduce 1. Open a remote path on windows. i.e. \\server\directory\pictures 2. Allow the scan to complete 3. Close the application 4. View the remote DB size (clearly too small to hold the thumbnails) 5. Open the application and directory again ### Expected behavior - The remote database to be much larger, indicating that image blobs exist - Thumbnails to be cached when the application reopens - The remote database is occasionally synced in the background at low priority, through system file-copy from local (faster than using native sqlite on the share) - When the directory is changed or the application closes the local cache is synced, through standard file-copy instead of sqlite, before switching directory or terminating the application process - When opening the directory a second time, the database is copied, via system file copy mechanisms, to the local temp staging area before opened as a sqlite file - Thumbnails should only be regenerated if the file has changed from what's cached ### Actual behavior Using a thumbnail size of 384px on a directory with 3,500+ images, the saved database is less than 2mb. Opening the application and directory a second time results in effectively a full initial scan and thumbnail generation ### Additional context Wasn't tested on Linux, can't confirm it's limited Windows only.

ValleyGeek added the

Kind/Bug

Priority

Medium

labels

2026-06-10 04:39:49 +00:00

Grok referenced this issue from a commit

2026-06-10 04:58:53 +00:00

fix: sync full network workspace cache via file copy (#36)

Grok commented

2026-06-10 04:58:57 +00:00

Member

Fix implemented on `0.1.17/issue-36-network-db-sync`

Root cause: Network workspaces used metadata-only SQLite export to the share on close (DR-015). The share-side .bulk_image_organizer.db stayed small (no thumbnail BLOBs), and reopening could not restore a full cache from the share. Worker DB handles were also synced before scan workers were fully stopped on exit, which could prevent reliable WAL checkpointing.

Changes:

On workspace close, directory switch, and every 5 minutes while a network folder is open: checkpoint the local cache and file-copy the full database to the share.
On open: copy a newer share-side full cache into the local staging path before opening SQLite; legacy metadata-only share DBs still merge via import_share_metadata.
Metadata scan skips thumbnail regeneration when cached mod_date + filesize still match the source file.
Shutdown now releases worker DB locks before publishing the cache.

Verification: tests/test_workspace_cache.py, tests/test_image_cache_stale.py, focused scanner/DB tests, and ./scripts/smoke-ui-exit.sh.

Ready for Windows validation on a UNC/mapped-drive library with 3,500+ images — the share .bulk_image_organizer.db should grow to reflect thumbnail BLOBs after the first close/sync.

## Fix implemented on `0.1.17/issue-36-network-db-sync` **Root cause:** Network workspaces used metadata-only SQLite export to the share on close (DR-015). The share-side `.bulk_image_organizer.db` stayed small (no thumbnail BLOBs), and reopening could not restore a full cache from the share. Worker DB handles were also synced before scan workers were fully stopped on exit, which could prevent reliable WAL checkpointing. **Changes:** - On workspace close, directory switch, and every 5 minutes while a network folder is open: checkpoint the local cache and **file-copy** the full database to the share. - On open: copy a newer share-side full cache into the local staging path before opening SQLite; legacy metadata-only share DBs still merge via `import_share_metadata`. - Metadata scan skips thumbnail regeneration when cached `mod_date` + `filesize` still match the source file. - Shutdown now releases worker DB locks **before** publishing the cache. **Verification:** `tests/test_workspace_cache.py`, `tests/test_image_cache_stale.py`, focused scanner/DB tests, and `./scripts/smoke-ui-exit.sh`. Ready for Windows validation on a UNC/mapped-drive library with 3,500+ images — the share `.bulk_image_organizer.db` should grow to reflect thumbnail BLOBs after the first close/sync.

Grok referenced this issue from a commit

2026-06-10 05:57:04 +00:00

perf: scan stale thumbnails once per metadata pass (#36)

Grok referenced this issue from a commit

2026-06-10 06:15:25 +00:00

ui: show status and progress during network DB sync (#36)

Grok referenced this issue from a commit

2026-06-10 06:50:31 +00:00

ui: live scan progress with rate, ETA, and thumbnail bar (#36)

Grok commented

2026-06-11 05:13:47 +00:00

Member

Resolution (0.1.18 — branch `0.1.17/issue-36-network-db-sync`)

Network workspace cache sync now uses full SQLite file copy (WAL checkpoint + chunked OS copy) instead of metadata-only export, so the share-side .bulk_image_organizer.db includes thumbnail BLOBs and reopening restores the cache.

Sync triggers: workspace close, directory switch, successful post-scan, and 5-minute background timer while a network folder is open (when Settings → Workspace sync is enabled).

On open: newer share DB is copied to the local cache; legacy metadata-only share files are merged when no thumbnail BLOBs are present.

Related fixes on this branch:

Placeholder upserts preserve existing thumbnail BLOBs (COALESCE); grid rehydrates cached thumbs after preload.
Changed-file detection during discovery (filesize/mtime diff) for stale thumbnail regen without re-statting the entire share.
Scan progress UI (rate, ETA, network sync progress bar).
Faster NAS discovery walk (entry.stat, no per-file resolve).

Closing as fixed in upcoming PR to main.

## Resolution (0.1.18 — branch `0.1.17/issue-36-network-db-sync`) Network workspace cache sync now uses **full SQLite file copy** (WAL checkpoint + chunked OS copy) instead of metadata-only export, so the share-side `.bulk_image_organizer.db` includes thumbnail BLOBs and reopening restores the cache. **Sync triggers:** workspace close, directory switch, successful post-scan, and 5-minute background timer while a network folder is open (when Settings → Workspace sync is enabled). **On open:** newer share DB is copied to the local cache; legacy metadata-only share files are merged when no thumbnail BLOBs are present. **Related fixes on this branch:** - Placeholder upserts preserve existing thumbnail BLOBs (`COALESCE`); grid rehydrates cached thumbs after preload. - Changed-file detection during discovery (filesize/mtime diff) for stale thumbnail regen without re-statting the entire share. - Scan progress UI (rate, ETA, network sync progress bar). - Faster NAS discovery walk (`entry.stat`, no per-file `resolve`). Closing as fixed in upcoming PR to `main`.

Grok closed this issue

2026-06-11 05:13:54 +00:00

Grok referenced this issue from a pull request that will close it,

2026-06-11 05:13:54 +00:00

fix: network workspace cache full sync and scan performance (fixes #36) #39

ValleyGeek referenced this issue from a commit

2026-06-11 05:30:21 +00:00

fix: network workspace cache full sync and scan performance (fixes #36) (#39)