Background directory scan can freeze the main UI, contradicting "UI stays interactive during scans"

Claude commented

2026-08-26 04:59:41 +00:00

Member

Summary

The app's design intent is that the main UI stays fully interactive and takes first priority while background scans (directory open/rescan) run. In practice, interacting with the app during Stage 2 ("Updating Database") and Stage 3 ("Generating Thumbnails") of the directory-open scan (see #176 for the stage breakdown) can freeze the UI or make it noticeably sluggish. Investigation turned up three contributing issues, filed together since they compound each other.

1. Tagging/deleting during a scan can genuinely freeze the event loop (SQLite writer-lock contention)

WorkingDB.transaction() (db/working_db.py:256-262) opens a brand-new sqlite3.connect() for every single call, and every DB call anywhere in the app — background worker or main-thread UI action — goes through it. WAL mode lets readers proceed without blocking on a writer, but SQLite still only allows one writer at a time per database file.

During Stage 2 and Stage 3, the scan worker issues a steady stream of small write transactions (upsert_images batches) from a background thread. Tagging, marking DELETE, or rotating an image — all normal interactions expected to work while a scan runs — go through TagManager synchronously on the main thread (core/tag_manager.py:557-588, e.g. toggle_tag_on_paths → working_db.add_tags_to_images / remove_tags_from_images), which is also a write.

If the user's write lands while the worker's write transaction holds the file lock, the main thread's write has to wait — and that wait happens via PRAGMA busy_timeout = 5000 (working_db.py:159), a blocking retry loop inside the SQLite C library call that does not yield to Qt's event loop. A tag click during a scan can therefore freeze the whole UI for up to 5 seconds while it waits its turn for the writer lock. This is a direct violation of the "main UI takes first priority" intent, not just a performance nuance.

2. General sluggishness during Stage 3 (GIL contention from thumbnail generation)

Thumbnail decode/resize/encode (Pillow, in core/image_scanner.py's enrich_image_record) runs in a ThreadPoolExecutor spawned inside the deep ScanWorker's own background thread (thread_count configurable via PREF_THUMBNAIL_THREADS, default DEFAULT_THUMBNAIL_THREADS = 2). This is genuine CPU-bound Python work. Even off the main thread, it competes for the GIL with the main thread's own event loop and paint code — Pillow's C-level codecs release the GIL during actual decode, but the surrounding Python glue doesn't, so under sustained load this shows up as input lag/jank distinct from the hard freeze in #1.

3. Compounding factor: per-call connection churn

Because every WorkingDB call re-opens a connection from scratch — re-running PRAGMA setup (including a 1GB mmap_size and ~300MB cache_size) and a schema-version check every time (working_db.py:130-141) — the writer-lock window in #1 opens and closes far more often than necessary. There's no structural reason a single scan couldn't hold one connection open across its whole batch loop instead of reconnecting per batch, which would shrink the contention window significantly.

Suggested fix direction

Introduce a shared write-serialization point for WorkingDB — e.g. a single long-lived writer connection per instance guarded by a lock/queue — instead of each call opening and closing its own connection and relying on busy_timeout to paper over contention.
Consider whether main-thread writes (tag/delete/rotate) need a non-blocking-to-the-event-loop retry path (e.g. queue the write and apply it via a short worker round-trip) rather than a direct synchronous call that can stall on busy_timeout.
Re-evaluate default thumbnail_generation_threads / scheduling under sustained GIL contention, and/or move more of the Pillow work to a process pool if GIL contention proves significant in profiling.

Where found

Investigated on branch 0.8.0/dupe-compare-progress while working on directory-scan progress reporting; a follow-up to the stage-labeling work in #176.

## Summary The app's design intent is that the main UI stays fully interactive and takes first priority while background scans (directory open/rescan) run. In practice, interacting with the app during Stage 2 ("Updating Database") and Stage 3 ("Generating Thumbnails") of the directory-open scan (see #176 for the stage breakdown) can freeze the UI or make it noticeably sluggish. Investigation turned up three contributing issues, filed together since they compound each other. ## 1. Tagging/deleting during a scan can genuinely freeze the event loop (SQLite writer-lock contention) `WorkingDB.transaction()` (`db/working_db.py:256-262`) opens a brand-new `sqlite3.connect()` for every single call, and every DB call anywhere in the app — background worker or main-thread UI action — goes through it. WAL mode lets readers proceed without blocking on a writer, but SQLite still only allows **one writer at a time** per database file. During Stage 2 and Stage 3, the scan worker issues a steady stream of small write transactions (`upsert_images` batches) from a background thread. Tagging, marking DELETE, or rotating an image — all normal interactions expected to work while a scan runs — go through `TagManager` **synchronously on the main thread** (`core/tag_manager.py:557-588`, e.g. `toggle_tag_on_paths` → `working_db.add_tags_to_images` / `remove_tags_from_images`), which is also a write. If the user's write lands while the worker's write transaction holds the file lock, the main thread's write has to wait — and that wait happens via `PRAGMA busy_timeout = 5000` (`working_db.py:159`), a **blocking retry loop inside the SQLite C library call** that does not yield to Qt's event loop. A tag click during a scan can therefore freeze the whole UI for up to 5 seconds while it waits its turn for the writer lock. This is a direct violation of the "main UI takes first priority" intent, not just a performance nuance. ## 2. General sluggishness during Stage 3 (GIL contention from thumbnail generation) Thumbnail decode/resize/encode (Pillow, in `core/image_scanner.py`'s `enrich_image_record`) runs in a `ThreadPoolExecutor` spawned inside the deep `ScanWorker`'s own background thread (`thread_count` configurable via `PREF_THUMBNAIL_THREADS`, default `DEFAULT_THUMBNAIL_THREADS = 2`). This is genuine CPU-bound Python work. Even off the main thread, it competes for the GIL with the main thread's own event loop and paint code — Pillow's C-level codecs release the GIL during actual decode, but the surrounding Python glue doesn't, so under sustained load this shows up as input lag/jank distinct from the hard freeze in #1. ## 3. Compounding factor: per-call connection churn Because every `WorkingDB` call re-opens a connection from scratch — re-running `PRAGMA` setup (including a 1GB `mmap_size` and ~300MB `cache_size`) and a schema-version check every time (`working_db.py:130-141`) — the writer-lock window in #1 opens and closes far more often than necessary. There's no structural reason a single scan couldn't hold one connection open across its whole batch loop instead of reconnecting per batch, which would shrink the contention window significantly. ## Suggested fix direction - Introduce a shared write-serialization point for `WorkingDB` — e.g. a single long-lived writer connection per instance guarded by a lock/queue — instead of each call opening and closing its own connection and relying on `busy_timeout` to paper over contention. - Consider whether main-thread writes (tag/delete/rotate) need a non-blocking-to-the-event-loop retry path (e.g. queue the write and apply it via a short worker round-trip) rather than a direct synchronous call that can stall on `busy_timeout`. - Re-evaluate default `thumbnail_generation_threads` / scheduling under sustained GIL contention, and/or move more of the Pillow work to a process pool if GIL contention proves significant in profiling. ## Where found Investigated on branch `0.8.0/dupe-compare-progress` while working on directory-scan progress reporting; a follow-up to the stage-labeling work in #176.

Claude added the

Kind/Bug

Priority/High

Reviewed/Confirmed

labels

2026-08-26 04:59:47 +00:00

Claude referenced this issue

2026-08-26 05:22:54 +00:00

Standardized status messages, foreground progress dialogs, and N+1 fixes (0.9.0) #180

ValleyGeek referenced this issue from a commit

2026-08-26 05:24:59 +00:00

Standardized status messages, foreground progress dialogs, and N+1 fixes (0.9.0) (#180)

Claude referenced this issue from a commit

2026-08-27 21:08:01 +00:00

fix(ui): keep tag/DELETE writes from freezing the UI during a scan

Claude commented

2026-08-27 21:08:51 +00:00

Author

Member

Addressed on branch 0.9.1/issue-176-178-scan-freeze (commit abffa38):

Item 1 (main-thread write stall, the actual multi-second-freeze cause): WorkingDB.temporary_busy_timeout(ms) is a thread-local override so a main-thread write can opt into a short busy_timeout (150ms) without affecting a concurrent background worker's own connections on the same WorkingDB instance. ImageTagAssignmentCommand.redo()/undo() — the funnel for every tag toggle and DELETE mark via QUndoStack — now retry through a locked database with that short timeout, pumping the event loop between attempts (~6 attempts, ~1s worst case) instead of blocking once for the old 5s window. add_tags_to_images/remove_tags_from_images use INSERT OR IGNORE/plain DELETE, so retrying the whole apply_assignment_changes() call after a partial failure is idempotent. After exhausting retries it logs and no-ops rather than raising on the main thread.

Item 3 (connection churn): scan_metadata_ordered_and_upsert() reconnected on every dispatch-loop iteration just to poll the pending-enrichment queue. It now reuses one WorkingDB.open_connection() for that read across the whole loop; writes (_flush_batch) keep their own short-lived transaction() connections and commit-per-flush durability unchanged. Worth noting: this is a background-thread efficiency win, not itself a UI-freeze fix — that's item 1 above. It doesn't reduce writer-lock hold time, only per-call reconnect overhead.

Item 2 (thumbnail-generation GIL contention): deferred, no code change. The issue itself frames this as needing profiling evidence before touching thumbnail_generation_threads scheduling ("if GIL contention proves significant in profiling"), and the app already exposes a user-tunable PREF_THUMBNAIL_THREADS plus a benchmark-based "detect optimal threads" feature (core/thumbnail_benchmark.py). Flagging this back to you in case you want it split into its own tracked issue with a profiling plan, rather than left implicit here.

New tests cover the retry-through-transient-lock path, the graceful give-up-after-sustained-lock path, and thread-local isolation of the busy_timeout override (tests/test_pr10_undo.py, tests/db/test_working_db.py). Full suite passing. Not closing yet — development branch, not yet merged to main.

Addressed on branch `0.9.1/issue-176-178-scan-freeze` (commit `abffa38`): **Item 1 (main-thread write stall, the actual multi-second-freeze cause):** `WorkingDB.temporary_busy_timeout(ms)` is a thread-local override so a main-thread write can opt into a short `busy_timeout` (150ms) without affecting a concurrent background worker's own connections on the same `WorkingDB` instance. `ImageTagAssignmentCommand.redo()`/`undo()` — the funnel for every tag toggle and DELETE mark via `QUndoStack` — now retry through a locked database with that short timeout, pumping the event loop between attempts (~6 attempts, ~1s worst case) instead of blocking once for the old 5s window. `add_tags_to_images`/`remove_tags_from_images` use `INSERT OR IGNORE`/plain `DELETE`, so retrying the whole `apply_assignment_changes()` call after a partial failure is idempotent. After exhausting retries it logs and no-ops rather than raising on the main thread. **Item 3 (connection churn):** `scan_metadata_ordered_and_upsert()` reconnected on every dispatch-loop iteration just to poll the pending-enrichment queue. It now reuses one `WorkingDB.open_connection()` for that read across the whole loop; writes (`_flush_batch`) keep their own short-lived `transaction()` connections and commit-per-flush durability unchanged. Worth noting: this is a background-thread efficiency win, not itself a UI-freeze fix — that's item 1 above. It doesn't reduce writer-lock hold time, only per-call reconnect overhead. **Item 2 (thumbnail-generation GIL contention):** deferred, no code change. The issue itself frames this as needing profiling evidence before touching `thumbnail_generation_threads` scheduling ("if GIL contention proves significant in profiling"), and the app already exposes a user-tunable `PREF_THUMBNAIL_THREADS` plus a benchmark-based "detect optimal threads" feature (`core/thumbnail_benchmark.py`). Flagging this back to you in case you want it split into its own tracked issue with a profiling plan, rather than left implicit here. New tests cover the retry-through-transient-lock path, the graceful give-up-after-sustained-lock path, and thread-local isolation of the busy_timeout override (`tests/test_pr10_undo.py`, `tests/db/test_working_db.py`). Full suite passing. Not closing yet — development branch, not yet merged to `main`.

Grok referenced this issue

2026-08-27 22:39:28 +00:00

[Bug]: Thumbnail generation throughput collapses at ~125k images (5–8 img/s, idle CPU/share) vs ~30 img/s at 3.9k on the same NAS #184

Claude commented

2026-08-27 23:18:58 +00:00

Author

Member

Follow-up on the report of near-total UI unresponsiveness opening a 15,000-image already-cached directory on Windows 11: traced this to a regression in the #176 fix (the chunked cached-thumbnail hydrate loop), not this issue's busy_timeout/connection-reuse changes — see the comment on #176 for the root cause and fix (commit 2076b5b, now runs on a background worker instead of a main-thread loop with forced repaints per chunk).

The "Updating Database" stage sluggishness reported alongside it was very likely the same storm's paint/input backlog still draining on the main thread rather than a separate cause in this issue's changes — worth re-testing on the same 15k-image workspace now that the #176 regression is fixed, to confirm whether any sluggishness remains once that's isolated out.

Follow-up on the report of near-total UI unresponsiveness opening a 15,000-image already-cached directory on Windows 11: traced this to a regression in the **#176** fix (the chunked cached-thumbnail hydrate loop), not this issue's busy_timeout/connection-reuse changes — see the comment on #176 for the root cause and fix (commit `2076b5b`, now runs on a background worker instead of a main-thread loop with forced repaints per chunk). The "Updating Database" stage sluggishness reported alongside it was very likely the same storm's paint/input backlog still draining on the main thread rather than a separate cause in this issue's changes — worth re-testing on the same 15k-image workspace now that the #176 regression is fixed, to confirm whether any sluggishness remains once that's isolated out.

Claude referenced this issue from a commit

2026-08-28 01:57:43 +00:00

fix(ui): batch dataChanged into contiguous runs instead of full-model spans

Claude commented

2026-08-28 01:57:49 +00:00

Author

Member

Follow-up: after the #176 hydrate-worker fix, further testing on the same large workspace found "Updating Database" (stage 2, the placeholder discovery DB-write pass) still not properly responsive, and "Generating Thumbnails" (stage 3, the metadata/thumbnail scan) noticeably less responsive than it should be.

Root cause was in ImageListModel, not the scan workers themselves: upsert_images()/apply_thumbnail_batch() emitted one dataChanged spanning min(changed_rows)..max(changed_rows) per progress batch. Scan/thumbnail batches touch rows in filesystem-walk or DB-query order, not the model's row order, so a batch's changed rows are typically scattered across the whole model — for a 15k-row model that meant a dataChanged range spanning nearly every row, tens of times over the course of a scan, and QSortFilterProxyModel's dataChanged handling costs roughly O(range size) per call regardless of how many rows in that range actually changed.

Fixed in commit fd0ce3b: ImageListModel._emit_data_changed_for_rows() emits one dataChanged per strictly-contiguous run of sorted row indices instead of one min..max span, bounding the total emitted range cost to the number of changed rows. Applied to upsert_images(), apply_thumbnail_batch(), and the tag-rename/recolor batch path. New tests assert the emitted ranges directly (scattered rows produce separate tight runs, not one wide span). Full suite (788 tests) passing.

This should improve both stage 2 and stage 3, since both go through the same model methods. Thumbnail-generation GIL contention (the deferred item 2 from the original report) is still untouched — happy to revisit if stage 3 still isn't responsive enough after this.

Follow-up: after the #176 hydrate-worker fix, further testing on the same large workspace found "Updating Database" (stage 2, the placeholder discovery DB-write pass) still not properly responsive, and "Generating Thumbnails" (stage 3, the metadata/thumbnail scan) noticeably less responsive than it should be. Root cause was in `ImageListModel`, not the scan workers themselves: `upsert_images()`/`apply_thumbnail_batch()` emitted one `dataChanged` spanning `min(changed_rows)..max(changed_rows)` per progress batch. Scan/thumbnail batches touch rows in filesystem-walk or DB-query order, not the model's row order, so a batch's changed rows are typically scattered across the whole model — for a 15k-row model that meant a `dataChanged` range spanning nearly every row, tens of times over the course of a scan, and `QSortFilterProxyModel`'s `dataChanged` handling costs roughly O(range size) per call regardless of how many rows in that range actually changed. Fixed in commit `fd0ce3b`: `ImageListModel._emit_data_changed_for_rows()` emits one `dataChanged` per strictly-contiguous run of sorted row indices instead of one min..max span, bounding the total emitted range cost to the number of changed rows. Applied to `upsert_images()`, `apply_thumbnail_batch()`, and the tag-rename/recolor batch path. New tests assert the emitted ranges directly (scattered rows produce separate tight runs, not one wide span). Full suite (788 tests) passing. This should improve both stage 2 and stage 3, since both go through the same model methods. Thumbnail-generation GIL contention (the deferred item 2 from the original report) is still untouched — happy to revisit if stage 3 still isn't responsive enough after this.

Claude referenced this issue from a commit

2026-08-28 03:11:10 +00:00

docs: PR-prep for scan/hydrate freeze and responsiveness fixes (0.9.2)

Claude referenced this issue

2026-08-28 03:11:30 +00:00

Fix directory-open/scan freezes and unresponsiveness (0.9.2) #187

ValleyGeek referenced this issue from a commit

2026-08-28 04:45:17 +00:00

Fix directory-open/scan freezes and unresponsiveness (0.9.2) (#187)

Claude commented

2026-08-28 05:14:48 +00:00

Author