Silent freeze between hydrate dialog closing and grid populating on large directories

Claude commented

2026-08-26 03:23:51 +00:00

Member

Summary

After the network-hydrate dialog (NetworkHydrateDialog) finishes and closes, there is a visible gap/freeze before the directory grid populates and the scan's own progress reporting ("Discovering...") takes over. On large directories this can be a multi-second (or longer) apparent hang with no feedback, even though open_directory() sets an indeterminate "Preparing workspace..." status message at the start of this block.

Root cause

The entire block in open_directory() from hydrate_dialog.finish() through self._launch_directory_scan(root) (src/bulk_image_organizer/ui/main_window.py:4466-4531) runs synchronously on the main thread with zero _pump_ui_events() calls anywhere in it — unlike the hydrate loop immediately before it, which pumps on every progress callback. Because Qt doesn't repaint until control returns to the event loop, the "Preparing workspace..." message set at the top of the block (main_window.py:4474-4479) never actually renders before the block's work is done — the app looks exactly as frozen as if that message didn't exist.

Concretely, in order:

record_workspace_open(...) — cheap.
_prune_disabled_types_from_current_db() (main_window.py:3800-3832) — runs a full get_all_images(include_thumbnails=False) table read, iterated row-by-row in Python to check each file extension against the enabled set.
_start_background_share_sync() / _stop_background_share_sync() — cheap.
The preload block (main_window.py:4494-4518):
- get_all_images(include_thumbnails=False) again — a second full-table read duplicating step 2's query, with no caching/reuse between them.
- TagManager.image_dicts_with_tags(existing) — one bulk tag query, but still O(N) Python work building a dict per image.
- self.model.set_images(plains) — resets the grid model.
- self._working_db.get_thumbnail_blob_dicts() (db/working_db.py:464-475) — likely the single biggest cost on a large directory: one query that loads the thumbnail_blob for every image that has a cached thumbnail, all at once, into memory, with no chunking and no progress signal. On a large workspace this can be hundreds of MB of BLOB data deserialized synchronously.
- self.model.apply_thumbnail_batch(thumb_batch) — iterates every entry, then fires a single dataChanged spanning the entire row range, forcing the view to reconsider every visible thumbnail's paint at once.
_refresh_dupe_trash_counts() — a few aggregate COUNT queries, cheap.
_refresh_tags_ui() — tag panel refresh from DB, plus a redundant second call to _refresh_dupe_trash_counts().
global_config.add_recent_dir(...) — cheap.
Only then does _launch_directory_scan(root) run, restoring visible progress ("Discovering...").

This is the same class of bug fixed twice earlier on branch 0.8.0/dupe-compare-progress (the post-hydrate preload N+1, and get_sort_eligible_images_by_tag's N+1) — except here the cost is concentrated in a small number of large, unchunked queries/operations rather than a per-row loop, so the fix needs chunked reporting rather than just batching.

Suggested fix

Drop the redundant second get_all_images() call (steps 2 and 4 can share one fetch).
Chunk the thumbnail-blob load (get_thumbnail_blob_dicts) and report progress through the existing core/status_message.py standard, the same way other long-running stages do.
Add _pump_ui_events() calls at appropriate points in this block so status text actually renders instead of being silently overwritten by the time the event loop next runs.

Where found

Investigated on branch 0.8.0/dupe-compare-progress while working on dialog/status-message progress reporting across the app.

## Summary After the network-hydrate dialog (`NetworkHydrateDialog`) finishes and closes, there is a visible gap/freeze before the directory grid populates and the scan's own progress reporting ("Discovering...") takes over. On large directories this can be a multi-second (or longer) apparent hang with no feedback, even though `open_directory()` sets an indeterminate "Preparing workspace..." status message at the start of this block. ## Root cause The entire block in `open_directory()` from `hydrate_dialog.finish()` through `self._launch_directory_scan(root)` (`src/bulk_image_organizer/ui/main_window.py:4466-4531`) runs synchronously on the main thread with **zero `_pump_ui_events()` calls** anywhere in it — unlike the hydrate loop immediately before it, which pumps on every progress callback. Because Qt doesn't repaint until control returns to the event loop, the "Preparing workspace..." message set at the top of the block (`main_window.py:4474-4479`) never actually renders before the block's work is done — the app looks exactly as frozen as if that message didn't exist. Concretely, in order: 1. `record_workspace_open(...)` — cheap. 2. `_prune_disabled_types_from_current_db()` (`main_window.py:3800-3832`) — runs a **full `get_all_images(include_thumbnails=False)`** table read, iterated row-by-row in Python to check each file extension against the enabled set. 3. `_start_background_share_sync()` / `_stop_background_share_sync()` — cheap. 4. The preload block (`main_window.py:4494-4518`): - `get_all_images(include_thumbnails=False)` **again** — a second full-table read duplicating step 2's query, with no caching/reuse between them. - `TagManager.image_dicts_with_tags(existing)` — one bulk tag query, but still O(N) Python work building a dict per image. - `self.model.set_images(plains)` — resets the grid model. - `self._working_db.get_thumbnail_blob_dicts()` (`db/working_db.py:464-475`) — **likely the single biggest cost on a large directory**: one query that loads the `thumbnail_blob` for every image that has a cached thumbnail, all at once, into memory, with no chunking and no progress signal. On a large workspace this can be hundreds of MB of BLOB data deserialized synchronously. - `self.model.apply_thumbnail_batch(thumb_batch)` — iterates every entry, then fires a single `dataChanged` spanning the entire row range, forcing the view to reconsider every visible thumbnail's paint at once. 5. `_refresh_dupe_trash_counts()` — a few aggregate `COUNT` queries, cheap. 6. `_refresh_tags_ui()` — tag panel refresh from DB, plus a redundant second call to `_refresh_dupe_trash_counts()`. 7. `global_config.add_recent_dir(...)` — cheap. 8. Only then does `_launch_directory_scan(root)` run, restoring visible progress ("Discovering..."). This is the same class of bug fixed twice earlier on branch `0.8.0/dupe-compare-progress` (the post-hydrate preload N+1, and `get_sort_eligible_images_by_tag`'s N+1) — except here the cost is concentrated in a small number of large, unchunked queries/operations rather than a per-row loop, so the fix needs chunked reporting rather than just batching. ## Suggested fix - Drop the redundant second `get_all_images()` call (steps 2 and 4 can share one fetch). - Chunk the thumbnail-blob load (`get_thumbnail_blob_dicts`) and report progress through the existing `core/status_message.py` standard, the same way other long-running stages do. - Add `_pump_ui_events()` calls at appropriate points in this block so status text actually renders instead of being silently overwritten by the time the event loop next runs. ## Where found Investigated on branch `0.8.0/dupe-compare-progress` while working on dialog/status-message progress reporting across the app.

Claude added the

Kind/Bug

Priority/High

Reviewed/Confirmed

labels

2026-08-26 03:23:57 +00:00

Grok referenced this issue

2026-08-26 03:32:06 +00:00

[Bug]: Cancelling a directory scan while opening a remote directory does not stop promptly #177

Claude referenced this issue from a commit

2026-08-26 03:46:26 +00:00

ui(scan): relabel directory-open scan progress into 3 stages

Claude referenced this issue

2026-08-26 04:59:41 +00:00

Background directory scan can freeze the main UI, contradicting "UI stays interactive during scans" #178

Claude referenced this issue

2026-08-26 05:22:54 +00:00

Standardized status messages, foreground progress dialogs, and N+1 fixes (0.9.0) #180

ValleyGeek referenced this issue from a commit

2026-08-26 05:24:59 +00:00

Standardized status messages, foreground progress dialogs, and N+1 fixes (0.9.0) (#180)

Claude referenced this issue from a commit

2026-08-27 20:42:11 +00:00

fix(ui): eliminate silent freeze between hydrate close and grid populate

Claude referenced this issue from a commit

2026-08-27 20:56:11 +00:00

fix(ui): honor Cancel promptly during remote directory open/scan

Claude commented

2026-08-27 21:08:36 +00:00

Author

Member

Fixed on branch 0.9.1/issue-176-178-scan-freeze (commit aa080a3):

_prune_disabled_types_from_current_db() now accepts a pre-fetched image list and returns the surviving rows, so open_directory() fetches get_all_images() once and reuses it for preload instead of querying twice back-to-back.
New WorkingDB.iter_thumbnail_blob_dicts() / count_images_with_thumbnails() chunk the cached-thumbnail load instead of one unchunked query pulling every BLOB into memory at once; open_directory() reports real per-chunk progress through the standard build_status_message() format.
_pump_ui_events() calls added after each stage of the post-hydrate block so the "Preparing workspace..." status actually renders between stages instead of being silently overwritten.

Covered by new tests in tests/db/test_working_db.py (chunked thumbnail load) plus the existing open_directory preload/hydrate test suite in tests/test_pr13_config.py, all passing. Not closing yet — this is on a development branch, not yet merged to main.

Fixed on branch `0.9.1/issue-176-178-scan-freeze` (commit `aa080a3`): - `_prune_disabled_types_from_current_db()` now accepts a pre-fetched image list and returns the surviving rows, so `open_directory()` fetches `get_all_images()` once and reuses it for preload instead of querying twice back-to-back. - New `WorkingDB.iter_thumbnail_blob_dicts()` / `count_images_with_thumbnails()` chunk the cached-thumbnail load instead of one unchunked query pulling every BLOB into memory at once; `open_directory()` reports real per-chunk progress through the standard `build_status_message()` format. - `_pump_ui_events()` calls added after each stage of the post-hydrate block so the "Preparing workspace..." status actually renders between stages instead of being silently overwritten. Covered by new tests in `tests/db/test_working_db.py` (chunked thumbnail load) plus the existing `open_directory` preload/hydrate test suite in `tests/test_pr13_config.py`, all passing. Not closing yet — this is on a development branch, not yet merged to `main`.

Claude referenced this issue

2026-08-27 21:08:43 +00:00

[Bug]: Cancelling a directory scan while opening a remote directory does not stop promptly #177

Grok referenced this issue

2026-08-27 22:39:28 +00:00

[Bug]: Thumbnail generation throughput collapses at ~125k images (5–8 img/s, idle CPU/share) vs ~30 img/s at 3.9k on the same NAS #184

Claude referenced this issue from a commit

2026-08-27 23:18:44 +00:00

fix(ui): move cached-thumbnail hydration off the main thread

Claude commented

2026-08-27 23:18:54 +00:00

Author

Member

Follow-up: the chunked thumbnail-hydrate loop from the initial fix (commit aa080a3) was itself a regression on large cached workspaces, reported on Windows 11 with a 15,000-image already-cached directory — near-total UI unresponsiveness while loading thumbnails, and lingering sluggishness afterward. Root cause: the loop was chunked correctly, but it still ran the query and a forced processEvents() pump on the main thread for every chunk (~75 chunks at the default chunk size) — each pump forced a synchronous dataChanged + repaint/decode cycle, so the fix traded one silent-but-fast blocking call for dozens of forced synchronous repaint storms. Net effect was worse than the original bug.

Fixed in commit 2076b5b: the hydrate loop now runs on a background CachedThumbnailLoadWorker (QRunnable) that delivers chunks through the existing ThumbnailSignals bundle (same signal/slot infra as the thumbnail-generation worker). The main thread only applies a batch when Qt's event loop naturally processes the queued cross-thread signal — no explicit processEvents() forcing synchronous repaints — so Qt coalesces the paint work instead of doing it once per chunk. open_directory() kicks this off and proceeds straight to the scan without waiting on it.

New tests cover the worker's chunking/cancel behavior directly and an integration test proving open_directory() returns without blocking on it (deterministic via a blocked first DB call, not timing-dependent). Full suite (785 tests) passing.

Follow-up: the chunked thumbnail-hydrate loop from the initial fix (commit `aa080a3`) was itself a regression on large cached workspaces, reported on Windows 11 with a 15,000-image already-cached directory — near-total UI unresponsiveness while loading thumbnails, and lingering sluggishness afterward. Root cause: the loop was chunked correctly, but it still ran the query and a forced `processEvents()` pump on the *main thread* for every chunk (~75 chunks at the default chunk size) — each pump forced a synchronous `dataChanged` + repaint/decode cycle, so the fix traded one silent-but-fast blocking call for dozens of forced synchronous repaint storms. Net effect was worse than the original bug. Fixed in commit `2076b5b`: the hydrate loop now runs on a background `CachedThumbnailLoadWorker` (`QRunnable`) that delivers chunks through the existing `ThumbnailSignals` bundle (same signal/slot infra as the thumbnail-generation worker). The main thread only applies a batch when Qt's event loop naturally processes the queued cross-thread signal — no explicit `processEvents()` forcing synchronous repaints — so Qt coalesces the paint work instead of doing it once per chunk. `open_directory()` kicks this off and proceeds straight to the scan without waiting on it. New tests cover the worker's chunking/cancel behavior directly and an integration test proving `open_directory()` returns without blocking on it (deterministic via a blocked first DB call, not timing-dependent). Full suite (785 tests) passing.

Claude referenced this issue

2026-08-27 23:18:58 +00:00

Background directory scan can freeze the main UI, contradicting "UI stays interactive during scans" #178

Claude referenced this issue

2026-08-28 01:57:49 +00:00

Background directory scan can freeze the main UI, contradicting "UI stays interactive during scans" #178

Claude referenced this issue from a commit

2026-08-28 03:11:10 +00:00

docs: PR-prep for scan/hydrate freeze and responsiveness fixes (0.9.2)

Claude referenced this issue from a pull request that will close it,

2026-08-28 03:11:30 +00:00

Fix directory-open/scan freezes and unresponsiveness (0.9.2) #187