Fix directory-open/scan freezes and unresponsiveness (0.9.2) #187

Merged

ValleyGeek merged 6 commits from 0.9.1/issue-176-178-scan-freeze into main 2026-08-28 04:45:15 +00:00

Conversation 0 Commits 6 Files changed 15 +930 -56

Claude commented 2026-08-28 03:11:30 +00:00

Member

Copy link

Summary

Fixes #176, #177, #178, plus two regressions found during manual re-testing on a large (15k-image) cached workspace on Windows 11.

• #176 — open_directory()'s post-hydrate block (stale-type prune, DB preload, thumbnail hydrate) ran with zero _pump_ui_events() calls, so its "Preparing workspace..." status never rendered before the block's work finished — looked frozen on large workspaces. Fixed by deduping the redundant get_all_images() query, chunking the cached-thumbnail load with real progress, and pumping between stages.
• #177 — Cancel during a network directory open (hydrate dialog, or Cancel Scan once discovery/enrichment starts) didn't stop promptly. predict_hydrate_action() now runs off-thread so Cancel isn't stuck behind an uninterruptible syscall; Cancel Scan is live during the preload phase; the metadata-scan executor stops waiting on in-flight decodes once cancelled; redundant post-cancel work is skipped.
• #178 — Main-thread tag/DELETE writes could block up to 5s colliding with a scan worker's writer lock; WorkingDB.temporary_busy_timeout() (thread-local) plus a bounded retry in ImageTagAssignmentCommand fixes that. The scan's hottest read path also now reuses one connection instead of reconnecting every loop iteration.
• Follow-up regression: the #176 chunked thumbnail-hydrate loop still ran on the main thread with a forced processEvents() pump per chunk — on a 15k-image cached workspace this was a severe regression, worse than the bug it fixed. Moved to a background CachedThumbnailLoadWorker delivering batches via the existing ThumbnailSignals queue instead.
• Follow-up regression: ImageListModel.upsert_images()/apply_thumbnail_batch() emitted one dataChanged spanning the full min..max of a scattered batch's changed rows — near-full-model-spanning ranges, tens of times per scan, dominating main-thread cost during the "Updating Database" / "Generating Thumbnails" scan stages. Now emits per contiguous run instead.

Test plan

• ruff check . / ruff format --check . clean
• Full pytest suite: 788 passed, 1 skipped (includes new tests for every fix above, several of which were verified to actually catch their regression by temporarily reverting the fix and confirming the test fails)
• ./scripts/smoke-ui-exit.sh clean after each worker/cancel-path change
• pip install -e . + scripts/get-version.py --canonical normalizes 0.9.2 correctly
• ./scripts/build-linux-nuitka-smoke.sh packages and verifies cleanly
• Manual re-test by the requester on a real 15k-image cached workspace on Windows 11, across two rounds of follow-up fixes

🤖 Generated with Claude Code

https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ

## Summary Fixes #176, #177, #178, plus two regressions found during manual re-testing on a large (15k-image) cached workspace on Windows 11. - **#176** — `open_directory()`'s post-hydrate block (stale-type prune, DB preload, thumbnail hydrate) ran with zero `_pump_ui_events()` calls, so its "Preparing workspace..." status never rendered before the block's work finished — looked frozen on large workspaces. Fixed by deduping the redundant `get_all_images()` query, chunking the cached-thumbnail load with real progress, and pumping between stages. - **#177** — Cancel during a network directory open (hydrate dialog, or Cancel Scan once discovery/enrichment starts) didn't stop promptly. `predict_hydrate_action()` now runs off-thread so Cancel isn't stuck behind an uninterruptible syscall; Cancel Scan is live during the preload phase; the metadata-scan executor stops waiting on in-flight decodes once cancelled; redundant post-cancel work is skipped. - **#178** — Main-thread tag/DELETE writes could block up to 5s colliding with a scan worker's writer lock; `WorkingDB.temporary_busy_timeout()` (thread-local) plus a bounded retry in `ImageTagAssignmentCommand` fixes that. The scan's hottest read path also now reuses one connection instead of reconnecting every loop iteration. - **Follow-up regression**: the #176 chunked thumbnail-hydrate loop still ran on the main thread with a forced `processEvents()` pump per chunk — on a 15k-image cached workspace this was a severe regression, *worse* than the bug it fixed. Moved to a background `CachedThumbnailLoadWorker` delivering batches via the existing `ThumbnailSignals` queue instead. - **Follow-up regression**: `ImageListModel.upsert_images()`/`apply_thumbnail_batch()` emitted one `dataChanged` spanning the full min..max of a scattered batch's changed rows — near-full-model-spanning ranges, tens of times per scan, dominating main-thread cost during the "Updating Database" / "Generating Thumbnails" scan stages. Now emits per contiguous run instead. ## Test plan - [x] `ruff check .` / `ruff format --check .` clean - [x] Full `pytest` suite: 788 passed, 1 skipped (includes new tests for every fix above, several of which were verified to actually catch their regression by temporarily reverting the fix and confirming the test fails) - [x] `./scripts/smoke-ui-exit.sh` clean after each worker/cancel-path change - [x] `pip install -e .` + `scripts/get-version.py --canonical` normalizes `0.9.2` correctly - [x] `./scripts/build-linux-nuitka-smoke.sh` packages and verifies cleanly - [x] Manual re-test by the requester on a real 15k-image cached workspace on Windows 11, across two rounds of follow-up fixes 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ

Claude added 6 commits 2026-08-28 03:11:31 +00:00

fix(ui): eliminate silent freeze between hydrate close and grid populate ... aa080a3fda

open_directory()'s post-hydrate preload block (stale-type prune, DB
preload, thumbnail hydration) ran synchronously on the main thread with
no _pump_ui_events() calls, so the "Preparing workspace..." status set
at the top of the block never actually painted before the block's work
finished -- the app looked frozen on large workspaces even though
progress reporting code was already in place.

- _prune_disabled_types_from_current_db() now accepts a pre-fetched
  image list and returns the surviving rows, so open_directory() fetches
  get_all_images() once and reuses it for preload instead of querying
  twice back-to-back.
- WorkingDB.iter_thumbnail_blob_dicts()/count_images_with_thumbnails()
  chunk the cached-thumbnail load instead of one unchunked query pulling
  every BLOB into memory at once; open_directory() reports real progress
  per chunk through the standard build_status_message() format.
- _pump_ui_events() calls added after each stage of the block so status
  text actually renders between stages.

Threading notes: all new work is plain DB I/O on the main thread behind
_pump_ui_events(), same pattern already used for the hydrate-copy loop
just above this block. No worker/QRunnable changes.

Fixes #176.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ

fix(ui): honor Cancel promptly during remote directory open/scan ... 6dc8b7a106

Cancelling during a network-directory open (hydrate dialog Cancel, or
Cancel Scan once discovery/enrichment starts) did not stop promptly:

- predict_hydrate_action() ran inline on the main thread with no cancel
  checkpoint -- a single stat()/SELECT syscall that can't be interrupted
  mid-call on a hung/slow share. It now runs via a background
  ThreadPoolExecutor while open_directory() polls the hydrate dialog's
  Cancel state; on cancel the open aborts immediately (HydrateCancelled)
  without waiting for the background call to return -- its read-only
  result is simply discarded whenever it eventually arrives.
- Cancel Scan was unavailable during open_directory()'s post-hydrate
  preload phase (prune/preload/thumbnail-hydrate, issue #176) -- no
  worker existed yet to receive .cancel(). _scan_in_progress is now set
  (and the toolbar updated) as soon as that phase starts; a new
  _open_cancel_requested flag lets _on_cancel_scan() signal cancellation
  before a real worker exists, checked at each safe point in the preload
  block via _preload_cancel_checkpoint().
- scan_metadata_ordered_and_upsert()'s ThreadPoolExecutor now checks
  is_cancelled() between each already-completed future in a chunk
  (rather than only after the whole chunk drains) and shuts down with
  wait=False, cancel_futures=True instead of the `with` block's implicit
  wait=True -- cancel no longer waits for an in-flight remote decode
  that's still running when it fires.
- _on_deep_finished() skips the get_image_count()/add_recent_dir() round
  trip on a cancelled scan, matching the existing skip of prune/dupe/sync
  for that case.

Threading notes: predict_hydrate_action() runs on a plain
concurrent.futures worker thread and touches no Qt objects; the polling
loop that watches it lives entirely on the main thread via the existing
_pump_ui_events() pattern. The executor shutdown change in
image_scanner.py only affects how/when already-dispatched CPU/IO futures
are awaited -- no new thread touches widgets/models.

Fixes #177.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ

fix(ui): keep tag/DELETE writes from freezing the UI during a scan ... abffa38fdb

Tagging, marking DELETE, or other main-thread WorkingDB writes could
freeze the whole UI for up to 5s when they collided with a background
scan worker's writer lock -- WorkingDB's busy_timeout=5000 retry is a
single blocking sqlite3 call that never yields to the Qt event loop
(issue #178, item 1).

- WorkingDB.temporary_busy_timeout(ms): a thread-local override so a
  main-thread write can opt into a short busy_timeout (150ms) without
  affecting a concurrent background worker's own connections on the
  same WorkingDB instance.
- ImageTagAssignmentCommand.redo()/undo() (the funnel for every tag
  toggle and DELETE mark via QUndoStack) now retry through a locked
  database with that short timeout, pumping the event loop between
  attempts (~6 attempts, ~1s worst case) instead of blocking once for
  the old 5s window. add_tags_to_images/remove_tags_from_images use
  INSERT OR IGNORE / plain DELETE, so retrying the whole
  apply_assignment_changes() call after a partial failure is
  idempotent. After exhausting retries it logs and no-ops rather than
  raising on the main thread.

Also shrinks connection churn on the scan worker's hottest read path
(item 3): scan_metadata_ordered_and_upsert() reconnected on every
dispatch-loop iteration just to poll the pending-enrichment queue. It
now reuses one WorkingDB.open_connection() for that read across the
whole loop; writes (_flush_batch) keep their own short-lived
transaction() connections and commit-per-flush durability unchanged --
this is a background-thread efficiency win, not a UI-freeze fix (that's
the busy_timeout change above).

Item 2 (thumbnail-generation GIL contention) is deferred per the
issue's own framing -- it calls for profiling evidence before touching
thumbnail_generation_threads scheduling, and the app already exposes a
user-tunable PREF_THUMBNAIL_THREADS plus a benchmark-based "detect
optimal threads" feature.

Threading notes: temporary_busy_timeout() only ever runs on the main
thread (driven by QUndoStack.push(), itself main-thread-only) and is
thread-local specifically so it cannot leak into a worker thread's
connections. open_connection() in the scan worker is read-only CPU/IO
work on the existing background thread; it touches no Qt objects.

Fixes #178.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ

fix(ui): move cached-thumbnail hydration off the main thread ... 2076b5bd51

Regression from the #176 fix: chunking the cached-thumbnail load was
correct, but it still ran the query loop directly on the main thread
and forced a processEvents() pump after every chunk. On a large cached
workspace (15k+ images, reported on Windows 11) that meant dozens of
synchronous dataChanged + repaint/decode cycles back-to-back -- the UI
was nearly unresponsive to selection/scrolling during the load, and the
app was noticeably sluggish afterward too (input/paint backlog from the
storm), worse than the single unchunked query it replaced.

CachedThumbnailLoadWorker (QRunnable, core/thumbnail_worker.py) reads
the same WorkingDB.iter_thumbnail_blob_dicts() chunks off the main
thread and delivers them through the existing ThumbnailSignals bundle.
The main thread's _on_thumbnails_ready() slot (already used by the
thumbnail-generation worker) applies each batch whenever Qt's event
loop naturally gets to the queued cross-thread signal -- no explicit
processEvents() forcing a synchronous repaint in between, so Qt
coalesces the dataChanged/update() calls into far fewer actual paints
instead of one per chunk.

open_directory() now kicks this worker off (via _start_cache_hydrate_worker())
right after the preload step and proceeds straight to _launch_directory_scan()
without waiting on it; the two run concurrently (read-only, no contention
with scan writes). Wired into the existing cancel paths
(_cancel_active_scans(), _on_cancel_scan()) alongside the other scan-adjacent
workers, so a directory switch, Cancel Scan, or app shutdown stops it too.

Threading notes: CachedThumbnailLoadWorker only performs WorkingDB reads
and emits plain-data signals (list[dict] batches, int progress) -- no
QWidget/QPixmap/model access on the worker thread, matching the
Threading Contract. It's constructed on the main thread
(assert_main_thread()) like every other QRunnable here.

Follow-up to #176/#178.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ

fix(ui): batch dataChanged into contiguous runs instead of full-model spans ... fd0ce3b562

Follow-up to #178: opening a large (15k+ image) directory was still
sluggish/unresponsive during the "Updating Database" and "Generating
Thumbnails" scan stages even after the previous fixes (thread-local
busy_timeout retry, reduced connection churn, async cache-thumbnail
hydration). Root cause was in ImageListModel, not the scan workers:
upsert_images()/apply_thumbnail_batch() emitted one dataChanged spanning
min(changed_rows)..max(changed_rows) per batch. Scan/thumbnail progress
batches touch rows in filesystem-walk or DB-query order, not the
model's row order, so a batch's changed rows are typically scattered
across the whole model -- for a 15k-row model that meant a dataChanged
range spanning nearly every row, tens of times per scan (once per
batch), and QSortFilterProxyModel's dataChanged handling costs roughly
O(range size) per call regardless of how many rows in that range
actually changed.

ImageListModel._emit_data_changed_for_rows() replaces the min..max span
with one emit per strictly-contiguous run of sorted (deduped) row
indices, bounding the total emitted range cost to the number of
changed rows instead of the total row count. Used by upsert_images(),
apply_thumbnail_batch(), and the tag-rename/recolor batch update (same
pattern, lower frequency).

Threading notes: none -- this is a main-thread-only model change, no
worker/signal changes.

Follow-up to #178.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ

docs: PR-prep for scan/hydrate freeze and responsiveness fixes (0.9.2) ... f990787908

- Bump VERSION to the next release (0.9.1 -> 0.9.2, patch: bug fixes
  only, no new feature tranche) per docs/VERSIONING.md.
- Add a dated CHANGELOG.md entry summarizing the four fixes on this
  branch (issues #176, #177, #178 and the two follow-up regressions
  found during manual testing).
- Document WorkingDB.temporary_busy_timeout()'s thread-local
  main-thread-write-retry pattern in docs/DESIGN.md, next to the
  per-directory working DB pragma block it overrides.

No resources/help/USER_GUIDE.md change: every fix on this branch is
an internal responsiveness/reliability correction with no new or
changed user-facing feature, option, or workflow to document there.

Verified before this commit: ruff check/format clean, full pytest
suite (788 passed, 1 skipped), pip install -e . + get-version.py
--canonical normalizes 0.9.2 correctly, and
./scripts/build-linux-nuitka-smoke.sh packages and verifies cleanly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ

ValleyGeek merged commit 025fb516b8 into main 2026-08-28 04:45:15 +00:00

ValleyGeek deleted branch 0.9.1/issue-176-178-scan-freeze 2026-08-28 04:45:15 +00:00

ValleyGeek referenced this pull request from a commit 2026-08-28 04:45:17 +00:00

Fix directory-open/scan freezes and unresponsiveness (0.9.2) (#187)

Claude referenced this pull request 2026-08-28 05:14:45 +00:00

Silent freeze between hydrate dialog closing and grid populating on large directories #176

Claude referenced this pull request 2026-08-28 05:14:46 +00:00

[Bug]: Cancelling a directory scan while opening a remote directory does not stop promptly #177

Claude referenced this pull request 2026-08-28 05:14:48 +00:00

Background directory scan can freeze the main UI, contradicting "UI stays interactive during scans" #178

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

Kind/Bug

Something isn't working as expected

  

Kind/Feature

New capability or user-requested feature

  

Priority/High

High priority

  

Priority/Medium

Medium priority

  

Reviewed/Confirmed

Bug or feature report confirmed valid

  

Compat/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

No labels

Kind/Bug

Kind/Feature

Priority/High

Priority/Medium

Reviewed/Confirmed

Compat/Breaking

Kind/Bug

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

ai-collab/bulk-image-organizer!187

No description provided.