Large-workspace open, scan, and Sort into Folders (0.10.0) #195

Merged
ValleyGeek merged 19 commits from 0.9.4/issue-179-184-dupe-pip-thumb-perf into main 2026-11-02 05:09:59 +00:00
Member

Summary

Release 0.10.0 (from 0.9.4). Large workspaces stay usable while they open and scan, and Sort into Folders reports and finishes its work without the per-file stall.

  • Open / scan (issues #184, #190, #191, DR-045). Discovery streams files as the walk runs. Missing files are pruned when the walk finishes, before thumbnails start, and a cancelled walk does not delete rows it never reached. Enrichment starts only after that prune, in the current sort order, and scrolling promotes the visible rows. Cached-thumbnail load uses seek pagination. SQL parameter lists are chunked so workspaces past SQLite's variable limit still preload, hydrate, and prune. The network cache copy leaves the UI thread. The grid fills in chunks. Workspaces of 20,000 images or more show a note that the app may be briefly unresponsive while that prepare step runs.
  • Duplicates (issue #179). Multi keeps each image's tag pip colors after a tag is applied.
  • Sort into Folders. The progress dialog stacks Planning Moves and Moving Files, and the planning count stays on screen while files move. Planning lists each destination folder once. Moving reuses one database connection and commits each path update on its own, instead of opening a new connection per file (400 local files: about 27/s before, the full sort in 0.15s after).

Closes #179, #184, #190, and #191.

Tests

ruff check, ruff format --check, and pytest: 881 passed, 1 skipped, on the code commits. This PR-prep commit is version and docs only (VERSION 0.10.0, changelog, design, outline, user guide).

## Summary Release **0.10.0** (from 0.9.4). Large workspaces stay usable while they open and scan, and Sort into Folders reports and finishes its work without the per-file stall. - **Open / scan (issues #184, #190, #191, DR-045).** Discovery streams files as the walk runs. Missing files are pruned when the walk finishes, before thumbnails start, and a cancelled walk does not delete rows it never reached. Enrichment starts only after that prune, in the current sort order, and scrolling promotes the visible rows. Cached-thumbnail load uses seek pagination. SQL parameter lists are chunked so workspaces past SQLite's variable limit still preload, hydrate, and prune. The network cache copy leaves the UI thread. The grid fills in chunks. Workspaces of 20,000 images or more show a note that the app may be briefly unresponsive while that prepare step runs. - **Duplicates (issue #179).** Multi keeps each image's tag pip colors after a tag is applied. - **Sort into Folders.** The progress dialog stacks Planning Moves and Moving Files, and the planning count stays on screen while files move. Planning lists each destination folder once. Moving reuses one database connection and commits each path update on its own, instead of opening a new connection per file (400 local files: about 27/s before, the full sort in 0.15s after). Closes #179, #184, #190, and #191. ## Tests `ruff check`, `ruff format --check`, and `pytest`: 881 passed, 1 skipped, on the code commits. This PR-prep commit is version and docs only (`VERSION` 0.10.0, changelog, design, outline, user guide).
Grok added 17 commits 2026-10-31 20:59:30 +00:00
fix(dupes): keep Multi tag pip colors after tag apply (issue #179)
Some checks failed
CI / lint (push) Successful in 14s
CI / test (push) Successful in 6m44s
CI / build-appimage (push) Has been cancelled
CI / build-windows-exe (push) Has been cancelled
CI / windows-exe (push) Has been cancelled
CI / appimage (push) Has been cancelled
b6bb08fccf
_after_tag_apply's dupes-Multi branch passed raw WorkingDB dicts (whose
tag_colors is always {}) straight to sync_image_tags(), which overwrote
the correct in-memory colors with the empty map from any get_image_dicts_for_dupe_tag()
call -- turning pips gray after toggling DELETE, Keep, or any other tag
apply while Multi is showing. Enrich via _enrich_dupe_group_images()
first, matching the viewer-return path's _sync_dupe_multi_view_tags().

Sets dev VERSION 0.9.4-bug.179-184 for this branch (issues #179, #184).
fix(scan): load the pending metadata/thumbnail queue once per scan (issue #184)
All checks were successful
CI / lint (push) Successful in 20s
CI / test (push) Successful in 4m45s
CI / appimage (push) Successful in 16m44s
CI / build-appimage (push) Successful in 0s
CI / windows-exe (push) Successful in 19m39s
CI / build-windows-exe (push) Successful in 0s
a8e760afb5
scan_metadata_ordered_and_upsert() re-queried the full pending-enrichment set
from SQLite (LEFT JOIN tags, GROUP BY, plus reading every matched row's
thumbnail_blob just to null-check it) and re-sorted the entire remaining
queue on every dispatch chunk (thread_count rows, default 2). That made the
loop's own bookkeeping O(n^2 log n) in the number of pending images: cheap at
a few thousand images, but the dominant cost at ~125k, where CPU sat near
idle and share throughput barely moved while thumbnail generation crawled at
5-8 img/s instead of the ~30 img/s a much smaller workspace on the same NAS
got.

Load the pending queue once, right after discovery completes, and keep it in
memory; each chunk is claimed by advancing a cursor instead of re-querying.
A mid-scan Sort By / Sort Order change (DR-014 / issue #17) still reorders
work, but only the not-yet-claimed remainder of the in-memory list, not the
whole table. Also drop the now-redundant thumbnail_blob column from the
pending-enrichment SELECT in favor of a presence flag, so rows that already
have a (possibly large) thumbnail and are only pending on metadata no longer
pull that BLOB payload across the query at all.

failed_paths/enriched_unflushed bookkeeping is removed: both existed solely
to keep the old per-chunk re-query from re-selecting an in-flight or failed
row, which is now impossible since a row is popped from the in-memory queue
exactly once regardless of outcome.

Threading notes: no change to the Threading Contract. The dispatch loop
remains the sole writer to working_db and sole caller of on_progress/on_batch;
enrich_image_record() still runs only on ThreadPoolExecutor worker threads and
still returns plain ImageRecord data, never touching Qt objects.

Sets dev VERSION 0.9.4-bug.179-184 for this branch (issues #179, #184).
fix(db): use seek pagination for cached-thumbnail hydrate (issue #190)
All checks were successful
CI / lint (push) Successful in 16s
CI / test (push) Successful in 4m44s
CI / appimage (push) Successful in 16m48s
CI / build-appimage (push) Successful in 0s
CI / windows-exe (push) Successful in 19m39s
CI / build-windows-exe (push) Successful in 0s
688459d655
WorkingDB.iter_thumbnail_blob_dicts() -- used by CachedThumbnailLoadWorker to
display already-cached thumbnails on directory open, independently of the live
scan -- paged through cached rows with ORDER BY filename, relative_path LIMIT
? OFFSET ?. images had no index covering filename, so every chunk call did a
full-table scan + temp-b-tree sort of the whole table, then discarded OFFSET
rows before returning chunk_size. Cost per chunk grew with OFFSET, making a
full drain O(n^2) in image count: found live-testing the #184 branch on a
~125k-image network workspace with ~75k images already cached from prior
runs, where the first screenful of already-cached thumbnails (needing zero
decode work) failed to appear for minutes. Measured before/after on a
125k-row DB with 75k pre-cached thumbnails, chunk_size=200 (matching the real
worker): a chunk near the end of the cached range went from 4.9s to 0.16s,
and a full drain of all 75k cached rows went from an extrapolated many
minutes to 0.95s.

Fix: add idx_images_filename(filename, relative_path) (WORKING_DB_VERSION 3
-> 4, additive migration) and switch to keyset/seek pagination -- each chunk
seeks past the last (filename, relative_path) returned instead of
re-scanning from the start, so cost stays flat regardless of position.

This is a different bug from #184: #184's O(n^2) was in the live
pending-enrichment dispatch loop (already fixed); this one is in the
separate already-cached-thumbnail hydrate path that #184 did not touch.
Bundling it onto this branch since it was found testing #184's own fix and
touches the same file/table.

Threading notes: no change to the Threading Contract. iter_thumbnail_blob_dicts()
still only runs from CachedThumbnailLoadWorker's QRunnable.run() (worker
thread), still delivers plain dicts via the existing ThumbnailSignals queue;
no widget/model access added on the worker side.

VERSION: 0.9.4-bug.179-184 -> 0.9.4-bug.179-184-190.
fix(db): chunk SQL parameters and stage prune in a temp table (issue #191)
Some checks failed
CI / lint (push) Successful in 25s
CI / test (push) Successful in 10m22s
CI / build-appimage (push) Has been cancelled
CI / build-windows-exe (push) Has been cancelled
CI / appimage (push) Has been cancelled
CI / windows-exe (push) Has been cancelled
dd9032e613
Every WorkingDB query that expanded one "?" per caller-supplied id/path bound
the whole sequence in a single statement. Above the connection's
SQLITE_LIMIT_VARIABLE_NUMBER that raises "too many SQL variables".

That limit is compile-time. SQLite's own default is 999 below 3.32 and 32766
from 3.32 on; Debian/Ubuntu patch it to 250000, which is what this project's
Linux CI and dev machines report. So a 125k-image workspace passes every test
here and fails on the official CPython-for-Windows build -- which is where the
125k report came from.

The blast radius was much wider than one failed query. open_directory() wrapped
the preload's tag fetch AND _start_cache_hydrate_worker() in one try:

    plains = mgr.image_dicts_with_tags(existing)   # raises at 125k
    self.model.set_images(plains)                  # never runs
    ...
    self._start_cache_hydrate_worker()             # never runs
    except Exception as exc:
        logger.warning("Preload from DB failed (non-fatal): %s", exc)

So on Windows at 125k: no cached grid preload, no cached-thumbnail hydrate at
all, and no prune (prune_images_not_in raised the same way, swallowed into the
scan summary's errors list) -- all behind one logger.warning. The placeholders
the reporter saw came from the discovery scan; the cached thumbnails never
loaded because nothing was ever launched to load them.

This also means issue #190's fix, while correct on its own terms (the O(n^2)
OFFSET pagination was measured), could not have affected that report: the
worker whose query it optimized was never started on that platform.

Changes:

- _resolve_sql_variable_limit() reads the real budget from the connection via
  Connection.getlimit() (Python 3.11+), falling back to 999, reserving headroom
  for non-list parameters. Resolved once per WorkingDB on first connect.
- _chunked() splits any caller-supplied sequence to that budget. Applied to
  get_tags_for_images, get_images_by_paths, get_cache_metadata_for_paths,
  get_images_with_thumbnails_for_paths, get_dupe_group_representatives, and
  delete_images_by_relative_paths, merging per-chunk results.
- prune_images_not_in() rewritten around a temp table. A NOT IN list cannot be
  chunked by splitting it -- each chunk would match nearly every row and delete
  almost the whole table -- so the keep-set is inserted with executemany and
  matched via a subquery, removing the parameter ceiling entirely. It also
  gained an on_pruned callback that reports removed paths in batches, so the
  UI can stream removals instead of applying one lump at the end.
- open_directory()'s single try split in two, so a preload failure can no
  longer prevent the cached-thumbnail hydrate worker from starting.

Testing: 9 new tests in tests/db/test_sql_variable_limit.py force a small
variable limit on every connection, so this sandbox's patched 250000 cannot
mask a regression. All 9 verified to fail against the pre-fix code with the
exact OperationalError. Full suite 814 passed, 1 skipped; ruff clean; offscreen
UI-exit smoke clean.

Still needs confirmation on the reporting platform:
    import sqlite3
    sqlite3.connect(':memory:').getlimit(sqlite3.SQLITE_LIMIT_VARIABLE_NUMBER)
32766 confirms the diagnosis for that machine; 250000 would rule it out.

VERSION: 0.9.4-bug.179-184-190 -> 0.9.4-bug.179-184-190-191

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ
The scan and thumbnail workers report progress once per image. On the reported
125k-image workspace that is ~125,000 cross-thread queued signal emissions,
each allocating an event, taking the receiving thread's queue mutex, and
landing in a main-thread slot that rebuilds a status string and rewrites the
status label and progress bar. The per-call cost is O(1); the rate is what
starves the UI, and it competes with paint and input events in the same queue.

New core/progress_throttle.py (no Qt) forwards progress at ~10/s. A call is
always forwarded when it is the first, when the total changes, or when it is
the final item; otherwise only once the interval has elapsed. flush() delivers
the last dropped sample before finished, so the final on-screen count matches
the last item actually processed.

Applied at the signal-emission boundary in ScanWorker and ThumbnailWorker, not
inside the scanner functions. That placement is deliberate: the scanners' own
per-item on_progress callback is what several tests use to prove no image is
enriched twice under concurrent dispatch (see
test_ordered_metadata_scan_no_duplicate_processing_under_concurrency, which
asserts sorted(seen_paths) == names). Throttling inside image_scanner.py would
destroy that contract; throttling the emission leaves it untouched.

Emitted counts are always the real ones, never sampled, so RateTracker's
10-second rolling window and its ETA are unaffected -- ~10 samples/second is
far more resolution than that window needs.

Testing: 8 new tests drive a fake monotonic clock to pin the emission-count
bound, exact count forwarding, stage-transition and final-item passthrough,
and flush behavior, plus an end-to-end ScanWorker test asserting the scanner
still sees all 5000 items while fewer than 100 signals are emitted. Scan suites
(test_pr9_sort, test_image_scanner, test_thumbnail_generator,
test_thumbnail_benchmark) all pass unchanged; ruff clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ
perf(grid): stop resorting the whole model on filter-only changes (issue #191)
All checks were successful
CI / lint (push) Successful in 29s
CI / test (push) Successful in 8m23s
CI / windows-exe (push) Successful in 17m9s
CI / build-windows-exe (push) Successful in 0s
CI / appimage (push) Successful in 19m46s
CI / build-appimage (push) Successful in 0s
9e671e8b8d
Returning from the viewer, and every tag add/remove/rename, called
filter_proxy.invalidate() unconditionally. That re-sorts AND re-filters the
entire source model. Neither operation changes row ordering unless the active
sort key reads tags -- SORT_TAG_PRIORITY is the only one of the five that does;
filename/date/size/resolution never consult them.

Measured on a synthetic model (offscreen, 40,000 rows):

    invalidate()       (re-sort + re-filter)    2.580s
    invalidateFilter() (re-filter only)         0.178s   14.5x cheaper

Extrapolated to the reported 125k-row workspace that is roughly 8.9s -> 0.6s
per call, which matches the reported "5-30 second freeze when opening an image
and returning to the grid" far better than anything measured previously.
_refresh_tag_filter_views() has ~14 call sites, so the same cost was also being
paid on ordinary tagging.

Changes:

- ImageFilterProxy.invalidate_filter_only() for view transitions that change no
  data (viewer -> grid), and invalidate_after_tag_change() which re-sorts only
  when _sort_key == SORT_TAG_PRIORITY. Putting the condition inside the proxy
  keeps all ~14 call sites honest.
- ImageListModel.row_data() returns the live row dict without copying.
  lessThan() ran twice per comparison and filterAcceptsRow() once per row, both
  through get_image(), which copies -- roughly four million throwaway dicts per
  full re-sort at 125k rows. Neither mutates what it reads.
- ImageListModel.sort_tuple() caches the sort key per row, built from
  core.image_sort.sort_key_tuple so proxy order and the scan queue's order stay
  identical by construction. Invalidated per row by every mutator that can
  change a sort field; apply_thumbnail_batch deliberately does not invalidate,
  since no sort key reads thumbnail_blob.
- remove_images() now emits one beginRemoveRows/endRemoveRows per contiguous
  run instead of one per row (20 rows in two runs: 20 signals -> 2). Each pair
  forces the proxy to remap, and streaming prune will push batches through here.
- upsert_images() no longer evicts a row's decoded pixmap unless the incoming
  dict actually carries a thumbnail_blob, and no longer marks a row changed
  when the merge changed nothing. Discovery re-upserts a placeholder dict for
  every file it walks, so the old unconditional pop discarded every thumbnail
  the cache-hydrate worker had just delivered and then repainted them all.

Honest note on the sort-key cache: on its own it is only ~1.2x on a full
re-sort (2.62s vs 3.09s at 40k rows). The dominant cost of a re-sort is
PySide6's per-comparison Python/C++ callback overhead, which caching cannot
remove. It is kept because it makes the re-sorts that genuinely must happen
cheaper and removes the ~4M dict allocations; the 14.5x above comes from not
re-sorting at all when there is no ordering reason to.

One existing test updated: test_pr8_viewer.py::test_viewer_restore_invalidates_
filter_once asserted its "once" contract by monkeypatching invalidate() itself,
so it was coupled to the method rather than the behavior it names. It now
asserts exactly one filter-only invalidation and zero full re-sorts, which is
strictly stronger.

Testing: 10 new tests in tests/test_image_filter_proxy_perf.py, all verified to
fail against the pre-fix code -- including the two behavioral ones
(placeholder upsert evicting a hydrated pixmap; 20 removal signals where 2
suffice). Full suite 832 passed, 1 skipped; ruff clean; offscreen UI-exit smoke
clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ
fix(db): chunk the dupe-path SQL parameter expansions too (issue #191)
All checks were successful
CI / lint (push) Successful in 32s
CI / test (push) Successful in 8m36s
CI / windows-exe (push) Successful in 19m13s
CI / build-windows-exe (push) Successful in 0s
CI / appimage (push) Successful in 21m49s
CI / build-appimage (push) Successful in 0s
1fb10a4dde
Confirmed on the reporting platform: the affected Windows 11 build reports
SQLITE_LIMIT_VARIABLE_NUMBER = 32766, SQLite's unpatched compile-time default.
That makes issue #191 a measured diagnosis rather than an inferred one, and
puts the breakage threshold at ~32,766 images -- which also explains #184's A/B
comparison, where the same share behaved normally at ~3,900 images and did not
at ~125,000.

Three further call sites scale with duplicate-group count rather than image
count, and a 125k workspace can accumulate more than 32,766 dupe-group tags:

- remove_dupe_tags_with_single_member() -- highest risk, runs after every scan
- get_image_dicts_for_dupe_tags()
- get_dupe_group_representatives()

All three now chunk to the connection's parameter budget. The expansions left
unchunked are genuinely bounded: one by column count, two by the user's active
tag-filter selection.

Also fixes a bug in the previous commit's own clamp.
_resolve_sql_variable_limit() used max(64, min(limit - RESERVED, 20000)), whose
floor can return a chunk size LARGER than the reported limit when that limit is
small -- defeating the headroom reserved for the fixed parameters some queries
bind alongside the chunked list (get_dupe_group_representatives binds the
DELETE tag name plus its chunk). Now min(max(1, limit - RESERVED), 20000).
Caught by the new dupe-representative test at a simulated limit of 64, which
failed with the exact OperationalError this issue is about.

Testing: 3 new tests (12 total in tests/db/test_sql_variable_limit.py). Full
suite 835 passed, 1 skipped; ruff clean; offscreen UI-exit smoke clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ
Phase 2 of the responsive-open work.

The hydrate-on-open copy and every sync-to-share copy ran inline on the main
thread, kept nominally responsive only by _pump_ui_events() calls fired from
inside their own progress callbacks. On the reported 125k workspace the local
cache carries thumbnail BLOBs and runs to a gigabyte or more, so those copies
blocked the event loop for as long as the share took to move the file. The
5-minute background sync timer did the same thing with no dialog at all to
explain the freeze.

- New core/workspace_sync_worker.py::WorkspaceCopyWorker (QRunnable). Threading
  notes: the payload is filesystem open()/read()/write() on Path objects plus,
  in sync mode, one short-lived sqlite3 WAL-checkpoint connection opened and
  closed inside the worker thread -- matching the existing "connections are
  short-lived, never shared across threads" contract. No WorkingDB instance and
  no Qt object crosses the boundary; progress/status/finished/error carry ints,
  strs and dicts only.
- New WorkspaceCopySignals in worker_utils.py, following the established
  per-worker signals + create_*_signals() factory pattern.
- MainWindow._run_workspace_copy() drives the worker while keeping the
  surrounding synchronous shape callers depend on: it pumps events (which is
  also what delivers the worker's queued progress signals to the dialog) and
  relays Cancel through to the worker. Used by the hydrate-on-open path and by
  _sync_workspace_cache_to_share_if_needed() for all three of its modes (modal
  close/switch, status-bar post-scan, silent 5-minute timer).
- import_share_metadata() now bulk-upserts in batches of 500 and reports
  progress per batch instead of one upsert_image() and one on_progress() per
  image -- the caller's progress handler pumps the event loop, so the old shape
  cost 125k serial writes AND 125k processEvents() calls on the GUI thread for
  a legacy metadata-only share. Its tag-assignment loop is likewise batched
  through the existing add_tags_to_images().

Testing: 7 new tests covering hydrate, sync, cancel-removes-partial-destination,
error reporting, the merge batching (asserts zero per-row writes and <=10
progress calls for 1000 rows), and that the resolved chunk budget stays under
the connection's real limit. Network dialog suites pass unchanged; ruff clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ
Phase 3 of the responsive-open work.

scan_directory() is already a streaming generator, but
scan_placeholders_and_upsert() drained it fully into a list before writing a
single row or emitting a single batch, then made a second pass to upsert. On a
125k-image network tree that meant no placeholder reached the grid until the
entire walk had finished. Its own comment recorded the scale it was written for:
"Memory for 10k paths is negligible".

Rows are now upserted and published as the walk yields them. Two cadences are
kept deliberately separate:

- on_batch (grid population) fires per batch_size records, as the walk runs.
- on_progress walk feedback stays on its own DISCOVERY_FOUND_PROGRESS_STEP
  cadence, so a slow network walk keeps reporting even while a batch is still
  filling. Preserving that is why
  test_placeholder_discovery_progress_every_twenty_found still passes unchanged
  -- it guards exactly this, and the first cut of this change broke it by
  folding walk feedback into the batch flush.

on_total() still fires once the walk ends (it cannot know the total earlier),
followed by the determinate seeded/total tick, so the UI's indeterminate ->
determinate stage transition is unchanged.

A side effect worth noting: cancelling during discovery now keeps whatever was
seeded. Previously the collect-then-write shape meant a cancel mid-walk
discarded every row it had found, since nothing had been written yet.

Testing: 4 new tests in tests/test_scan_streaming.py, three verified to fail
against the pre-fix code -- batches all arriving after the walk (assert 30 < 30),
zero rows written mid-walk (0 >= 8), and cancel leaving nothing behind
(assert 0 < 0). Existing scanner and sort suites pass unchanged; ruff clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ
Phase 4 of the responsive-open work, and the one the owner's requirement 5
named directly: "Once [the directory scan] completes, the database prune/update
should trigger - this should be updating the grid/image strip/working set with
pruned and added images in near real-time."

Prune, singleton dupe-tag cleanup, and record_scan all ran at the very end of
scan_metadata_ordered_and_upsert() -- after EVERY thumbnail had been generated.
On a 125k workspace that defers reconciliation by hours past the walk that
established which files actually exist, and lands every removal in one lump.

Separately, both ScanWorkers were started back to back and the metadata one
spent that whole window in a time.sleep(0.05) poll loop occupying a QThreadPool
thread -- the same thread the cache-hydrate worker needs while the user is
waiting for thumbnails to appear.

Changes (recorded as DR-045):

- scan_placeholders_and_upsert() gained on_pruned and now, on clean completion,
  prunes rows whose files the walk did not find and removes singleton dupe-group
  tags. Discovery already owns the authoritative filesystem path set and has
  already paid for the walk, so this is where DR-007's "filesystem is the source
  of truth" belongs. Summary carries pruned/pruned_paths/dupe_links_removed.
- scan_metadata_ordered_and_upsert() no longer prunes or cleans dupe tags. It
  still records scan_history, whose new-image count needs enrichment to have run.
- ScanSignals.images_pruned (plain list[str]) -> MainWindow._on_images_pruned(),
  which drops rows in the batches discovery reports them. Combined with the
  contiguous-run remove_images() from 9e671e8, a streamed prune costs one
  beginRemoveRows per run rather than one per row.
- _launch_directory_scan() starts only discovery; _on_placeholder_finished()
  starts enrichment via the new _start_metadata_scan_worker(), freeing a pool
  thread during discovery and giving strict phase ordering for free -- the
  existing is_discovery_complete() gate is set only after the discovery worker
  including its new tail has finished. So: walk -> prune -> enrichment, with
  prune done before the first thumbnail, which is the serialization the owner
  chose over overlap (prune writes vs 16 enrichment threads on one writer lock).
- _prune_singleton_dupe_groups() no longer runs unconditionally in
  _on_deep_finished(); the post-prune tag refresh is gated on
  dupe_links_removed > 0, since that helper reloads every model row's tags
  (O(workspace)) and the common case has nothing to clean up.

Cancel semantics are stricter and deliberately so: a cancelled walk skips prune,
dupe cleanup and history, and does not start enrichment. A partial walk has not
established which files exist, so deleting rows it merely did not reach would be
data loss.

DR-014 is untouched: enrichment still processes in current UI sort order and
still re-orders its remaining queue on a mid-scan sort change.

Two existing tests updated, both because they asserted a behavior that moved
rather than a behavior that broke:
- test_pr9_sort.py::test_metadata_scan_skips_stale_db_rows_not_on_disk bundled
  two claims. Enrichment *skipping* stale rows is still its subject and still
  asserted; *removing* them is now discovery's job, so it asserts pruned == 0
  and the rows surviving, with prune coverage moved to test_scan_streaming.py.
- test_workspace_cache.py::test_import_share_metadata_reports_progress_and_
  respects_cancel drove its cancel predicate off the progress-call count, which
  no longer ticks per image after the Phase 2 batching. Cancellation is still
  checked per row, so it now drives cancel from its own counter and asserts the
  merge stopped before flushing anything; a companion test covers the batched
  progress cadence.

Testing: 5 new tests in tests/test_scan_streaming.py (prune runs in discovery,
streams in batches, is skipped on cancel; enrichment starts only after discovery
finishes, and not at all when it was cancelled). Full suite 850 passed,
1 skipped; ruff clean; offscreen UI-exit smoke clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ
feat(thumbnails): work in the user's sort order, with viewport jump-ahead (issue #191)
All checks were successful
CI / lint (push) Successful in 16s
CI / test (push) Successful in 6m16s
CI / appimage (push) Successful in 17m22s
CI / build-appimage (push) Successful in 0s
CI / windows-exe (push) Successful in 19m55s
CI / build-windows-exe (push) Successful in 0s
4cf201de19
Phase 5 of the responsive-open work, covering the owner's requirement 6:
"missing thumbnails should start generating from the beginning of the sort order
the user has selected."

Both thumbnail paths used hardcoded SQL ordering.
iter_thumbnail_blob_dicts() is ORDER BY filename, relative_path and
get_relative_paths_missing_thumbnails() is ORDER BY filename, and neither worker
accepted a work order at all. With any sort other than filename-ascending active,
hydration and catch-up generation filled the grid in from the wrong end entirely
-- the user watched the rows they were looking at stay blank while work happened
somewhere off screen.

Supporting all five sort keys times two directions in SQL would mean ten
index/query variants, and tag_priority cannot be indexed at all. The main thread
already knows the order, so it supplies it:

- ImageListModel.relative_paths_in_sort_order() builds the order from the same
  cached sort tuples lessThan uses, so it is by construction the order the grid
  displays.
- WorkingDB.get_thumbnail_blobs_for_paths() resolves an explicit, caller-ordered
  chunk via idx_images_relpath -- no OFFSET, and it does not need the filename
  index added for #190. It returns a dict because SQL IN makes no order
  guarantee; the worker re-imposes its own chunk order.
- CachedThumbnailLoadWorker takes relative_paths and walks them in order.
  Passing nothing keeps the existing seek-paginated whole-table walk, so #190's
  fix stays live and covered for that path.
- _start_thumbnail_generation() re-sequences the missing set into the current
  sort order before handing it to generate_thumbnails_for_db(), which already
  accepted an explicit work list.

Viewport jump-ahead (the owner's choice over strict order alone): at 125k images
the user can scroll far past the head of the sort order long before the queue
reaches them. New core/work_queue.py::OrderedWorkQueue (no Qt) hands out
promoted paths first, then sort order, and never hands out the same path twice
however many times it is promoted. The grid's vertical scrollbar drives a 150ms
debounced promotion of the visible range -- covering wheel, drag, keyboard and
programmatic scrolls alike. Only plain strings cross the thread boundary; the
main thread writes via promote(), the worker reads via take(), one lock guards
both.

reorder() is present for a mid-run Sort By change to re-sequence remaining work
without redoing anything already handed out; wiring that to the sort controls is
left as a follow-up, since the requirement is about where work *starts*.

One existing test updated: test_pr13_config.py's background-hydrate test blocked
on count_images_with_thumbnails() to prove open_directory() returns without
waiting. The ordered path never calls that method, so the worker finished before
the assertion. It now blocks whichever first DB call the worker makes, covering
both the ordered and fallback paths -- the test's actual intent is unchanged.

Testing: 14 new tests covering queue ordering/promotion/idempotence/reorder,
model sort ordering in both directions, ordered emission, promotion moving the
visible range to the front, the unordered fallback still working, the DB helper
skipping blobless and absent rows, and that the hydrate worker is handed the
grid's current sort order. Full suite 866 passed, 1 skipped; ruff clean;
offscreen UI-exit smoke clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ
fix(ui): populate the grid in chunks so preload stops freezing the UI (issue #191)
All checks were successful
CI / lint (push) Successful in 29s
CI / test (push) Successful in 5m41s
CI / windows-exe (push) Successful in 19m51s
CI / build-windows-exe (push) Successful in 0s
CI / appimage (push) Successful in 21m10s
CI / build-appimage (push) Successful in 0s
43d75f829c
Reported regression: "Preparing workspace" replaced the sync dialog and left the
application non-interactive for a long stretch, with the cached thumbnails
loading before it moved on.

Reproduced offscreen on a 125,000-row cached workspace: open_directory() took
12.7s, entirely inside a single "Preparing workspace..." status with no progress
and no event-loop pumping. Profiled:

    9.87s  ImageListModel.set_images
    1.27s  WorkingDB.get_all_images
    1.03s  _prune_disabled_types_from_current_db
    0.60s  TagManager.image_dicts_with_tags

This is a consequence of fixing #191 rather than a new cost: on the reporter's
Windows build the preload used to raise "too many SQL variables" and abort in
milliseconds, so this work never ran at all. Now it runs.

set_images() itself is only ~0.3s on a bare model. The other ~9.5s is the
attached sort/filter proxy rebuilding its entire mapping in one uninterruptible
operation when the model resets. Measured alternatives, all worse:

    set_images, proxy attached (current)      10.20s
    dynamicSortFilter off during populate     27.83s
    proxy detached during populate            18.44s
      (filter-only pass)                       0.55s
      (full re-sort)                           9.08s

So the sort dominates, and it is not our lessThan: replacing it with a
precomputed role compared natively in C++ via setSortRole only moved 9.56s ->
8.72s. The real cost is ~2 million data() calls crossing into a Python model,
which is inherent to sorting this many rows through a proxy.

It cannot be made much smaller, but it can be broken up. New
_preload_rows_into_model() inserts PRELOAD_CHUNK_ROWS (2,000) at a time, pumping
the event loop and checking the cancel checkpoint between chunks:

    one-shot set_images                       10.28s   one 10.3s freeze
    chunked add_images (2,000/chunk)           6.84s   worst chunk 269ms

Faster overall AND the freeze becomes ~270ms slices. open_directory() on the
125k reproduction now returns in 9.5s with a live count ticking every ~100ms,
the progress bar moving, rows appearing progressively, and Cancel working
throughout -- instead of one silent non-interactive block.

This is the streaming preload deliberately deferred when the phases were
planned, on the reasoning that the synchronous version satisfied requirement 4
literally. That reasoning was wrong for the freeze it caused; the report is what
showed it.

Testing: 3 new tests (chunked insert with no blocking set_images call, per-chunk
progress reporting, cancel stopping mid-preload). Full suite 869 passed,
1 skipped; ruff clean; offscreen UI-exit smoke clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ
perf(open): warn on large workspaces and cut the remaining preload overhead (issue #191)
All checks were successful
CI / lint (push) Successful in 17s
CI / test (push) Successful in 5m21s
CI / windows-exe (push) Successful in 17m18s
CI / build-windows-exe (push) Successful in 0s
CI / appimage (push) Successful in 18m38s
CI / build-appimage (push) Successful in 0s
a5ed8f9ead
Follow-up to 43d75f8, which chunked the grid population. Two further changes to
the "Preparing workspace" phase.

1. Tell the user when it will take a while.

The phase is unavoidably multi-second on a large cached workspace, so it now
says so instead of leaving the user guessing. Above
LARGE_WORKSPACE_PRELOAD_ROWS (20,000) the status reads "Preparing large
workspace, the app may be briefly unresponsive..." rather than the plain
"Preparing workspace...".

The size is resolved with a COUNT(*) before any of the slow work, so the warning
appears up front rather than three seconds in -- the read, the type prune and
the grid population all happen after it. The chunked insert then keeps the same
wording alongside its running count.

2. Remove work that was pure overhead.

Profiled on a 125,000-row cached workspace (offscreen), open_directory() total:

    before this commit          9.33s
    after                       8.25s

- Type prune: 1.03s -> 0.09s. It built a pathlib.Path per row purely to read
  .suffix. New path_utils.relative_path_suffix() does it with string slicing,
  ~20x cheaper. It is exactly equivalent for every path this app stores --
  relative POSIX strings from the scanner's os.scandir walk -- verified against
  pathlib over 30,000 generated realistic paths, plus a unit test. It is
  deliberately not a general Path.suffix replacement: it diverges on trailing
  separators and "." components, which a filesystem walk cannot produce. The
  live Settings image-type toggle, which walks every row the same way, uses it
  too.

- Row read: get_all_images() built 125,000 ImageRecord dataclasses that the
  preload converted straight back to dicts via to_dict() -- about 1.2s of round
  trip. New WorkingDB.get_all_image_dicts() and
  TagManager.attach_tags_to_dicts() give the preload the dict shape directly.
  The record-based image_dicts_with_tags() is unchanged for its other caller.

What remains is 6.67s of grid population, which is the sort/filter proxy
building its row mapping. That is the architectural floor described in 43d75f8:
a native setSortRole comparison only moved it ~10%, because the cost is ~2
million data() calls into a Python model, not our lessThan. Shrinking it further
means pre-sorting the source and making the proxy filter-only, which touches
DR-014, selection handling, and the viewer's row mapping -- deliberately not
attempted here.

Net for the reported symptom: 12.73s of silent freeze -> 8.25s that reports its
size up front, ticks a live count every ~100ms, and stays interactive with
Cancel working.

One existing test updated: test_open_directory_warns_before_reading_rows (added
earlier in this same commit's work) patched get_all_images, which the preload no
longer calls; it now patches get_all_image_dicts.

Testing: 6 new tests (warning threshold wording, warning ordering before the row
read, dict fast path matching the record path field for field, attach_tags_to_dicts
matching image_dicts_with_tags, and relative_path_suffix matching pathlib).
Full suite 874 passed, 1 skipped; ruff clean; offscreen UI-exit smoke clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ
fix(ui): show sort planning progress before moves start
All checks were successful
CI / lint (push) Successful in 17s
CI / test (push) Successful in 3m46s
CI / appimage (push) Successful in 15m50s
CI / build-appimage (push) Successful in 0s
CI / windows-exe (push) Successful in 52m54s
CI / build-windows-exe (push) Successful in 0s
6bd367b351
After confirm, the Sort into Folders progress dialog opened at 0/N and
stayed there while the worker re-read tags and checked every destination
on disk. It now opens on a planning note with an indeterminate bar, then
ticks through those destination checks before the move stage begins.
Cancel during planning stops before any file is moved.

Threading notes: SortWorker still emits only plain ints and strings on
the existing SortSignals.progress signal. Planning and moving stay on
the worker thread; the dialog updates from the main-thread slot.
fix(ui): keep sort planning on screen while files move
All checks were successful
CI / lint (push) Successful in 9s
CI / test (push) Successful in 3m45s
CI / appimage (push) Successful in 16m40s
CI / build-appimage (push) Successful in 0s
CI / windows-exe (push) Successful in 19m29s
CI / build-windows-exe (push) Successful in 0s
90bed7483b
The progress dialog stacks Stage 1 (Planning Moves) and Stage 2
(Moving Files). Moving updates only its own line and bar. Planning
stays at its last count for the rest of the operation.
fix(sort): stop per-file database reconnects in the move loop
All checks were successful
CI / lint (push) Successful in 6s
CI / test (push) Successful in 3m33s
CI / appimage (push) Successful in 16m23s
CI / build-appimage (push) Successful in 0s
CI / windows-exe (push) Successful in 16m57s
CI / build-windows-exe (push) Successful in 0s
08ebc74789
Each Sort into Folders path update opened a new SQLite connection, reran
pragmas, and rechecked the schema. 400 updates took 14.7s (27 files/s)
before any disk move. Updates now share one connection and still commit
per file, so the writer lock is not held between files. Planning lists
each destination folder once instead of stating every file, and the
dialog status line is built only for samples the progress throttle emits.

A 400-file local sort now finishes in 0.15s.

Threading notes: the reused connection is opened and closed on the worker
thread. Progress signals stay plain ints and strings.
docs: PR-prep for 0.10.0
Some checks failed
CI / lint (push) Successful in 8s
CI / test (push) Failing after 2m11s
CI / appimage (push) Has been skipped
CI / windows-exe (push) Has been skipped
CI / build-appimage (push) Failing after 0s
CI / build-windows-exe (push) Failing after 0s
7df56b92a2
Release bump 0.9.4 -> 0.10.0. Changelog, design, outline, and the user
guide cover streamed discovery with prune before enrichment, sort-order
thumbnails, and the stacked Sort into Folders progress dialog.
fix(db): pause cyclic GC while any thread is inside sqlite3
Some checks failed
CI / lint (push) Successful in 9s
CI / test (push) Failing after 2m33s
CI / appimage (push) Has been skipped
CI / windows-exe (push) Has been skipped
CI / build-appimage (push) Failing after 0s
CI / build-windows-exe (push) Failing after 0s
1ad83457e2
CPython 3.12 can segfault when one thread finalizes a sqlite3
connection while another thread is inside the module. Directory open
hit that window: the thumbnail hydrate worker and the main thread both
apply pragmas at the same time.

Threading notes: workers still open their own connections and still
emit plain data only. The guard is a process-wide refcount around
sqlite3 calls. It does not share a connection across threads and does
not touch widgets.
fix(db): keep cyclic GC off the sqlite3 module
All checks were successful
CI / lint (push) Successful in 6s
CI / test (push) Successful in 3m37s
CI / appimage (push) Successful in 13m25s
CI / build-appimage (push) Successful in 0s
CI / windows-exe (push) Successful in 24m45s
CI / build-windows-exe (push) Successful in 0s
b981780aac
Leaving the duplicate viewer still segfaulted on CI: a comparison-image
load was inside a collection while the main thread counted duplicate
tags. Disabling GC only for the duration of a call does not stop a
collection that has already started, and waiting for that collection
deadlocks when a finalizer needs the UI thread.

Automatic cyclic GC now stays off for the process. Collections run on
the main thread when no sqlite3 call is in flight. A collect from a
worker, or from the main thread during a call, returns without running.

Threading notes: workers still open their own connections and never
collect. The UI thread does not wait on a worker collector.
ValleyGeek deleted branch 0.9.4/issue-179-184-dupe-pip-thumb-perf 2026-11-02 05:09:59 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
ai-collab/bulk-image-organizer!195
No description provided.