ai-collab/bulk-image-organizer

Watch

Fork 0

[Bug]: Duplicate-set switching is 1–2s on large workspaces despite neighbor image cache #182

New issue

Closed

opened 2026-08-27 20:41:18 +00:00 by Grok · 2 comments

Grok commented

2026-08-27 20:41:18 +00:00

Member

Affected area

Duplicates workbench — switching the active duplicate set for review (Multi comparison, Single viewer, W/S next/prev, toolbox list click). Neighbor image preload (DupeMultiGrid.preload_group / _image_cache) is supposed to make adjacent-set changes near-instant.

Report

On a large workspace (~9,000+ images, ~600+ duplicate sets), changing from one duplicate set to the next is delayed 1–2 seconds. Adjacent-group image caching exists specifically so this navigation is near-instant; at this scale it is not.

This is the same user-visible failure as closed issue #118 ([Bug]: Slow Dupe Group Change), which was fixed for hundreds of images. Changelog 0.2.1 recorded that group switching became “near-instant … even in directories with hundreds of images.” Issue #138 later cut Dupe Grid reload/select cost at ~2,200 images / ~270 groups. The 9k / 600+ case still hits a 1–2s pause per set change.

Steps to reproduce

Open a workspace with ~9,000+ images and ~600+ duplicate groups (F-DUPE_* / V-DUPE_*).
Open the Duplicates workbench and review in Multi (and/or Single).
Step through nearby sets with W / S, the group list, or next/prev.
Repeat on adjacent groups after the first pair has had time to preload.

Expected

Switching to an already-preloaded neighboring set is near-instant (on the order of a frame or two, not a perceptible stall).
Neighbor decode/preload stays off the critical path; the current set paints from cache.
Cost of a set change does not scale with total workspace size (9k images / 600 groups).

Actual

1–2 second delay between selecting the next set and the review UI catching up, even for nearby sets that should already be in the image cache.

Code notes (investigation, not a measured profile)

Every set change still runs main-thread work that the image cache does not skip:

Full-model filter pass — _apply_dupe_group_filter → ImageFilterProxy.set_dupe_group_filter → invalidateFilter() walks all ~9,000 source rows (filterAcceptsRow). #118 stopped the extra full sort; the filter pass is still O(workspace).
Dupe list/status refresh — _refresh_tag_filter_views(..., refresh_dupe_list=False) still calls _refresh_dupe_workbench_from_filter, which re-queries get_dupe_tag_summaries() (aggregate over all dupe memberships) on each navigation.
DB round-trips even on a cache hit — _refresh_dupe_multi_view always does get_all_tags() + get_image_dicts_for_dupe_tag for the current set, then _preload_adjacent_dupe_groups does two more get_image_dicts_for_dupe_tag calls. Each WorkingDB call opens a new connection. The QImage cache is only consulted after that, inside set_group → _rebuild_grid.
Grid rebuild on every switch — DupeMultiGrid.set_group always _rebuild_grid(): detach pooled cells, destroy row wrappers, recompute tight layout, reparent cells, QImage.copy() + set_source_image (full-res pixmap on the UI thread). Cell pooling and _image_cache avoid re-decode and widget construction, not this layout/paint work.
Preload radius is ±1 and decode is async. If the stall is the items above, waiting for preload cannot help — the pause happens before/without using the cached pixels as a fast path.

Related: #118 (closed), #138 (Dupe Grid batching / skip invisible Multi refresh).

Suggested fix direction

Profile first (confirm which of the above dominates at 9k/600). Likely direction:

Do not re-filter the 9k-row thumbnail model on every dupe-set change unless a view actually showing that proxy needs it (Multi/Single review can bind the group from the dupe-tag query/cache alone).
Skip get_dupe_tag_summaries / get_all_tags / get_image_dicts_for_dupe_tag when the destination set (and its neighbors) are already known and image-cached.
Fast path in set_group: if the new group's images are in _image_cache and the cell count/layout key is unchanged, swap pixmaps/metadata without tearing down row widgets.
Keep preload off the UI thread; consider a slightly larger neighbor radius only after the switch path itself is cheap.

Threading: workers may decode QImages and emit them; widgets/pixmaps/layout stay on the main thread.

Acceptance criteria

On a ~9,000-image / ~600-group workspace, switching to an adjacent, already-preloaded duplicate set is near-instant (no 1–2s UI stall).
Jumping to a set that is not in the neighbor cache may still decode, but must not re-scan/re-filter the whole workspace as a prerequisite.
Dupe Grid selection (when Multi is not visible) stays as cheap as after #138 — no regression that reloads Multi on every Grid click.
W/S, toolbox list, and next/prev all share the fast path.
Keep/DELETE/tag actions and undo in Multi/Single still update the current set correctly.

### Affected area Duplicates workbench — switching the **active duplicate set** for review (Multi comparison, Single viewer, W/S next/prev, toolbox list click). Neighbor image preload (`DupeMultiGrid.preload_group` / `_image_cache`) is supposed to make adjacent-set changes near-instant. ### Report On a large workspace (**~9,000+ images, ~600+ duplicate sets**), changing from one duplicate set to the next is delayed **1–2 seconds**. Adjacent-group image caching exists specifically so this navigation is near-instant; at this scale it is not. This is the same user-visible failure as closed issue **#118** (`[Bug]: Slow Dupe Group Change`), which was fixed for **hundreds** of images. Changelog 0.2.1 recorded that group switching became “near-instant … even in directories with hundreds of images.” Issue **#138** later cut Dupe Grid reload/select cost at ~2,200 images / ~270 groups. The 9k / 600+ case still hits a 1–2s pause per set change. ### Steps to reproduce 1. Open a workspace with ~9,000+ images and ~600+ duplicate groups (`F-DUPE_*` / `V-DUPE_*`). 2. Open the Duplicates workbench and review in **Multi** (and/or **Single**). 3. Step through nearby sets with **W** / **S**, the group list, or next/prev. 4. Repeat on adjacent groups after the first pair has had time to preload. ### Expected - Switching to an already-preloaded neighboring set is **near-instant** (on the order of a frame or two, not a perceptible stall). - Neighbor decode/preload stays off the critical path; the current set paints from cache. - Cost of a set change does **not** scale with total workspace size (9k images / 600 groups). ### Actual - **1–2 second delay** between selecting the next set and the review UI catching up, even for nearby sets that should already be in the image cache. ### Code notes (investigation, not a measured profile) Every set change still runs **main-thread** work that the image cache does not skip: 1. **Full-model filter pass** — `_apply_dupe_group_filter` → `ImageFilterProxy.set_dupe_group_filter` → `invalidateFilter()` walks **all ~9,000 source rows** (`filterAcceptsRow`). #118 stopped the extra full **sort**; the filter pass is still O(workspace). 2. **Dupe list/status refresh** — `_refresh_tag_filter_views(..., refresh_dupe_list=False)` still calls `_refresh_dupe_workbench_from_filter`, which re-queries `get_dupe_tag_summaries()` (aggregate over all dupe memberships) on each navigation. 3. **DB round-trips even on a cache hit** — `_refresh_dupe_multi_view` always does `get_all_tags()` + `get_image_dicts_for_dupe_tag` for the current set, then `_preload_adjacent_dupe_groups` does two more `get_image_dicts_for_dupe_tag` calls. Each `WorkingDB` call opens a new connection. The QImage cache is only consulted **after** that, inside `set_group` → `_rebuild_grid`. 4. **Grid rebuild on every switch** — `DupeMultiGrid.set_group` always `_rebuild_grid()`: detach pooled cells, destroy row wrappers, recompute tight layout, reparent cells, `QImage.copy()` + `set_source_image` (full-res pixmap on the UI thread). Cell pooling and `_image_cache` avoid re-decode and widget *construction*, not this layout/paint work. 5. **Preload radius is ±1** and decode is async. If the stall is the items above, waiting for preload cannot help — the pause happens before/without using the cached pixels as a fast path. Related: #118 (closed), #138 (Dupe Grid batching / skip invisible Multi refresh). ### Suggested fix direction Profile first (confirm which of the above dominates at 9k/600). Likely direction: - Do **not** re-filter the 9k-row thumbnail model on every dupe-set change unless a view actually showing that proxy needs it (Multi/Single review can bind the group from the dupe-tag query/cache alone). - Skip `get_dupe_tag_summaries` / `get_all_tags` / `get_image_dicts_for_dupe_tag` when the destination set (and its neighbors) are already known and image-cached. - Fast path in `set_group`: if the new group's images are in `_image_cache` and the cell count/layout key is unchanged, swap pixmaps/metadata without tearing down row widgets. - Keep preload off the UI thread; consider a slightly larger neighbor radius only after the switch path itself is cheap. Threading: workers may decode `QImage`s and emit them; widgets/pixmaps/layout stay on the main thread. ### Acceptance criteria - [ ] On a ~9,000-image / ~600-group workspace, switching to an adjacent, already-preloaded duplicate set is near-instant (no 1–2s UI stall). - [ ] Jumping to a set that is **not** in the neighbor cache may still decode, but must not re-scan/re-filter the whole workspace as a prerequisite. - [ ] Dupe Grid selection (when Multi is not visible) stays as cheap as after #138 — no regression that reloads Multi on every Grid click. - [ ] W/S, toolbox list, and next/prev all share the fast path. - [ ] Keep/DELETE/tag actions and undo in Multi/Single still update the current set correctly.

Grok added the

Kind/Bug

Priority/High

Reviewed/Confirmed

labels

2026-08-27 20:41:32 +00:00

Grok referenced this issue from a commit

2026-08-27 20:55:37 +00:00

ui: make adjacent duplicate-set switching near-instant (issue #182)

Grok commented

2026-08-27 20:55:43 +00:00

Author

Member

Implemented on branch 0.9.1/issue-182-dupe-set-switch (VERSION 0.9.1-bug.182).

Adjacent duplicate-set review (W/S, toolbox list, next/prev) no longer does the work that made neighbor caching useless at ~9k images / ~600 groups:

Thumbnail proxy is not re-filtered on Multi/Grid navigation; the group tag is stored and applied when Single/viewer actually reads rows (ensure_filter_applied).
Dupe-list get_dupe_tag_summaries() is not re-run on a mere set change.
Current + neighbor membership dicts and tag colors are cached; switching to a preloaded neighbor hits that cache. Missing neighbors load in one get_image_dicts_for_dupe_tags round trip (no thumbnail blobs).
Already-reviewed groups skip the mark_dupe_group_reviewed write. Keyboard next/prev no longer double-apply via tag_selected.
Multi paints from a main-thread pixmap cache and drops decoded images outside the current ±1 window.

Please try W/S through nearby sets on the 9k/600 workspace. PR to main waits on your go-ahead.

Implemented on branch `0.9.1/issue-182-dupe-set-switch` (VERSION `0.9.1-bug.182`). Adjacent duplicate-set review (W/S, toolbox list, next/prev) no longer does the work that made neighbor caching useless at ~9k images / ~600 groups: - Thumbnail proxy is **not** re-filtered on Multi/Grid navigation; the group tag is stored and applied when Single/viewer actually reads rows (`ensure_filter_applied`). - Dupe-list `get_dupe_tag_summaries()` is not re-run on a mere set change. - Current + neighbor membership dicts and tag colors are cached; switching to a preloaded neighbor hits that cache. Missing neighbors load in **one** `get_image_dicts_for_dupe_tags` round trip (no thumbnail blobs). - Already-reviewed groups skip the `mark_dupe_group_reviewed` write. Keyboard next/prev no longer double-apply via `tag_selected`. - Multi paints from a main-thread pixmap cache and drops decoded images outside the current ±1 window. Please try W/S through nearby sets on the 9k/600 workspace. PR to `main` waits on your go-ahead.

Grok referenced this issue from a commit

2026-08-28 05:13:09 +00:00

ui: make adjacent duplicate-set switching near-instant (issue #182)

Grok referenced this issue from a commit

2026-08-28 17:49:40 +00:00

fix: drop ghost filmstrip slot when switching sets in Single (issue #182)

Grok referenced this issue from a pull request that will close it,

2026-08-28 19:34:46 +00:00

Make adjacent duplicate-set switching near-instant (0.9.3, #182) #188

Grok commented

2026-08-28 19:34:57 +00:00

Author

Member

Implemented and opened for review in PR #188 (0.9.3).

Adjacent set changes (W/S, toolbox, next/prev) no longer re-filter the full thumbnail model or re-query dupe summaries. Current ±1 membership and Multi pixmap cache are the fast path; Single applies the proxy filter when it reads rows.

Also covered: Single filmstrip ghost after DELETE + Hide Deleted + group switch (unmapped extra slot / 1:1 leftover image).

Closing this issue in favor of the PR.

Implemented and opened for review in PR #188 (`0.9.3`). Adjacent set changes (W/S, toolbox, next/prev) no longer re-filter the full thumbnail model or re-query dupe summaries. Current ±1 membership and Multi pixmap cache are the fast path; Single applies the proxy filter when it reads rows. Also covered: Single filmstrip ghost after DELETE + Hide Deleted + group switch (unmapped extra slot / 1:1 leftover image). Closing this issue in favor of the PR.

Grok closed this issue

2026-08-28 19:34:57 +00:00

ValleyGeek referenced this issue from a commit

2026-08-28 19:35:20 +00:00

Make adjacent duplicate-set switching near-instant (0.9.3, #182) (#188)

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

ai-collab/bulk-image-organizer#182

No description provided.

Rows
Columns