Large-workspace open, scan, and Sort into Folders (0.10.0) #195
No reviewers
Labels
No labels
Kind/Bug
Kind/Feature
Priority/High
Priority/Medium
Reviewed/Confirmed
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
ai-collab/bulk-image-organizer!195
Loading…
Reference in a new issue
No description provided.
Delete branch "0.9.4/issue-179-184-dupe-pip-thumb-perf"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Release 0.10.0 (from 0.9.4). Large workspaces stay usable while they open and scan, and Sort into Folders reports and finishes its work without the per-file stall.
Closes #179, #184, #190, and #191.
Tests
ruff check,ruff format --check, andpytest: 881 passed, 1 skipped, on the code commits. This PR-prep commit is version and docs only (VERSION0.10.0, changelog, design, outline, user guide)._after_tag_apply's dupes-Multi branch passed raw WorkingDB dicts (whose tag_colors is always {}) straight to sync_image_tags(), which overwrote the correct in-memory colors with the empty map from any get_image_dicts_for_dupe_tag() call -- turning pips gray after toggling DELETE, Keep, or any other tag apply while Multi is showing. Enrich via _enrich_dupe_group_images() first, matching the viewer-return path's _sync_dupe_multi_view_tags(). Sets dev VERSION 0.9.4-bug.179-184 for this branch (issues #179, #184).Every WorkingDB query that expanded one "?" per caller-supplied id/path bound the whole sequence in a single statement. Above the connection's SQLITE_LIMIT_VARIABLE_NUMBER that raises "too many SQL variables". That limit is compile-time. SQLite's own default is 999 below 3.32 and 32766 from 3.32 on; Debian/Ubuntu patch it to 250000, which is what this project's Linux CI and dev machines report. So a 125k-image workspace passes every test here and fails on the official CPython-for-Windows build -- which is where the 125k report came from. The blast radius was much wider than one failed query. open_directory() wrapped the preload's tag fetch AND _start_cache_hydrate_worker() in one try: plains = mgr.image_dicts_with_tags(existing) # raises at 125k self.model.set_images(plains) # never runs ... self._start_cache_hydrate_worker() # never runs except Exception as exc: logger.warning("Preload from DB failed (non-fatal): %s", exc) So on Windows at 125k: no cached grid preload, no cached-thumbnail hydrate at all, and no prune (prune_images_not_in raised the same way, swallowed into the scan summary's errors list) -- all behind one logger.warning. The placeholders the reporter saw came from the discovery scan; the cached thumbnails never loaded because nothing was ever launched to load them. This also means issue #190's fix, while correct on its own terms (the O(n^2) OFFSET pagination was measured), could not have affected that report: the worker whose query it optimized was never started on that platform. Changes: - _resolve_sql_variable_limit() reads the real budget from the connection via Connection.getlimit() (Python 3.11+), falling back to 999, reserving headroom for non-list parameters. Resolved once per WorkingDB on first connect. - _chunked() splits any caller-supplied sequence to that budget. Applied to get_tags_for_images, get_images_by_paths, get_cache_metadata_for_paths, get_images_with_thumbnails_for_paths, get_dupe_group_representatives, and delete_images_by_relative_paths, merging per-chunk results. - prune_images_not_in() rewritten around a temp table. A NOT IN list cannot be chunked by splitting it -- each chunk would match nearly every row and delete almost the whole table -- so the keep-set is inserted with executemany and matched via a subquery, removing the parameter ceiling entirely. It also gained an on_pruned callback that reports removed paths in batches, so the UI can stream removals instead of applying one lump at the end. - open_directory()'s single try split in two, so a preload failure can no longer prevent the cached-thumbnail hydrate worker from starting. Testing: 9 new tests in tests/db/test_sql_variable_limit.py force a small variable limit on every connection, so this sandbox's patched 250000 cannot mask a regression. All 9 verified to fail against the pre-fix code with the exact OperationalError. Full suite 814 passed, 1 skipped; ruff clean; offscreen UI-exit smoke clean. Still needs confirmation on the reporting platform: import sqlite3 sqlite3.connect(':memory:').getlimit(sqlite3.SQLITE_LIMIT_VARIABLE_NUMBER) 32766 confirms the diagnosis for that machine; 250000 would rule it out. VERSION: 0.9.4-bug.179-184-190 -> 0.9.4-bug.179-184-190-191 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJReturning from the viewer, and every tag add/remove/rename, called filter_proxy.invalidate() unconditionally. That re-sorts AND re-filters the entire source model. Neither operation changes row ordering unless the active sort key reads tags -- SORT_TAG_PRIORITY is the only one of the five that does; filename/date/size/resolution never consult them. Measured on a synthetic model (offscreen, 40,000 rows): invalidate() (re-sort + re-filter) 2.580s invalidateFilter() (re-filter only) 0.178s 14.5x cheaper Extrapolated to the reported 125k-row workspace that is roughly 8.9s -> 0.6s per call, which matches the reported "5-30 second freeze when opening an image and returning to the grid" far better than anything measured previously. _refresh_tag_filter_views() has ~14 call sites, so the same cost was also being paid on ordinary tagging. Changes: - ImageFilterProxy.invalidate_filter_only() for view transitions that change no data (viewer -> grid), and invalidate_after_tag_change() which re-sorts only when _sort_key == SORT_TAG_PRIORITY. Putting the condition inside the proxy keeps all ~14 call sites honest. - ImageListModel.row_data() returns the live row dict without copying. lessThan() ran twice per comparison and filterAcceptsRow() once per row, both through get_image(), which copies -- roughly four million throwaway dicts per full re-sort at 125k rows. Neither mutates what it reads. - ImageListModel.sort_tuple() caches the sort key per row, built from core.image_sort.sort_key_tuple so proxy order and the scan queue's order stay identical by construction. Invalidated per row by every mutator that can change a sort field; apply_thumbnail_batch deliberately does not invalidate, since no sort key reads thumbnail_blob. - remove_images() now emits one beginRemoveRows/endRemoveRows per contiguous run instead of one per row (20 rows in two runs: 20 signals -> 2). Each pair forces the proxy to remap, and streaming prune will push batches through here. - upsert_images() no longer evicts a row's decoded pixmap unless the incoming dict actually carries a thumbnail_blob, and no longer marks a row changed when the merge changed nothing. Discovery re-upserts a placeholder dict for every file it walks, so the old unconditional pop discarded every thumbnail the cache-hydrate worker had just delivered and then repainted them all. Honest note on the sort-key cache: on its own it is only ~1.2x on a full re-sort (2.62s vs 3.09s at 40k rows). The dominant cost of a re-sort is PySide6's per-comparison Python/C++ callback overhead, which caching cannot remove. It is kept because it makes the re-sorts that genuinely must happen cheaper and removes the ~4M dict allocations; the 14.5x above comes from not re-sorting at all when there is no ordering reason to. One existing test updated: test_pr8_viewer.py::test_viewer_restore_invalidates_ filter_once asserted its "once" contract by monkeypatching invalidate() itself, so it was coupled to the method rather than the behavior it names. It now asserts exactly one filter-only invalidation and zero full re-sorts, which is strictly stronger. Testing: 10 new tests in tests/test_image_filter_proxy_perf.py, all verified to fail against the pre-fix code -- including the two behavioral ones (placeholder upsert evicting a hydrated pixmap; 20 removal signals where 2 suffice). Full suite 832 passed, 1 skipped; ruff clean; offscreen UI-exit smoke clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJReported regression: "Preparing workspace" replaced the sync dialog and left the application non-interactive for a long stretch, with the cached thumbnails loading before it moved on. Reproduced offscreen on a 125,000-row cached workspace: open_directory() took 12.7s, entirely inside a single "Preparing workspace..." status with no progress and no event-loop pumping. Profiled: 9.87s ImageListModel.set_images 1.27s WorkingDB.get_all_images 1.03s _prune_disabled_types_from_current_db 0.60s TagManager.image_dicts_with_tags This is a consequence of fixing #191 rather than a new cost: on the reporter's Windows build the preload used to raise "too many SQL variables" and abort in milliseconds, so this work never ran at all. Now it runs. set_images() itself is only ~0.3s on a bare model. The other ~9.5s is the attached sort/filter proxy rebuilding its entire mapping in one uninterruptible operation when the model resets. Measured alternatives, all worse: set_images, proxy attached (current) 10.20s dynamicSortFilter off during populate 27.83s proxy detached during populate 18.44s (filter-only pass) 0.55s (full re-sort) 9.08s So the sort dominates, and it is not our lessThan: replacing it with a precomputed role compared natively in C++ via setSortRole only moved 9.56s -> 8.72s. The real cost is ~2 million data() calls crossing into a Python model, which is inherent to sorting this many rows through a proxy. It cannot be made much smaller, but it can be broken up. New _preload_rows_into_model() inserts PRELOAD_CHUNK_ROWS (2,000) at a time, pumping the event loop and checking the cancel checkpoint between chunks: one-shot set_images 10.28s one 10.3s freeze chunked add_images (2,000/chunk) 6.84s worst chunk 269ms Faster overall AND the freeze becomes ~270ms slices. open_directory() on the 125k reproduction now returns in 9.5s with a live count ticking every ~100ms, the progress bar moving, rows appearing progressively, and Cancel working throughout -- instead of one silent non-interactive block. This is the streaming preload deliberately deferred when the phases were planned, on the reasoning that the synchronous version satisfied requirement 4 literally. That reasoning was wrong for the freeze it caused; the report is what showed it. Testing: 3 new tests (chunked insert with no blocking set_images call, per-chunk progress reporting, cancel stopping mid-preload). Full suite 869 passed, 1 skipped; ruff clean; offscreen UI-exit smoke clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TrNqVfQQpmRfwhoUkKkkiJ