[Bug]: Thumbnail generation throughput collapses at ~125k images (5–8 img/s, idle CPU/share) vs ~30 img/s at 3.9k on the same NAS #184

New issue

Open

opened 2026-08-27 22:39:28 +00:00 by Grok · 0 comments

Grok commented

2026-08-27 22:39:28 +00:00

Member

Affected area

Directory-open Stage 3 — Generating Thumbnails (scan_metadata_ordered_and_upsert in core/image_scanner.py). Live scan, not the separate missing-thumbnail catch-up worker.

App / environment

App: 0.9.1+bug.176.178 Beta (Windows 11)
Share: remote NAS, 10 GbE, idle
System: CPU ~6% during the slow run; app RSS < 1 GB with 60+ GB free

Report

Opening a ~125,000-image workspace on the NAS, thumbnail generation runs at 5–8 images/s with occasional hangs. The NAS only shows occasional ~100 Mbps read spikes. CPU and RAM are not the limit.

Switching to a ~3,900-image directory on the same share, the same stage runs at ~30 images/s and pulls 200–300 Mbps. That rules out the NIC, NAS, and host as the bottleneck: the same decode/network path is fine until workspace size grows.

At 5 img/s, 125k remaining thumbnails is on the order of 7 hours. At the 3.9k rate it would be ~70 minutes. The gap is bookkeeping that scales with workspace size, not image I/O.

Related but not the same bug:

#176 — silent freeze between hydrate and grid populate (preload BLOBs). Already addressed on this build.
#178 — UI freeze from main-thread SQLite busy_timeout / connection churn. Item 1 (tag-write stall) and item 3 (reuse one pending-queue connection) are on this build. Item 2 (Pillow GIL) was deferred and assumed CPU-bound decode; observed CPU is 6%, so GIL is not what is starving throughput here.

Steps to reproduce

On Windows 11, run 0.9.1+bug.176.178 Beta.
Open a network workspace with ~125,000 images (local SQLite cache, share on 10 GbE).
Let the scan reach Generating Thumbnails and watch rate, CPU, RAM, and share throughput.
Open a ~3,900-image directory on the same share and compare the same stage.

Expected

Thumbnail generation is limited by decode + share read, not by how many rows are already in the working DB.
Rate on the 125k tree is in the same ballpark as the 3.9k tree on the same NAS (tens of img/s, hundreds of Mbps), not 4–6× slower with the CPU and link idle.
Occasional hangs are not part of steady-state generation.

Actual

Workspace	Observed rate	Share	CPU
~3,900 images	~30 img/s	200–300 Mbps	(healthy decode load)
~125,000 images	5–8 img/s, hangs	~100 Mbps spikes	~6%

Code notes (not a measured profile — matches the A/B)

scan_metadata_ordered_and_upsert() (DR-014 / DR-030) keeps the re-query + re-sort remaining pending rows every dispatch chunk so a mid-scan Sort By change reorders leftover work. Chunk size is thread_count (default 2).

Each outer-loop iteration does:

ctx.get_discovery_paths() — copy the entire 125k path set.
get_images_pending_enrichment_as_dicts() — full images scan with LEFT JOIN image_tags / tags, GROUP BY i.id, and SELECT … i.thumbnail_blob only to set has_thumbnail. No LIMIT. No pending-condition index. On the 176/178 branch this reuses one read connection ( #178 item 3 ) but still returns every remaining pending row.
Filter that list against discovery_paths in Python.
sort_image_dicts() over all remaining pending dicts.
Dispatch 2 enrich_image_record() jobs, emit progress per image, on_batch per chunk.
Repeat ~62,500 times.

That is O(n² log n) queue work for n pending images. At ~4k the pending query/sort is cheap and decode/network dominate (~30 img/s). At 125k the bookkeeping dominates: worker threads sit idle, CPU stays ~6%, the share barely moves, and a 125k-row query+sort looks like a hang.

Secondary costs on the same loop (not the 3.9k vs 125k smoking gun, but they add):

DB writes still reconnect per upsert_images flush (batch_size=64).
Main thread applies each 2-image batch (upsert_images + apply_thumbnail_batch) into a 125k-row model.
Network workspaces start a 5-minute full-DB share copy (_share_sync_timer); a growing 125k-thumbnail cache file can stall the UI/writer if a sync overlaps generation. User-reported share activity was reads, so this is a hang candidate, not the steady 5–8 img/s.

DR-029 already found that a 300-row pending query was only ~2.24 ms/call. That does not stay cheap at 125k, and it is paid per 2 images, not once.

Suggested fix direction

Keep DR-014 (enrich in current UI sort order; reorder when Sort By / Sort Order changes). Drop the “re-read and re-sort the entire pending set every chunk” implementation:

Load the pending queue once after discovery completes (paths + sort keys only; thumbnail_blob IS NOT NULL as a flag, no BLOB payload, no tag JOIN unless the active sort is tag-priority).
Keep an in-memory deque/heap; pop the next thread_count rows. Do not re-query SQLite to learn what is left.
Re-sort only when MetadataScanContext sort fields change (dirty flag), not every chunk.
Optionally index pending rows (width/height/thumbnail_blob IS NULL).
Profile before changing PREF_THUMBNAIL_THREADS / process-pool ( #178 item 2 ). This report is idle-CPU queue work, not decode saturation.

Acceptance criteria

On a ~125k-image network workspace, Stage 3 thumbnail rate is not 4–6× worse than a ~4k tree on the same share; CPU and share read should rise together with decode, not sit idle.
Steady generation has no multi-second hangs attributable to full-table pending queries or full-queue sorts.
Mid-scan Sort By / Sort Order still reorders remaining enrichment work (DR-014), just not by rescanning the whole table every chunk.
~4k / local-directory thumbnail rates do not regress vs current 30 img/s-class behavior.
Threading Contract unchanged: Pillow on worker threads, widgets/models on the main thread, short DB transactions.

## Affected area Directory-open **Stage 3 — Generating Thumbnails** (`scan_metadata_ordered_and_upsert` in `core/image_scanner.py`). Live scan, not the separate missing-thumbnail catch-up worker. ## App / environment - **App:** `0.9.1+bug.176.178 Beta` (Windows 11) - **Share:** remote NAS, 10 GbE, idle - **System:** CPU ~6% during the slow run; app RSS < 1 GB with 60+ GB free ## Report Opening a **~125,000-image** workspace on the NAS, thumbnail generation runs at **5–8 images/s** with **occasional hangs**. The NAS only shows **occasional ~100 Mbps read spikes**. CPU and RAM are not the limit. Switching to a **~3,900-image** directory **on the same share**, the same stage runs at **~30 images/s** and pulls **200–300 Mbps**. That rules out the NIC, NAS, and host as the bottleneck: the same decode/network path is fine until workspace size grows. At 5 img/s, 125k remaining thumbnails is on the order of **7 hours**. At the 3.9k rate it would be ~70 minutes. The gap is bookkeeping that scales with workspace size, not image I/O. Related but **not the same bug:** - **#176** — silent freeze between hydrate and grid populate (preload BLOBs). Already addressed on this build. - **#178** — UI freeze from main-thread SQLite `busy_timeout` / connection churn. Item 1 (tag-write stall) and item 3 (reuse one pending-queue connection) are on this build. Item 2 (Pillow GIL) was deferred and assumed CPU-bound decode; **observed CPU is 6%**, so GIL is not what is starving throughput here. ## Steps to reproduce 1. On Windows 11, run `0.9.1+bug.176.178 Beta`. 2. Open a network workspace with **~125,000** images (local SQLite cache, share on 10 GbE). 3. Let the scan reach **Generating Thumbnails** and watch rate, CPU, RAM, and share throughput. 4. Open a **~3,900-image** directory on the **same share** and compare the same stage. ## Expected - Thumbnail generation is limited by decode + share read, not by how many rows are already in the working DB. - Rate on the 125k tree is in the same ballpark as the 3.9k tree on the same NAS (tens of img/s, hundreds of Mbps), not 4–6× slower with the CPU and link idle. - Occasional hangs are not part of steady-state generation. ## Actual | Workspace | Observed rate | Share | CPU | |---|---|---|---| | ~3,900 images | ~30 img/s | 200–300 Mbps | (healthy decode load) | | ~125,000 images | 5–8 img/s, hangs | ~100 Mbps spikes | ~6% | ## Code notes (not a measured profile — matches the A/B) `scan_metadata_ordered_and_upsert()` (DR-014 / DR-030) keeps the **re-query + re-sort remaining pending rows every dispatch chunk** so a mid-scan Sort By change reorders leftover work. Chunk size is `thread_count` (**default 2**). Each outer-loop iteration does: 1. `ctx.get_discovery_paths()` — **copy the entire 125k path set**. 2. `get_images_pending_enrichment_as_dicts()` — full `images` scan with `LEFT JOIN image_tags` / `tags`, `GROUP BY i.id`, and `SELECT … i.thumbnail_blob` only to set `has_thumbnail`. No `LIMIT`. No pending-condition index. On the 176/178 branch this reuses one read connection ( #178 item 3 ) but still **returns every remaining pending row**. 3. Filter that list against `discovery_paths` in Python. 4. `sort_image_dicts()` over **all remaining pending dicts**. 5. Dispatch **2** `enrich_image_record()` jobs, emit progress per image, `on_batch` per chunk. 6. Repeat ~62,500 times. That is **O(n² log n)** queue work for n pending images. At ~4k the pending query/sort is cheap and decode/network dominate (~30 img/s). At 125k the bookkeeping dominates: worker threads sit idle, CPU stays ~6%, the share barely moves, and a 125k-row query+sort looks like a hang. Secondary costs on the same loop (not the 3.9k vs 125k smoking gun, but they add): - DB writes still reconnect per `upsert_images` flush (`batch_size=64`). - Main thread applies each 2-image batch (`upsert_images` + `apply_thumbnail_batch`) into a 125k-row model. - Network workspaces start a **5-minute** full-DB share copy (`_share_sync_timer`); a growing 125k-thumbnail cache file can stall the UI/writer if a sync overlaps generation. User-reported share activity was **reads**, so this is a hang candidate, not the steady 5–8 img/s. DR-029 already found that a 300-row pending query was only ~2.24 ms/call. That does not stay cheap at 125k, and it is paid **per 2 images**, not once. ## Suggested fix direction Keep DR-014 (enrich in current UI sort order; reorder when Sort By / Sort Order **changes**). Drop the “re-read and re-sort the entire pending set every chunk” implementation: - Load the pending queue **once** after discovery completes (paths + sort keys only; `thumbnail_blob IS NOT NULL` as a flag, no BLOB payload, no tag JOIN unless the active sort is tag-priority). - Keep an in-memory deque/heap; pop the next `thread_count` rows. Do **not** re-query SQLite to learn what is left. - Re-sort only when `MetadataScanContext` sort fields change (dirty flag), not every chunk. - Optionally index pending rows (`width`/`height`/`thumbnail_blob` IS NULL). - Profile before changing `PREF_THUMBNAIL_THREADS` / process-pool ( #178 item 2 ). This report is idle-CPU queue work, not decode saturation. ## Acceptance criteria - [ ] On a ~125k-image network workspace, Stage 3 thumbnail rate is not 4–6× worse than a ~4k tree on the same share; CPU and share read should rise together with decode, not sit idle. - [ ] Steady generation has no multi-second hangs attributable to full-table pending queries or full-queue sorts. - [ ] Mid-scan Sort By / Sort Order still reorders **remaining** enrichment work (DR-014), just not by rescanning the whole table every chunk. - [ ] ~4k / local-directory thumbnail rates do not regress vs current 30 img/s-class behavior. - [ ] Threading Contract unchanged: Pillow on worker threads, widgets/models on the main thread, short DB transactions.

Grok added the

Kind/Bug

Priority/High

Reviewed/Confirmed

labels

2026-08-27 22:39:47 +00:00