[Feature]: Specify Thumbnail Generatation Threads

ValleyGeek commented

2026-06-12 20:33:46 +00:00

Owner

Problem or use case

Allow the user to speed up thumbnail creation by using more threads

Proposed solution

Implement multiple threads for thumbnail generation
Add an 'advanced' setting for the user to specify a thumbnail creation thread count
default to two threads

Alternatives considered

No response

### Problem or use case Allow the user to speed up thumbnail creation by using more threads ### Proposed solution - Implement multiple threads for thumbnail generation - Add an 'advanced' setting for the user to specify a thumbnail creation thread count - default to two threads ### Alternatives considered _No response_

ValleyGeek added the

Kind/Feature

Priority

Low

labels

2026-06-12 20:33:46 +00:00

ValleyGeek changed title from ~~[Feature]: Specify Thhumbnail Generatation Threads~~ to [Feature]: Specify Thumbnail Generatation Threads

2026-06-13 06:24:31 +00:00

Claude referenced this issue from a commit

2026-06-20 12:11:15 +00:00

feat(thumbnails): configurable thumbnail generation thread count

Claude commented

2026-06-20 12:11:34 +00:00

Member

Implemented on branch 0.1.42/issue-64-thumbnail-threads (queued for PR to main, not yet opened; version bumped to 0.1.43).

generate_thumbnails_for_db() (core/thumbnail_batch.py) now decodes/resizes/encodes each batch's images across a ThreadPoolExecutor instead of strictly sequentially.
New Thumbnail generation threads advanced setting (Settings → Advanced), default 2, range 1–16.
ThumbnailWorker forwards the configured count as max_workers; both call sites in main_window.py read it via a new _thumbnail_worker_threads() helper.

Threading notes: the executor is used solely for the pure-function generate_thumbnail_jpeg() call (file read + Pillow decode/resize/encode — no DB or Qt access). All WorkingDB writes and on_progress/on_batch callbacks still happen only on the calling thread (the ThumbnailWorker's own QRunnable thread), so the "connections are short-lived, never shared across threads" contract holds — the extra helper threads never touch the DB, model, or any QWidget/QPixmap/QGraphics* object. The default for any other direct caller of generate_thumbnails_for_db() that doesn't pass max_workers is unchanged (1, sequential).

Added unit tests for multi-worker generation (same results as sequential, progress ordering, missing-file handling, mid-scan cancellation) and a settings-persistence test for the new spinbox. Full suite green (404 passed), ruff clean, UI smoke test clean.

Closing as implemented.

Implemented on branch `0.1.42/issue-64-thumbnail-threads` (queued for PR to `main`, not yet opened; version bumped to 0.1.43). - `generate_thumbnails_for_db()` (`core/thumbnail_batch.py`) now decodes/resizes/encodes each batch's images across a `ThreadPoolExecutor` instead of strictly sequentially. - New **Thumbnail generation threads** advanced setting (Settings → Advanced), default **2**, range 1–16. - `ThumbnailWorker` forwards the configured count as `max_workers`; both call sites in `main_window.py` read it via a new `_thumbnail_worker_threads()` helper. **Threading notes:** the executor is used solely for the pure-function `generate_thumbnail_jpeg()` call (file read + Pillow decode/resize/encode — no DB or Qt access). All `WorkingDB` writes and `on_progress`/`on_batch` callbacks still happen only on the calling thread (the `ThumbnailWorker`'s own `QRunnable` thread), so the "connections are short-lived, never shared across threads" contract holds — the extra helper threads never touch the DB, model, or any `QWidget`/`QPixmap`/`QGraphics*` object. The default for any other direct caller of `generate_thumbnails_for_db()` that doesn't pass `max_workers` is unchanged (1, sequential). Added unit tests for multi-worker generation (same results as sequential, progress ordering, missing-file handling, mid-scan cancellation) and a settings-persistence test for the new spinbox. Full suite green (404 passed), ruff clean, UI smoke test clean. Closing as implemented.

Claude closed this issue

2026-06-20 12:11:40 +00:00

ValleyGeek commented

2026-06-20 16:09:56 +00:00

Author

Owner

There doesn't appear to be any change in behavior when changing the thread count. CPU usage is the same at 16 threads as it is at 1. img/s sec stays the same (~22 on my test system).

ValleyGeek reopened this issue

2026-06-20 16:09:56 +00:00

Claude referenced this issue from a commit

2026-06-20 17:06:00 +00:00

fix: address re-tested feedback on #111, #86, #64; polish #89 (Copied feedback)

Claude commented

2026-06-20 17:07:07 +00:00

Member

Investigated on branch 0.1.47/issue-111-86-64-89-feedback-fixes — being upfront that I did not find a code defect, and want to share what I checked rather than guess further.

I verified, end to end, that the thread-count setting is correctly wired: the Settings spinbox persists immediately (set_pref) → _thumbnail_worker_threads() reads it back → both call sites pass it into ThumbnailWorker(max_workers=...) → generate_thumbnails_for_db() actually uses it to size a ThreadPoolExecutor. To make sure that executor genuinely parallelizes the work (rather than, say, the GIL serializing the Pillow decode/resize/encode calls), I ran a local benchmark against this exact code path with 120 synthetic 1600×1200 JPEGs on this (8-core) machine:

workers= 1  elapsed=1.93s  img/s=62.1
workers= 4  elapsed=0.92s  img/s=131.1   (~2.1x)
workers= 8  elapsed=1.33s  img/s=90.1
workers=16  elapsed=1.23s  img/s=97.2

So the mechanism does produce real throughput scaling here, not "no effect at all" — though it's worth noting it actually got worse going from 4 to 8 workers on this machine, presumably thread/GIL contention overtaking the available cores past a certain point; more threads isn't strictly better even when it's working as designed.

Given that, your "16 threads behaves identically to 1, ~22 img/s either way" result looks environment-specific rather than a bug in this code — possibilities I can't test from here: the workspace already had thumbnails cached for those images (so there was no actual generation work happening regardless of thread count), a disk I/O ceiling (especially on spinning disks, more concurrent readers doesn't help and can hurt), or antivirus/Defender real-time scanning each file open, which is a common source of exactly this symptom on Windows.

If you're up for it, a couple of things that would help narrow it down: (1) was the directory you tested on a fresh scan with no existing thumbnails, and (2) roughly how many images / what kind of storage (HDD vs SSD, local vs network share)? Leaving this open for now since I don't have a fix to point to yet, just ruled out the most likely code-level culprits.

Investigated on branch `0.1.47/issue-111-86-64-89-feedback-fixes` — being upfront that **I did not find a code defect**, and want to share what I checked rather than guess further. I verified, end to end, that the thread-count setting is correctly wired: the Settings spinbox persists immediately (`set_pref`) → `_thumbnail_worker_threads()` reads it back → both call sites pass it into `ThumbnailWorker(max_workers=...)` → `generate_thumbnails_for_db()` actually uses it to size a `ThreadPoolExecutor`. To make sure that executor genuinely parallelizes the work (rather than, say, the GIL serializing the Pillow decode/resize/encode calls), I ran a local benchmark against this exact code path with 120 synthetic 1600×1200 JPEGs on this (8-core) machine: ``` workers= 1 elapsed=1.93s img/s=62.1 workers= 4 elapsed=0.92s img/s=131.1 (~2.1x) workers= 8 elapsed=1.33s img/s=90.1 workers=16 elapsed=1.23s img/s=97.2 ``` So the mechanism does produce real throughput scaling here, not "no effect at all" — though it's worth noting it actually got *worse* going from 4 to 8 workers on this machine, presumably thread/GIL contention overtaking the available cores past a certain point; more threads isn't strictly better even when it's working as designed. Given that, your "16 threads behaves identically to 1, ~22 img/s either way" result looks environment-specific rather than a bug in this code — possibilities I can't test from here: the workspace already had thumbnails cached for those images (so there was no actual generation work happening regardless of thread count), a disk I/O ceiling (especially on spinning disks, more concurrent readers doesn't help and can hurt), or antivirus/Defender real-time scanning each file open, which is a common source of exactly this symptom on Windows. If you're up for it, a couple of things that would help narrow it down: (1) was the directory you tested on a fresh scan with no existing thumbnails, and (2) roughly how many images / what kind of storage (HDD vs SSD, local vs network share)? Leaving this open for now since I don't have a fix to point to yet, just ruled out the most likely code-level culprits.

ValleyGeek commented

2026-06-20 22:11:09 +00:00

Author

Owner

Add a 'Benchmark and Set Optimal' button to the right of the setting for the thread count that resets the cache for the first 200 images in the active directory then runs performs a generation run for those 200 images at each thread level, up to the systems core count or 16 (whichever comes first). Once complete, it provides a report dialog with the results that user can copy and sets the thread count to the optimal img/s value.

My previous test was performed by closing and deleting cache using a few different thread counts on a local directory with 839 desktop background images ranging from 1024x768 to 4k, spread across 14 sub-directories. The system is an AMD Ryzen 9 3950x 16 core CPU with 128gb ram and a kingston SFYRD2000G SSD. The system it idling at 3% CPU and 20% memory usage with the rest available to the application.

Add a 'Benchmark and Set Optimal' button to the right of the setting for the thread count that resets the cache for the first 200 images in the active directory then runs performs a generation run for those 200 images at each thread level, up to the systems core count or 16 (whichever comes first). Once complete, it provides a report dialog with the results that user can copy and sets the thread count to the optimal img/s value. My previous test was performed by closing and deleting cache using a few different thread counts on a local directory with 839 desktop background images ranging from 1024x768 to 4k, spread across 14 sub-directories. The system is an AMD Ryzen 9 3950x 16 core CPU with 128gb ram and a kingston SFYRD2000G SSD. The system it idling at 3% CPU and 20% memory usage with the rest available to the application.

Claude referenced this issue from a commit

2026-06-21 05:48:29 +00:00

feat(settings): Benchmark and Set Optimal button for thumbnail threads (#64)

Claude commented

2026-06-21 05:48:51 +00:00

Member

Implemented exactly as specified, on the same branch (0.1.47/issue-111-86-64-89-feedback-fixes, now at 0.1.50).

Added a Benchmark and Set Optimal button to the right of the thumbnail generation thread-count setting (Settings → Advanced). On click (requires an open directory):

Samples up to 200 images from the open directory.
Clears their cached thumbnail before each level, then times a full generation pass at every thread level from 1 up to this system's core count, capped at 16.
Shows progress as each level completes (with a Cancel option).
Sets the thread-count setting to whichever level had the highest images/sec, and updates the spinbox immediately.
Shows a report dialog listing every level's throughput, with Copy to Clipboard (same "Copied" feedback as Export Tags from #89) so you can paste the results back here.

Given your system specs (Ryzen 9 3950X, 16 cores, fast NVMe SSD, system otherwise idle) — that's a very different machine profile than the dev sandbox I benchmarked from earlier, with much more parallel headroom and no I/O bottleneck to speak of. I'd be curious what the benchmark reports on your actual machine; if threading genuinely has no effect there either, the report's per-level numbers would tell us a lot more than the single 1-vs-16 comparison did. Whenever you get a chance to run it, feel free to paste the copied report here.

Implemented exactly as specified, on the same branch (`0.1.47/issue-111-86-64-89-feedback-fixes`, now at 0.1.50). Added a **Benchmark and Set Optimal** button to the right of the thumbnail generation thread-count setting (Settings → Advanced). On click (requires an open directory): - Samples up to 200 images from the open directory. - Clears their cached thumbnail before each level, then times a full generation pass at every thread level from 1 up to this system's core count, capped at 16. - Shows progress as each level completes (with a Cancel option). - Sets the thread-count setting to whichever level had the highest images/sec, and updates the spinbox immediately. - Shows a report dialog listing every level's throughput, with **Copy to Clipboard** (same "Copied" feedback as Export Tags from #89) so you can paste the results back here. Given your system specs (Ryzen 9 3950X, 16 cores, fast NVMe SSD, system otherwise idle) — that's a very different machine profile than the dev sandbox I benchmarked from earlier, with much more parallel headroom and no I/O bottleneck to speak of. I'd be curious what the benchmark reports on your actual machine; if threading genuinely has no effect there either, the report's per-level numbers would tell us a lot more than the single 1-vs-16 comparison did. Whenever you get a chance to run it, feel free to paste the copied report here.

Grok commented

2026-06-21 07:24:39 +00:00

Member

I'm on a different computer and ran the test from it. There's a discrepancy between the benchmark and the actual live generator. During a fresh scan of the directory I used for benchmark the maximum real speed was 14 images per second.

The benchmark returned very different results.

Threads | Images/sec | Seconds | Images
------- | ---------- | ------- | ------
      1 |       33.7 |    5.93 |    200
      2 |       59.1 |    3.39 |    200
      3 |       73.8 |    2.71 |    200
      4 |       83.3 |    2.40 |    200
      5 |       84.4 |    2.37 |    200
      6 |       85.6 |    2.34 |    200
      7 |       88.1 |    2.27 |    200
      8 |       88.9 |    2.25 |    200 *

* Optimal: 8 thread(s)

A) The benchmark should properly simulate the scan operation
B) The bottleneck during the scan needs investigating. If the single DB write thread is the issue with no areas for efficiency gain, could the thumbnails be cached in memory quickly for the user experience then written to the DB on disk in the background?

I'm on a different computer and ran the test from it. There's a discrepancy between the benchmark and the actual live generator. During a fresh scan of the directory I used for benchmark the maximum real speed was 14 images per second. The benchmark returned very different results. ``` Threads | Images/sec | Seconds | Images ------- | ---------- | ------- | ------ 1 | 33.7 | 5.93 | 200 2 | 59.1 | 3.39 | 200 3 | 73.8 | 2.71 | 200 4 | 83.3 | 2.40 | 200 5 | 84.4 | 2.37 | 200 6 | 85.6 | 2.34 | 200 7 | 88.1 | 2.27 | 200 8 | 88.9 | 2.25 | 200 * * Optimal: 8 thread(s) ``` A) The benchmark should properly simulate the scan operation B) The bottleneck during the scan needs investigating. If the single DB write thread is the issue with no areas for efficiency gain, could the thumbnails be cached in memory quickly for the user experience then written to the DB on disk in the background?

Claude referenced this issue from a commit

2026-06-21 07:39:41 +00:00

fix(scan): batch DB writes in the live ordered metadata scan (#64)

Claude commented

2026-06-21 07:39:56 +00:00

Member

Thank you, that benchmark report cracked the discrepancy open. You're right on point (A): the benchmark doesn't simulate the real scan, because they're literally different code paths.

Root cause: the benchmark (and the separate "missing thumbnails" catch-up worker) call generate_thumbnails_for_db(), which does use the configurable thread-count setting. A real fresh directory scan calls a completely different function (scan_metadata_ordered_and_upsert()), a single-image-at-a-time loop that never used the thread pool at all — the thread-count setting has had zero effect on real scan speed this whole time, on any machine, regardless of core count. That's the actual reason your "16 threads = 1 thread" observation from the start of this issue held up.

I profiled that loop directly rather than guessing further. The per-iteration "what's still pending" query costs ~2.2ms — cheap. The per-image database write, which the loop was committing after every single image, costs ~20.5ms — and a single batched write of the same rows costs ~0.2ms/row, about 100x cheaper per row than one-at-a-time. That's your point (B), confirmed empirically: the database write was the real bottleneck, not threading.

Fix pushed (same branch, now 0.1.52): the live scan now buffers enriched images and writes them in batches of 64 instead of one at a time, while still respecting cancellation and not losing a partial trailing batch. It deliberately does not change the one-image-at-a-time, always-current-sort-order behavior that lets you re-sort mid-scan (issue #17) — only how often it hits the database. End to end on 300 real 1600×1200 JPEGs in my sandbox: ~26 img/s before → ~46.5 img/s after, roughly 1.8x.

To be upfront about scope: this does not make the thread-count setting affect the live scan — it's still single-threaded by design, so on your 16-core machine you likely still won't see scan speed scale with the slider. Properly parallelizing the live scan's per-image work while preserving live mid-scan re-sorting is a larger, separate change I haven't attempted yet. If the batching win alone isn't enough once you retest, let me know and I'll take that on next — wanted to get this verified, lower-risk piece in front of you first rather than bundle it with something bigger and unverified.

Recorded as DR-029 in docs/DECISIONS.md. Whenever you get a chance, a real-scan retest (ideally the same 839-image, 14-sub-directory set) would help confirm whether the improvement holds up at that scale on your hardware.

Thank you, that benchmark report cracked the discrepancy open. You're right on point (A): the benchmark doesn't simulate the real scan, because they're literally different code paths. **Root cause:** the benchmark (and the separate "missing thumbnails" catch-up worker) call `generate_thumbnails_for_db()`, which does use the configurable thread-count setting. A real fresh directory scan calls a completely different function (`scan_metadata_ordered_and_upsert()`), a single-image-at-a-time loop that **never used the thread pool at all** — the thread-count setting has had zero effect on real scan speed this whole time, on any machine, regardless of core count. That's the actual reason your "16 threads = 1 thread" observation from the start of this issue held up. I profiled that loop directly rather than guessing further. The per-iteration "what's still pending" query costs ~2.2ms — cheap. The per-image database write, which the loop was committing after every single image, costs ~20.5ms — and a single batched write of the same rows costs ~0.2ms/row, **about 100x cheaper per row than one-at-a-time**. That's your point (B), confirmed empirically: the database write was the real bottleneck, not threading. **Fix pushed** (same branch, now 0.1.52): the live scan now buffers enriched images and writes them in batches of 64 instead of one at a time, while still respecting cancellation and not losing a partial trailing batch. It deliberately does *not* change the one-image-at-a-time, always-current-sort-order behavior that lets you re-sort mid-scan (issue #17) — only how often it hits the database. End to end on 300 real 1600×1200 JPEGs in my sandbox: ~26 img/s before → ~46.5 img/s after, roughly 1.8x. To be upfront about scope: **this does not make the thread-count setting affect the live scan** — it's still single-threaded by design, so on your 16-core machine you likely still won't see scan speed scale with the slider. Properly parallelizing the live scan's per-image work while preserving live mid-scan re-sorting is a larger, separate change I haven't attempted yet. If the batching win alone isn't enough once you retest, let me know and I'll take that on next — wanted to get this verified, lower-risk piece in front of you first rather than bundle it with something bigger and unverified. Recorded as DR-029 in `docs/DECISIONS.md`. Whenever you get a chance, a real-scan retest (ideally the same 839-image, 14-sub-directory set) would help confirm whether the improvement holds up at that scale on your hardware.

Claude referenced this issue from a commit

2026-06-21 16:39:03 +00:00

feat(scan): parallelize live-scan thumbnail generation; align benchmark (#64)

Claude commented

2026-06-21 16:39:23 +00:00

Member

Follow-up to the 0.1.52 write-batching fix, now pushed to 0.1.47/issue-111-86-64-89-feedback-fixes (0.1.53).

You clarified the original intent precisely: the live directory scan that populates the grid with thumbnails as they're generated is the specific target of the thread-count setting — not just the separate "missing thumbnails" catch-up pass. That part was still single-threaded after 0.1.52; the setting had no effect on it at all.

This round:

scan_metadata_ordered_and_upsert() now dispatches up to thread_count images per step to a thread pool (instead of one at a time), with thumbnails appearing in the grid as each image finishes — not batched behind the DB write.
The live scan now actually reads the "Thumbnail generation threads" setting (it didn't before).
The Benchmark and Set Optimal button was also realigned: it previously timed a different code path (the catch-up pass's function) that doesn't represent a fresh scan's actual work. It now times the exact same operation the live scan performs, so its suggested optimal thread count is measured against what the setting actually controls.

Verified end to end (300 real 1600×1200 JPEGs, this sandbox, 8 reported CPUs): 1/2/4/8/16 threads → 60.1 / 87.3 / 135.0 / 118.6 / 93.2 img/s — peak at 4 threads here, but the optimal point is machine-dependent, which is exactly why the Benchmark button exists rather than a hardcoded default.

Design rationale and the sort-order-reactivity trade-off (a mid-scan sort change now reorders the remaining queue at chunk granularity, ≤16 images, instead of after every single image) are recorded in docs/DECISIONS.md as DR-030. Full test suite green (503 passed, 1 skipped), lint clean, offscreen UI-exit smoke clean.

Please re-test when convenient — particularly whether thumbnails now visibly populate progressively during a fresh scan, and whether the Benchmark's suggested thread count now tracks real scan speed on your machine.

Follow-up to the 0.1.52 write-batching fix, now pushed to `0.1.47/issue-111-86-64-89-feedback-fixes` (0.1.53). You clarified the original intent precisely: the live directory scan that populates the grid with thumbnails as they're generated is the specific target of the thread-count setting — not just the separate "missing thumbnails" catch-up pass. That part was still single-threaded after 0.1.52; the setting had no effect on it at all. This round: - `scan_metadata_ordered_and_upsert()` now dispatches up to `thread_count` images per step to a thread pool (instead of one at a time), with thumbnails appearing in the grid as each image finishes — not batched behind the DB write. - The live scan now actually reads the "Thumbnail generation threads" setting (it didn't before). - The **Benchmark and Set Optimal** button was also realigned: it previously timed a different code path (the catch-up pass's function) that doesn't represent a fresh scan's actual work. It now times the exact same operation the live scan performs, so its suggested optimal thread count is measured against what the setting actually controls. Verified end to end (300 real 1600×1200 JPEGs, this sandbox, 8 reported CPUs): 1/2/4/8/16 threads → 60.1 / 87.3 / 135.0 / 118.6 / 93.2 img/s — peak at 4 threads here, but the optimal point is machine-dependent, which is exactly why the Benchmark button exists rather than a hardcoded default. Design rationale and the sort-order-reactivity trade-off (a mid-scan sort change now reorders the remaining queue at chunk granularity, ≤16 images, instead of after every single image) are recorded in `docs/DECISIONS.md` as DR-030. Full test suite green (503 passed, 1 skipped), lint clean, offscreen UI-exit smoke clean. Please re-test when convenient — particularly whether thumbnails now visibly populate progressively during a fresh scan, and whether the Benchmark's suggested thread count now tracks real scan speed on your machine.

ValleyGeek commented

2026-06-21 17:10:23 +00:00

Author

Owner

Feature working as intended

Claude commented

2026-06-21 17:12:55 +00:00

Member

Confirmed resolved — closing. Summary across all rounds: per-thread thumbnail pool (catch-up pass + benchmark), then DR-029's write-batching fix for the live scan, then this round's chunked-dispatch parallelization of the live scan itself plus realigning the Benchmark to measure that same operation. Thanks for the detailed real-machine testing throughout — it's what surfaced the benchmark/live-scan code-path mismatch in the first place.

Claude closed this issue

2026-06-21 17:13:00 +00:00

Claude referenced this issue

2026-06-22 00:34:53 +00:00

feat(thumbnails): configurable thumbnail generation thread count (#64) #124

Claude referenced this issue from a commit

2026-06-22 00:38:18 +00:00

docs: PR-prep for the 0.1.x feature-tranche close-out - bump to 0.2.0

Claude referenced this issue

2026-06-22 00:38:38 +00:00

Feedback-fix rounds for #111, #86, #64, #89, #109, #57, #58, #21 — bump to 0.2.0 #129

Claude referenced this issue from a commit