[Bug]: 'Scan Duplicate Files' Crashes App Nearly Immediately #111
Labels
No labels
Kind/Bug
Kind/Feature
Priority/High
Priority/Medium
Reviewed/Confirmed
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
ai-collab/bulk-image-organizer#111
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
App version
0.1.36 Beta
Platform
Windows
Steps to reproduce
Expected behavior
Duplicate file scan runs
Actual behavior
Program crashes and closes before scan starts displaying a file count for progress
Additional context
No present in v0.1.35
[Bug]: Scan for Duplicate Files Crashesto [Bug]: 'Scan Duplicate Files' Crashes App Nearly ImmediatelyFixed on branch
0.1.38/issue-111-dupe-scan-crash(queued for PR tomain, not yet opened).Root cause:
_start_dupe_scan()and its progress/finished/error signal handlers inmain_window.pyhad no exception handling of their own. Any exception raised while starting the scan, or while the main thread handled a queued signal from the backgroundDupeWorker, propagated straight out of a Qt slot instead of being caught. On the Windows build (--windows-console-mode=disable, no console) there's nowhere visible for that traceback to land, so the app simply appears to crash and close.Fix:
_fail_dupe_scan()helper that resets scan state and shows a "Duplicate Scan Error" dialog instead of letting the exception escape.sys.frozen, which Nuitka doesn't always set, so the real packaged app fell through to the dev logging path and ended up with no usable log output at all (a stdout handler bound toNoneon Windows, and no file handler). Packaged builds now get a durable log file under the user config directory, so a future crash like this one leaves a trace to diagnose from.Added regression tests (
tests/test_dupe_scan_crash_guard.py) that inject failures into the start path, the progress handler, and the finished handler, and assert the app surfaces an error dialog instead of raising.Closing — direction was clear from the report and the fix is implemented and tested. Reopen if it still reproduces once you've had a chance to try a build from this branch.
Application still crashes when attempting scan, no perceivable change in behavior. I was unable to locate the log file in the user directory.
Thanks for testing the previous fix and reporting back — sorry that one didn't land. Continued on branch
0.1.47/issue-111-86-64-89-feedback-fixes(chained after the dev work in progress).I want to be upfront: I could not reproduce this crash from this (Linux, source-run) development environment, and I don't have access to a Windows machine to test the packaged build directly. I diffed the actual application code between the
v0.1.35andv0.1.36tags and found nothing touching the dupe-scan path (_start_dupe_scan,DupeWorker, hashing) — so I don't think this was a code regression introduced between those two versions; it looks like a pre-existing issue that happened to surface around when you upgraded. Given the prior fix's try/except wrapping apparently never triggered (no dialog appeared, matching your report of "no perceivable change"), the most likely explanation is a native crash (e.g. inside a C extension during image decoding) that bypasses Python's exception handling entirely — that kind of crash can't be caught by anytry/exceptin the Python code, including the one added last time.What this round actually does, since I can't fix something I can't observe:
%APPDATA%\bulk-image-organizer\logs\on Windows) so it's no longer something you have to hunt for manually.try/exceptaround the per-image hashing call inside the scan loop, in case some unusual file triggers an exception type outside the set_compute_hashes()already catches — this degrades to "skip this one file" instead of risking the whole scan, on the chance the cause is a Python-level exception after all.If you're willing to retest: please try the scan again from this branch and, whether it crashes or not, check Help → Open Log Folder for
bulk-image-organizer.logand share its last ~20 lines (especially anything after "Dupe scan requested"). That would tell us conclusively whether this is a catchable Python error we can now fix directly, or a native crash that needs a different approach (e.g. isolating the hashing step in a subprocess so a hard crash there can't take the whole app down). Re-opening was the right call — leaving this open until we have that confirmation either way.Here's the full log from launching the application, opening a directory, and initiating a scan then crash:
Here's the content of the directory that was open
Running version
0.1.36 Betaand repeating generated this log before crashing:Version 0.1.35 Beta completed the scan and displayed results
Thank you for the detailed logs and directory listing — that was exactly what I needed, and I think I found it.
The key clue: in your 0.1.48 log, the very last line is
DupeWorker.run() starting, immediately followed by total silence — no progress, no error, nothing. That confirms a true native crash (not a Python exception — my checkpoint logging and the try/except guard from the last round would have caught and logged anything Python-level). In your 0.1.36 log, the crash also happens right afterImporting JpegImagePlugin, the first time Pillow loads a JPEG decoder that run — same place, just earlier in that version's lifecycle since fewer thumbnails were cached.Both of your test runs were an exact scan (
kind=exactin the 0.1.48 log — "Scan Duplicate Files" defaults to this). Here's the bug:_compute_hashes()always computed two things for every image, regardless of which scan kind needed them — the sha256 exact hash (pure standard library, no risk) and a perceptual hash via theimagehashlibrary, even on an exact-only scan that never uses it.imagehashpulls innumpy,scipy, andPyWavelets— a much heavier native-dependency chain (BLAS/LAPACK/FFT routines, often with their own vendored DLLs) than plainhashlib. That stack is a well-documented source of exactly this symptom — silent native crashes with no Python traceback — in frozen/Nuitka-style Windows builds, usually from a missing vendored DLL or an unsupported CPU instruction path being selected on first use.Fix (pushed to the same branch,
0.1.47/issue-111-86-64-89-feedback-fixes, now at 0.1.49): split the hashing so an exact scan never touchesimagehash/numpy/scipyat all — only the plain sha256 path runs. A perceptual ("visual") scan still needs the perceptual hash, so if that dependency chain really is the cause, a visual scan might still crash; exact scans should not. Also added per-image checkpoint logging (Hashing 1/57: <file>, etc.) so if anything still goes wrong, the log will show exactly which file it happened on instead of just the start of the loop.Could you retest from this branch? Specifically:
Recorded the full reasoning as DR-027 in
docs/DECISIONS.mdfor anyone following along. Leaving this open until you've had a chance to retest.Scan duplicate files has been fixed. Now we need to track down what changed in the visual duplicate scanner. This did previously work as desired so something has regressed the functionality.
That log confirmed it precisely — thank you for testing both scan kinds. Pushed the fix to the same branch (
0.1.47/issue-111-86-64-89-feedback-fixes, now at 0.1.51).The last line of your visual-scan log,
Hashing 1/19: ... (need_visual=True)followed by total silence, pinpointed the crash to the very first perceptual-hash computation — confirming DR-027's open suspicion. I readimagehash's own source:phash()doesimport scipy.fftpacklazily, inside its own function body, not when the module loads. That's exactly why everything else in the app (including the parts ofimagehashused elsewhere for comparing hashes, which are numpy-only) works fine, while the first call tophash()specifically is what crashes — that's the first and only momentscipy.fftpackactually gets imported and exercised.Fix: added
core/perceptual_hash.py::phash()— a hand-rolled 2D DCT (the same mathscipy.fftpack.dctdoes) using onlynumpy, which is already used safely elsewhere in this app. It never importsscipyanywhere. I verified it produces bit-for-bit identical hashes to the realimagehash.phash()across 10 random images, 5 different sizes/color modes, and an alpha-channel case — so this isn't a "close enough" approximation, and no existing cached hashes in your workspace are invalidated by the swap. Also confirmed it works correctly withscipyforced unimportable.Recorded as DR-028 in
docs/DECISIONS.md. Could you retest the visual scan once more from this branch? At this point both exact and visual scans should be clear of the dependency that was crashing them — but I'd like your confirmation on the real Windows build before calling this fully resolved.(Also, noticed your directory got reorganized into subfolders with renamed files between test runs — not an issue, just confirms the new files correctly triggered a fresh perceptual-hash computation rather than reusing a stale cached value, which is what let this surface again.)
Closing per confirmation that this is resolved. Recap of the full root cause for the record: the crash was always a native (non-Python-catchable) crash inside
imagehash.phash()'s lazyimport scipy.fftpack, which only fires the first time a perceptual hash is computed in a given run. It surfaced two ways depending on scan kind:imagehash.phash()unconditionally even though they never needed it — fixed in 0.1.49 by only computing the hash type each scan kind actually needs (DR-027).imagehash.phash()'s computation with a hand-rolled, scipy-free 2D DCT reimplementation incore/perceptual_hash.py, verified bit-identical to the original (DR-028) — so no existing cached hashes were invalidated by the swap.Thank you for the very precise crash logs across multiple versions and scan kinds — that's what made it possible to localize this without direct access to a Windows machine. Reopen if it resurfaces.