Verify firmware checksums and tolerate NVIDIA's bad metadata

Doridian commented

2026-09-02 21:01:28 -07:00

Owner

Capture the md5sum/sha fields the downloader widget's helper.php publishes
for every file and carry them through discovery -> merge -> Record as
expected_md5 / expected_sha256. On download, both md5 and sha256 of the
bytes are computed and recorded (Record.md5 is new alongside Record.sha256).

--verify is reworked: instead of blindly re-downloading, it refreshes the
expected hashes from upstream (keeping any it previously saw), then re-hashes
every mirrored file against the digest observed when it was mirrored -- which
reliably catches on-disk corruption -- and re-fetches only the ones that no
longer match. mirror_card / run_once return the count of files that are
damaged on disk and could not be re-fetched; amain exits non-zero on that.
fetch_one now returns an Outcome enum rather than a bool.

Two upstream data bugs surfaced and are worked around:

helper.php returns some URLs with a trailing newline. urlparse silently
strips \r\n\t so looks_like_firmware accepted them, then httpx raised
httpx.InvalidURL (not an HTTPError) which escaped fetch_one as a crashed
task. Widget URLs are now stripped at the source, looks_like_firmware
rejects any URL containing whitespace, the href regex excludes whitespace,
and fetch_one catches httpx.InvalidURL.
helper.php publishes md5sum/sha values that do not match the files it
serves for a minority of builds (they look shifted by one within a version
group; confirmed by downloading the files). A download whose bytes disagree
with the upstream-published checksum is therefore kept, not discarded --
logged and counted via upstream_checksum_disagrees() and a per-card summary
line. Only a mismatch against our own recorded digest is treated as
corruption.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

Capture the md5sum/sha fields the downloader widget's helper.php publishes for every file and carry them through discovery -> merge -> Record as expected_md5 / expected_sha256. On download, both md5 and sha256 of the bytes are computed and recorded (Record.md5 is new alongside Record.sha256). --verify is reworked: instead of blindly re-downloading, it refreshes the expected hashes from upstream (keeping any it previously saw), then re-hashes every mirrored file against the digest observed when it was mirrored -- which reliably catches on-disk corruption -- and re-fetches only the ones that no longer match. mirror_card / run_once return the count of files that are damaged on disk and could not be re-fetched; amain exits non-zero on that. fetch_one now returns an Outcome enum rather than a bool. Two upstream data bugs surfaced and are worked around: - helper.php returns some URLs with a trailing newline. urlparse silently strips \r\n\t so looks_like_firmware accepted them, then httpx raised httpx.InvalidURL (not an HTTPError) which escaped fetch_one as a crashed task. Widget URLs are now stripped at the source, looks_like_firmware rejects any URL containing whitespace, the href regex excludes whitespace, and fetch_one catches httpx.InvalidURL. - helper.php publishes md5sum/sha values that do not match the files it serves for a minority of builds (they look shifted by one within a version group; confirmed by downloading the files). A download whose bytes disagree with the upstream-published checksum is therefore kept, not discarded -- logged and counted via upstream_checksum_disagrees() and a per-card summary line. Only a mismatch against our own recorded digest is treated as corruption. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Doridian added 1 commit

2026-09-02 21:01:28 -07:00

Verify firmware checksums and tolerate NVIDIA's bad metadata

Nix check / check-nix (pull_request) Successful in 1m18s

Details

bbb80686ed

Capture the md5sum/sha fields the downloader widget's helper.php publishes
for every file and carry them through discovery -> merge -> Record as
expected_md5 / expected_sha256. On download, both md5 and sha256 of the
bytes are computed and recorded (Record.md5 is new alongside Record.sha256).

--verify is reworked: instead of blindly re-downloading, it refreshes the
expected hashes from upstream (keeping any it previously saw), then re-hashes
every mirrored file against the digest observed when it was mirrored -- which
reliably catches on-disk corruption -- and re-fetches only the ones that no
longer match. mirror_card / run_once return the count of files that are
damaged on disk and could not be re-fetched; amain exits non-zero on that.
fetch_one now returns an Outcome enum rather than a bool.

Two upstream data bugs surfaced and are worked around:

- helper.php returns some URLs with a trailing newline. urlparse silently
  strips \r\n\t so looks_like_firmware accepted them, then httpx raised
  httpx.InvalidURL (not an HTTPError) which escaped fetch_one as a crashed
  task. Widget URLs are now stripped at the source, looks_like_firmware
  rejects any URL containing whitespace, the href regex excludes whitespace,
  and fetch_one catches httpx.InvalidURL.

- helper.php publishes md5sum/sha values that do not match the files it
  serves for a minority of builds (they look shifted by one within a version
  group; confirmed by downloading the files). A download whose bytes disagree
  with the upstream-published checksum is therefore kept, not discarded --
  logged and counted via upstream_checksum_disagrees() and a per-card summary
  line. Only a mismatch against our own recorded digest is treated as
  corruption.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Doridian merged commit 4b8127f629 into main

2026-09-02 21:01:31 -07:00

Doridian deleted branch checksum-verification

2026-09-02 21:01:31 -07:00

Doridian referenced this pull request from a commit

2026-09-02 21:01:31 -07:00

Verify firmware checksums and tolerate NVIDIA's bad metadata (#2)

Rows
Columns

Verify firmware checksums and tolerate NVIDIA's bad metadata #2