Repository navigation
humility 0.13.0.699 failed to write cosmo-os QSPI on alfred #705
Description
I was trying to write a new host phase1 to flash on cosmo-os from alfred, and three attempts in a row failed with
variants of this.
alfred(cosmo-os)% h qspi -D /staff/andy/os/cosmo.rom && cycle
humility: attached to 1fc9:0143:WTECYVHYHSQD4 via CMSIS-DAP V1
humility: Using existing slot settings
2026-06-23T16:47:17.459592Z ERROR probe_rs::probe::cmsisdap::commands::transfer: Transfer count larger than requested number of transfers
2026-06-23T16:47:17.459834Z WARN probe_rs::session: Could not clear all hardware breakpoints: An ARM specific error occurred.
Caused by:
0: The debug probe encountered an error.
1: An error which is specific to the debug probe in use occurred.
2: Error handling CMSIS-DAP command Transfer.
3: Unexpected answer to command.
2026-06-23T16:47:17.461481Z WARN probe_rs::probe::cmsisdap: CMSIS_DAP: Only 1/2 transfers were executed, but no error was reported.
2026-06-23T16:47:17.461579Z WARN probe_rs::session: Failed to deconfigure device during shutdown: Arm(Other("Possible error in CMSIS-DAP probe: Only 1/2 transfers were executed, but no error was reported."))
2026-06-23T16:47:17.461974Z WARN probe_rs::probe::cmsisdap: CMSIS_DAP: Only 2/3 transfers were executed, but no error was reported.
humility qspi failed: An ARM specific error occurred.
Caused by:
0: The debug probe encountered an error.
1: An error which is specific to the debug probe in use occurred.
2: Error handling CMSIS-DAP command Transfer.
3: Timeout in USB communication.
alfred(cosmo-os)% h qspi -D /staff/andy/os/cosmo.rom && cycle
humility: attached to 1fc9:0143:WTECYVHYHSQD4 via CMSIS-DAP V1
humility: Using existing slot settings
humility: skipping sector 0
humility: erased 9.88 MiB in 12 seconds
2026-06-23T16:59:29.475567Z WARN probe_rs::session: Could not clear all hardware breakpoints: An ARM specific error occurred.
Caused by:
0: The debug probe encountered an error.
1: An error which is specific to the debug probe in use occurred.
2: Error handling CMSIS-DAP command Transfer.
3: Timeout in USB communication.
2026-06-23T16:59:29.526139Z WARN probe_rs::session: Failed to deconfigure device during shutdown: Arm(Probe(ProbeSpecific(BoxedProbeError(Send { command_id: Transfer, source: Timeout }))))
humility qspi failed: An ARM specific error occurred.
Caused by:
0: Error using access port FullyQualifiedApAddress { dp: Default, ap: V1(0) }.
1: Failed to read register DRW at address 0xd0c
2: The debug probe encountered an error.
3: An error which is specific to the debug probe in use occurred.
4: Error handling CMSIS-DAP command Transfer.
5: Timeout in USB communication.
alfred(cosmo-os)%
I then grabbed the humility binary from igor which just reports as 0.13.0 and that succeeded first time:
alfred(cosmo-os)% pfexec ./humility qspi -D /staff/andy/os/cosmo.rom
humility: attached to 1fc9:0143:WTECYVHYHSQD4 via CMSIS-DAP V1
humility: Using existing slot settings
humility: skipping sector 0
humility: erased 3.19 MiB in 3 seconds
humility: hashed 32.00 MiB, wrote 3.19 MiB in 6 minutes
Activity
labbott commentedon Jun 23, 2026
Looks like non-determinism. The upstream
nusbmaintainer found at least one other bug while testing some things on illumos so there may be some other issue lurking.labbott commentedon Jun 23, 2026
actually this is even weirder: we're using
CMSIS-DAP V1which still uses the old HID/libusb based back end. So this is still a bug but unfortunately one related to the libusb backend which is more annoying.citrus-it commentedon Jun 23, 2026
The "succeeded first time" could be because the failed attempt managed to write the first 6MB. Note that the successful one only needed to write 3.19MiB.
Having said that, a flash just now succeeded (although it felt very slow).
labbott commentedon Jun 23, 2026
I've also seen zero errors flashing from scratch on
cosmo-hubrisso far. This may be one to keep an eye on.labbott commentedon Jun 24, 2026
I reproduced this exactly once on
cosmo-hubris-spThose in the office noted that the fans when on full blast. I think this means the CPU was still halted at the time the error happened.
labbott commentedon Jun 26, 2026
This was a different error seen but I'm still classifying it as suspected weirdness
Edit: not related this was using
zfs.imginstead ofcosmo.romorgimlet.romlabbott commentedon Jul 1, 2026
I experimented with bumping the USB timeout to 10 seconds and still saw the issue. I think this means there's probably something deeper going on with USB. I will probably need to do some dtrace if I can figure out what I want to trace.
labbott commentedon Jul 2, 2026
This is an interesting case, we suddenly started seeing things like
Expected 254 responses, got 255 responses with data..which is that we're reading a bunch of extra zeros from USB. I think we do need more specific illumos USB probing here.Edit: upon further inspection those seem to happen in other parts of the output with no problem, so this still seems to be a classic case of USB timeout
labbott commentedon Jul 10, 2026
I ran an experiment where instead of using
read_timeoutwe just use blockingreadand it ran successfully most of the day and overnight whereas I usually get a fan party at least once during the work day. It's a little strange that bumping the timeout to an absurdly long time didn't seem to run successfully.In the interest of making forward progress and keeping others from running into this I may add some retries.
luqmana commentedon Jul 11, 2026
Looks like going from probe-rs 0.12 to 0.31 brought this change oxidecomputer/probe-rs@40103e9. The probe will sit in its retry loop a lot longer so we hit the USB timeout. But if the probe eventually returns a response we'd end up de-synced. Hence the "Only 1/2 transfers were executed" (stale response to a smaller batch) & "Transfer count larger than requested" (stale response to a larger one) errors. Unfortunately, seems like probe-rs only calls
CmsisDapDevice::drainon open.labbott commentedon Jul 13, 2026
Good find! I experimented with dropping the retries back down to 80 and still saw the USB timeout. I also would have expected the problem to go away with a large enough USB timeout.
I experimented with adding some retries around the USB read and it ran in a loop over the weekend. It seemed to go okay and never needed more than ~2 retries. Unless I can get significantly more insight as to what's going on I'm probably going to merge it as a workaround. These probes are running an old version of NXP probe firmware and are the older CMSIS DAP v1 which we'd like to deprecate anyway so I'm hesitant to spend too much time on it.
labbott commentedon Jul 14, 2026
oxidecomputer/probe-rs#58 planning to merge this then bumping humility (github CI is being strange for building here)