Repository navigation
[FEATURE] Add diagnostic dump for reset DSP failure with status = 0xffffffff #3267
Description
Is your feature request related to a problem? Please describe.
The following error signature appeared in recent fields reports for hatch/octopus/dedede:
(10 reports in last 3 months)
Board: hatch
Model: Dragonair
Image version: R85-13261.0.0
There are error messages:
[ 8.356962] sof-audio-pci 0000:00:1f.3: error: cl_dsp_init: timeout HDA_DSP_SRAM_REG_ROM_STATUS read
[ 8.356971] sof-audio-pci 0000:00:1f.3: error: status = 0x00000000 panic = 0x00000000
[ 8.356992] sof-audio-pci 0000:00:1f.3: error: Error code=0xffffffff: FW status=0xffffffff
[ 8.356994] sof-audio-pci 0000:00:1f.3: error: iteration 2 of Core En/ROM load failed: -110
[ 8.356996] sof-audio-pci 0000:00:1f.3: error: dsp init failed after 3 attempts with err: -110
[ 8.357018] sof-audio-pci 0000:00:1f.3: error: status = 0xffffffff panic = 0xffffffff
[ 8.357025] sof-audio-pci 0000:00:1f.3: error: failed to reset DSP
[ 8.357027] sof-audio-pci 0000:00:1f.3: error: failed to boot DSP firmware -110
[ 8.357028] sof-audio-pci 0000:00:1f.3: error: sof_probe_work failed err: -110
Also see something similar on octopus.
2020-07-21 16:03:09.175 3 kernel : [ 7.174025] sof-audio-pci 0000:00:0e.0: error: status = 0x00000000 panic = 0x00000000
2020-07-21 16:03:09.175 3 kernel : [ 7.174043] sof-audio-pci 0000:00:0e.0: error: Error code=0xffffffff: FW status=0xffffffff
2020-07-21 16:03:09.175 3 kernel : [ 7.174047] sof-audio-pci 0000:00:0e.0: error: iteration 0 of Core En/ROM load failed: -110
This particular failure case is not covered in the current DSP status, error code and dump, as a result, it's still clueless to debug further.
Describe the solution you'd like
Since the occurrence is pretty low among a large sample size, and it's difficult to repro in a dev environment, it's desirable to provide more specific diagnostic info to make failure log useful.
Describe alternatives you've considered
Add a dev_err hint to field engineer for things to try like flashing known good bios, report lspci output etc.
Additional context
Nope
Activity
lgirdwood commentedon Aug 3, 2020
@yongzhi1 the 0xffffffff value usually means the DSP IP is in D3. Maybe you could send a kernel PR that dumps some PM regs (both DSP and PCI) in the case we read 0xffffffff.
yongzhi1 commentedon Aug 3, 2020
@lgirdwood That's the idea, could you assign to an expert that has the best knowledge of what to be dumped?
lgirdwood commentedon Aug 4, 2020
@juimonen can you look at this ? We just need a few dev_err() dumps of PCI and DSP PM regs when this happens to help with debug.
juimonen commentedon Aug 5, 2020
@lgirdwood ok.
juimonen commentedon Aug 6, 2020
@lgirdwood @yongzhi1 thesofproject/linux#2347
only thing I could find (easy) was this 1 PM state... not sure if it makes the log too much better :)
yongzhi1 commentedon Aug 13, 2020
Thanks everyone, especially @juimonen and @ranj063 for providing the patches.
Lance also attempted a fix with re-try, just added the link here for reference as there is no way to verify the solution yet:
https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/2340686
plbossart commentedon Aug 13, 2020
FWIW I had a similar fix to work-around boot failures on Cyan during module load/unload tests.
thesofproject/linux#2271
Maybe we should add this capability for real, with a cleaner flow. If we find the root cause we can already reduce the loop to 1 later, but if we don't have the loops the device is stuck.