[BUG] [TGL Chrome] Fw panic when we overwrite the firmware & topology binary to the file system first time and boot it #4419
Description
Describe the bug
Fw panic when we copy the firmware & topology binary to the file system first time and boot it
To Reproduce
Steps to reproduce the behavior: (e.g. list commands or actions used to reproduce the bug)
Reproduction Rate
whenever we copy the firmware & topology for the first time on a board.
The problem goes away if we reboot
Expected behavior
No firmware panic and the sound card should bind successfully
Impact
showstopper
Environment
tgl-13 release.
Screenshots or console output
[ 7.550234] sof-audio-pci 0000:00:1f.3: error : DSP panic!
[ 7.550243] sof-audio-pci 0000:00:1f.3: status: fw entered - code 00000005
[ 7.550456] sof-audio-pci 0000:00:1f.3: error: runtime exception
[ 7.550458] sof-audio-pci 0000:00:1f.3: error: trace point 00004000
[ 7.550459] sof-audio-pci 0000:00:1f.3: error: panic at :0
[ 7.550461] sof-audio-pci 0000:00:1f.3: error: DSP Firmware Oops
[ 7.550463] sof-audio-pci 0000:00:1f.3: error: Exception Cause: LoadStorePIFD ataErrorCause, Synchronous PIF data error during LoadStore access
[ 7.550465] sof-audio-pci 0000:00:1f.3: EXCCAUSE 0x0000000d EXCVADDR 0xbe1d00 00 PS 0x00060925 SAR 0x00000000
[ 7.550466] sof-audio-pci 0000:00:1f.3: EPC1 0xbe02e3b7 EPC2 0xbe0424 0a EPC3 0x00000000 EPC4 0x00000000
[ 7.550468] sof-audio-pci 0000:00:1f.3: EPC5 0x00000000 EPC6 0x000000 00 EPC7 0x00000000 DEPC 0x00000000
[ 7.550470] sof-audio-pci 0000:00:1f.3: EPS2 0x00060b20 EPS3 0x000000 00 EPS4 0x00000000 EPS5 0x00000000
[ 7.550471] sof-audio-pci 0000:00:1f.3: EPS6 0x00000000 EPS7 0x000000 00 INTENABL 0x00000000 INTERRU 0x00000222
[ 7.550473] sof-audio-pci 0000:00:1f.3: stack dump from 0xbe131360
[ 7.550476] sof-audio-pci 0000:00:1f.3: 0xbe131360: 3fe6b541 a0000000 3fe5ad5 0 3eb5aa0d
[ 7.550478] sof-audio-pci 0000:00:1f.3: 0xbe131364: 00000000 3ff99999 9980000 0 be136fb0
[ 7.550479] sof-audio-pci 0000:00:1f.3: 0xbe131368: 00000000 00000000 0000000 0 00000000
[ 7.550481] sof-audio-pci 0000:00:1f.3: 0xbe13136c: b66db3a0 ffff89ff 45573dc 0 ffffb2e5
[ 7.550483] sof-audio-pci 0000:00:1f.3: 0xbe131370: 8ec1bb40 ffffffff 8f21b70 0 ffffffff
[ 7.550484] sof-audio-pci 0000:00:1f.3: 0xbe131374: 5c59e028 ffff89ff 000c080 0 00000000
[ 7.550486] sof-audio-pci 0000:00:1f.3: 0xbe131378: 00000000 00000000 45573e5 0 ffffb2e5
[ 7.550488] sof-audio-pci 0000:00:1f.3: 0xbe13137c: 45573e78 ffffb2e5 c055216 3 ffffffff
[ 7.812424] NET: Registered protocol family 38
[ 7.878571] x86/PAT: flashrom:2707 conflicting memory types fe000000-10000000 0 uncached-minus<->uncached
[ 7.878575] x86/PAT: reserve_memtype failed [mem 0xfe000000-0xffffffff], trac k uncached-minus, req uncached-minus
[ 7.889983] Bluetooth: rfcomm_init_sockets() RFCOMM socket layer initialized
[ 7.890009] Bluetooth: rfcomm_init() RFCOMM ver 1.11
[ 7.908778] NET: Registered protocol family 40
[ 7.932593] Bluetooth: msft_le_set_advertisement_filter_enable_cb() hci0: MSF T filter_enable is already on
[ 7.957788] input: cros_fp_input as /devices/virtual/input/input13
[ 8.051731] sof-audio-pci 0000:00:1f.3: error: ipc timed out for 0x40070000 s ize 12
[ 8.051740] sof-audio-pci 0000:00:1f.3: status: fw entered - code 00000005
[ 8.051947] sof-audio-pci 0000:00:1f.3: error: runtime exception
[ 8.051948] sof-audio-pci 0000:00:1f.3: error: trace point 00004000
[ 8.051950] sof-audio-pci 0000:00:1f.3: error: panic at :0
[ 8.051951] sof-audio-pci 0000:00:1f.3: error: DSP Firmware Oops
[ 8.051953] sof-audio-pci 0000:00:1f.3: error: Exception Cause: LoadStorePIFD ataErrorCause, Synchronous PIF data error during LoadStore access
[ 8.051954] sof-audio-pci 0000:00:1f.3: EXCCAUSE 0x0000000d EXCVADDR 0xbe1d00 00 PS 0x00060925 SAR 0x00000000
[ 8.051956] sof-audio-pci 0000:00:1f.3: EPC1 0xbe02e3b7 EPC2 0xbe0424 0a EPC3 0x00000000 EPC4 0x00000000
[ 8.051957] sof-audio-pci 0000:00:1f.3: EPC5 0x00000000 EPC6 0x000000 00 EPC7 0x00000000 DEPC 0x00000000
[ 8.051965] sof-audio-pci 0000:00:1f.3: EPS2 0x00060b20 EPS3 0x000000 00 EPS4 0x00000000 EPS5 0x00000000
[ 8.051966] sof-audio-pci 0000:00:1f.3: EPS6 0x00000000 EPS7 0x000000 00 INTENABL 0x00000000 INTERRU 0x00000222
[ 8.051968] sof-audio-pci 0000:00:1f.3: stack dump from 0xbe131360
[ 8.051970] sof-audio-pci 0000:00:1f.3: 0xbe131360: 3fe6b541 a0000000 3fe5ad5 0 3eb5aa0d
[ 8.051971] sof-audio-pci 0000:00:1f.3: 0xbe131364: 00000000 3ff99999 9980000 0 be136fb0
[ 8.051972] sof-audio-pci 0000:00:1f.3: 0xbe131368: 00000000 00000000 0000000 0 00000000
[ 8.051979] sof-audio-pci 0000:00:1f.3: 0xbe13136c: 406535c0 ffffb2e5 0000328 2 00000000
[ 8.051980] sof-audio-pci 0000:00:1f.3: 0xbe131370: 406535a0 ffffb2e5 2883140 0 49f1c1f5
[ 8.051985] sof-audio-pci 0000:00:1f.3: 0xbe131374: b49380d8 ffff89ff 5c59e02 8 ffff89ff
[ 8.051986] sof-audio-pci 0000:00:1f.3: 0xbe131378: 00000000 00000000 406536f 0 ffffb2e5
[ 8.051987] sof-audio-pci 0000:00:1f.3: 0xbe13137c: b4938000 ffff89ff 8dd6932 5 ffffffff
[ 8.052014] sof-audio-pci 0000:00:1f.3: error: hda irq intsts 0x00000000 intl ctl 0xc0000000 rirb 00
[ 8.052015] sof-audio-pci 0000:00:1f.3: error: dsp irq ppsts 0x00000000 adspi s 0x00000000
[ 8.052033] sof-audio-pci 0000:00:1f.3: error: host status 0x00000000 dsp sta tus 0x00000000 mask 0x00000003
[ 8.052035] sof-audio-pci 0000:00:1f.3: error: core 1 enable ipc failure -110
[ 8.052040] sof-audio-pci 0000:00:1f.3: error: enable core: -110
[ 8.052055] sof-audio-pci 0000:00:1f.3: ASoC: failed to load widget PCM99C
Activity
RDharageswari commentedon Jun 28, 2021
FW_panic_while_boot.txt
lgirdwood commentedon Jun 29, 2021
@RDharageswari can you confirm that you copy files (how ? over NFS, scp ?) and then immediately reboot (how) ?
This could also be kernel issue or FS cache. Have you tried copy files then unload/reload modules ?
@ujfalusi anything in kernel flow for FW loading that would break here ?
bkokoszx commentedon Jun 29, 2021
@lgirdwood @RDharageswari
I was able to reproduce this issue. I remove tplg and fw files, then I copy tplg/fw using scp and reboot by
sudo reboot. Sometimes after reboot I get dsp panic in dmesg.Once I also had a case, that just after reboot I did not have panic, but then I tried to run:
and after that I got similar panic.
ujfalusi commentedon Jun 29, 2021
Can it be not related to firmware copying and it is just a coincident? Or is it more likely to happen after delete fw/tplg, copy new fw/tplg, then reboot?
Fwiw, in the log the panic happens before sending (SOF_IPC_GLB_PM_MSG|SOF_IPC_PM_CORE_ENABLE) message (
ipc timed out for 0x40070000 size 12), it is hard to say if it is for thepipeline->coreor for theswidget->coreand also from where it happens.Can it be reproduced if only the firmware or the tplg file is deleted/copyied ?
After the first DSP crash you can reboot repeatably without any issue?
Can it be caused by unclean shutdown of the DSP in a way that same stale data remain from the previous boot and when we load the new tplg, the firmware tries to interpret it and fails?
Can you try to delete the fw/tplg files and then copy the same ones back? Does that also triggers a crash?
Can you try to update the fw/tplg and then shut down and then power on? Does the crash remain?
Can you reboot repeatedly without touching the fw/tplg and see if there is really no DSP crash?
I have been doing locally: update the fw/tplg, unload all drivers, load the drivers again and now the new firmware is in use. Can you try the same, so update the fw/tplg then remove/insert the modules and see if the crash happens also like that?
ranj063 commentedon Jun 29, 2021
@RDharageswari can you confirm this is tgl-013 specific problem?
RDharageswari commentedon Jun 29, 2021
I have seen this problem only with tgl-13. Did not hit it before.
plbossart commentedon Jun 29, 2021
This reminds me of the odd behavior with firmware being changed reported by @marc-hb thesofproject/linux#2954
Some sort of firmware caching issue?
20 remaining items
marc-hb commentedon Jul 12, 2021
Maybe this was just because the original firmware is not affected, only the new one.
Or because the copy adds a delay.
slawblauciak commentedon Jul 12, 2021
@marc-hb I was able to reproduce the issue as well. It only occurs after the FW binary is replaced.
marc-hb commentedon Jul 12, 2021
With a different version? If not then it could be a timing issue, has anyone tried to replace another, unrelated firmware file?
A delay can change the state of the DSP when the reboot starts.
slawblauciak commentedon Jul 12, 2021
No, the same version even.
marc-hb commentedon Jul 12, 2021
OK then this must be a timing issue. Can you please try to copy similar and unrelated files in the same directories and see if it reproduces the same?
cc: thesofproject/linux#3078
[-][BUG] [TGL] Fw panic when we overwrite the firmware & topology binary to the file system first time and boot it[/-][+][BUG] [TGL Chrome] Fw panic when we overwrite the firmware & topology binary to the file system first time and boot it[/+]github-actions commentedon Jun 19, 2025
This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.