Repository navigation
[TGL][BUG] "error : DSP panic!" occured when do multi-pipeline stress test. #2531
Description
Describe the bug
"DMIC: ASoC: trigger FE cmd: 1 failed: -110" occured when do multi-pipeline stress test. After issue occured, all pipleline can't work. Can be recovered by rebooting system.
To Reproduce
- Sudo reboot
- Run "./multiplex_pipeline-test.sh pcm0p pcm2p pcm6p pcm1c pcm3c".
scripts:
Reproduce rate
Test for 300 times, failed at 17th time.
Expected result
Multi-pipeline should work normally.
Actual result
"DMIC: ASoC: trigger FE cmd: 1 failed: -110" occured when do multi-pipeline stress test.
[ 458.545838] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x34001e successful
[ 458.545844] sof-audio-pci 0000:00:1f.3: ipc tx: 0x60040000: GLB_STREAM_MSG: TRIG_START
[ 458.545916] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x2034001e successful
[ 458.546224] sof-audio-pci 0000:00:1f.3: ipc tx succeeded: 0x60040000: GLB_STREAM_MSG: TRIG_START
[ 458.546306] sof-audio-pci 0000:00:1f.3: ipc tx: 0x60040000: GLB_STREAM_MSG: TRIG_START
[ 458.552455] sof-audio-pci 0000:00:1f.3: error : DSP panic!
[ 458.552461] sof-audio-pci 0000:00:1f.3: panic: dsp_oops_offset 788480 offset 788480
[ 458.552470] sof-audio-pci 0000:00:1f.3: status: fw entered - code 00000005
[ 458.552569] sof-audio-pci 0000:00:1f.3: error: can't enter idle
[ 458.552571] sof-audio-pci 0000:00:1f.3: error: trace point 00004000
[ 458.552572] sof-audio-pci 0000:00:1f.3: error: panic at src/lib/agent.c:51
[ 458.552574] sof-audio-pci 0000:00:1f.3: error: DSP Firmware Oops
[ 458.552576] sof-audio-pci 0000:00:1f.3: EXCCAUSE 0x0000003f EXCVADDR 0x00000000 PS 0x00060925 SAR 0x00000000
[ 458.552578] sof-audio-pci 0000:00:1f.3: EPC1 0x00000000 EPC2 0xbe03fd98 EPC3 0x00000000 EPC4 0x00000000
[ 458.552579] sof-audio-pci 0000:00:1f.3: EPC5 0xbe03fd98 EPC6 0x00000000 EPC7 0x00000000 DEPC 0x00000000
[ 458.552581] sof-audio-pci 0000:00:1f.3: EPS2 0x00060520 EPS3 0x00000000 EPS4 0x00000000 EPS5 0x00060520
[ 458.552582] sof-audio-pci 0000:00:1f.3: EPS6 0x00000000 EPS7 0x00000000 INTENABL 0x00000000 INTERRU 0x00010222
[ 458.552584] sof-audio-pci 0000:00:1f.3: stack dump from 0xbe071120
[ 458.552587] sof-audio-pci 0000:00:1f.3: 0xbe071120: be071150 00000001 00078600 00000001
[ 458.552589] sof-audio-pci 0000:00:1f.3: 0xbe071124: 00000000 00000000 00000033 00000000
[ 458.552590] sof-audio-pci 0000:00:1f.3: 0xbe071128: 95279400 77f0f1cd 9958f828 ffff94da
[ 458.552592] sof-audio-pci 0000:00:1f.3: 0xbe07112c: 9958f828 ffff94da 0000008b 00000000
[ 458.552594] sof-audio-pci 0000:00:1f.3: 0xbe071130: bd4c11cc ffffffff 93692600 ffff94da
[ 458.552596] sof-audio-pci 0000:00:1f.3: 0xbe071134: 95279400 77f0f1cd 9925ccc0 ffff94da
[ 458.552597] sof-audio-pci 0000:00:1f.3: 0xbe071138: 9925ccf8 ffff94da bd488e70 ffffffff
[ 458.552599] sof-audio-pci 0000:00:1f.3: 0xbe07113c: 00000000 00000000 00000000 00000000
[ 458.552600] sof-audio-pci 0000:00:1f.3: error: waking up any trace sleepers
[ 459.053474] sof-audio-pci 0000:00:1f.3: error: host status 0x00000000 dsp status 0x00000000 mask 0x00000003
[ 459.053476] sof-audio-pci 0000:00:1f.3: error: waking up any trace sleepers
[ 459.053480] DMIC: ASoC: trigger FE cmd: 1 failed: -110
[ 459.053486] sof-audio-pci 0000:00:1f.3: pcm: trigger stream 3 dir 1 cmd 0
[ 459.053491] sof-audio-pci 0000:00:1f.3: ipc tx: 0x60050000: GLB_STREAM_MSG: TRIG_STOP
[ 459.557722] sof-audio-pci 0000:00:1f.3: error: ipc timed out for 0x60050000 size 12
[ 459.557727] sof-audio-pci 0000:00:1f.3: info: preventing DSP entering D3 state to preserve context
[ 459.557736] sof-audio-pci 0000:00:1f.3: status: fw entered - code 00000005
[ 459.557848] sof-audio-pci 0000:00:1f.3: error: can't enter idle
[ 459.557849] sof-audio-pci 0000:00:1f.3: error: trace point 00004000
[ 459.557851] sof-audio-pci 0000:00:1f.3: error: panic at src/lib/agent.c:51
Test recipe
Kernel:https://github.com/thesofproject/linux/commits/topic/sof-dev commit:d05959
FW: https://github.com/thesofproject/sof/commits/master commit: 8e5ab9c
Tplg: Same with FW branch, sof-tgl-rt711-i2s-rt1308.tplg
Platform:TGL-U RVP with codec ALC711 in SDW mode and RT1308 in I2S mode
multipipeline_test_27_dmesg.log
Because of FW issue #2507, so can't cat the sof trace.
Activity
sinahuang commentedon Mar 12, 2020
This issue happened after blocked issue #2492 fixed with FW commit:30492d7, from my bi-sect result might be caused by 30492d7 or c596217
@tlauda Please help to check it.
sinahuang commentedon Mar 16, 2020
Tested with master(commit:8e5ab9c2)+sof-dev(commit: 0f90705), issue can be reproduced.
Test for 300 times, failed at 37th time.
multipipeline_test_37_logger_error.log
multipipeline_test_37_all_logger.log
multipipeline_test_37_dmesg.log
tlauda commentedon Mar 16, 2020
@sinahuang It looks like DMIC clocking problem. Are you sure your HW is correctly configured/reworked? We don't see such behavior on our TGL RVPs.
@emilchudzik Could you help with our rework list and BIOS settings?
@singalsu Any ideas?
sinahuang commentedon Mar 17, 2020
My HW configure/reworks as below:
Reworked SDW on ALC711 and I2S on 1308 on TGL-RVP .
BIOS settings:
Set Audio Link Mode to Advanced Link Config
Enable: HDA link, DMIC #0, DMIC DMA trace output can contain repeated trace data blocks #1 , SSP Topic/module - Test Ignore #2, SDW DMA trace output can contain repeated trace data blocks #1 (ALC1308 is on SSP Topic/module - Test Ignore #2, ALC711 is on SDW DMA trace output can contain repeated trace data blocks #1）
Enter HD Audio DSP Features Configuration, Enable I2S and select I2S Codec: Realtek ALC1308
singalsu commentedon Mar 17, 2020
@tlauda Those RORs should not happen. Apparently some slowness with DMA capture apparently does it. The other source for ROR is wrong configuration of decimation but I doubt it as happened since it worked until 37th time. I wonder if in addition to clearing the status bit should the FIFO be emptied by setting FINIT for OUTCTLRx? That would flush all the samples in FIFO and create discontinuity into audio.
tlauda commentedon Mar 17, 2020
@singalsu I think it won't help in the long run. It looks like there is something fundamentally wrong with the DMIC clocking.
@sinahuang Do you have any other TGL RVP? Maybe there is something wrong with the HW on this one, since we don't observe such issue on our boards.
sinahuang commentedon Mar 18, 2020
Used the other TGL RVP with codec ALC711 in SDW mode and RT1308 in I2S mode, issue still can be reproduced.
Test for 300 times, failed at 28th time.
multipipeline_test_28_dmesg.log
tlauda commentedon Mar 18, 2020
@sinahuang If it only reproduces with multiple pipelines, then please verify with higher clock. Please build FW with disabled low power clock: Platform -> [ ] Enable low power ring oscillator as the default clock.
sinahuang commentedon Mar 18, 2020
Checked on FW with disabled low power clock: Platform -> [ ] Enable low power ring oscillator as the default clock, Issue still can be reproduced.
tlauda commentedon Mar 18, 2020
@sinahuang Could you check again? I've rebuilt the master with this option disabled and the test has passed 300 times on your platform.
sinahuang commentedon Mar 19, 2020
Check it again with master (commit: f1fa3be) +sof-dev(commit:45f6424) after disabled low power clock: Platform -> [ ] Enable low power ring oscillator as the default clock , issue still can be reproduced.
Test for 300 times, failed at 29th time.
multipipeline_test_29_dmesg.log
tlauda commentedon Mar 19, 2020
@sinahuang Run 300 iterations again and test passed with low power clock disabled. Can you make sure you are using the right firmware?
sinahuang commentedon Mar 19, 2020
Checked this issue on latest FW (commit:f1fa3be9) with disabled low power clock: Platform -> [ ] Enable low power ring oscillator as the default clock,test for 300 times, issue can't be reproduced.
tlauda commentedon Mar 19, 2020
@mengdonglin Issue caused by overload of DSP. Similar to #2386 and #2280.
keyonjie commentedon Mar 24, 2020
I hit similar issue during DSM playback and echo reference capture running simultaneously, I think we should disable LPRO by default until we can handle it dynamically, thoughts? @mmaka1 @tlauda
[-][TGL][BUG] "DMIC: ASoC: trigger FE cmd: 1 failed: -110" occured when do multi-pipeline stress test.[/-][+][TGL][BUG] "error : DSP panic!" occured when do multi-pipeline stress test.[/+]slawblauciak commentedon Apr 7, 2020
Hi @sinahuang, can you please retest it now using the top of master branch?
The high performance clock is now default on TGL, as a result this issue should no longer occur.
sinahuang commentedon Apr 7, 2020
Issue can't be reproduced with sof-dev commit:052e45d and master commit: b4e8b1e on TGL-U RVP with codec ALC711 in SDW mode and RT1308 in I2S mode. So closed it.