Skip to content

[BUG][ADL]DSP panic when TRIG_START/TRIG_RELEASE in multiple-pause-resume on ADLP_RVP_NOCODEC #4328

New issue
New issue
Closed
Closed
[BUG][ADL]DSP panic when TRIG_START/TRIG_RELEASE in multiple-pause-resume on ADLP_RVP_NOCODEC#4328
Assignees
fredoh9
Labels
ADLApplies to Alder Lake platformI2SApplies to I2S bus for codec connectionIntel Linux Daily testsThis issue can be found in internal Linux daily testsP2Critical bugs or normal featuresbugSomething isn't working as expectedmulticoreIssues observed when not only core#0 is used.stress CI multicore tplgUsing stressful multicore CI topology sof-tgl-nocodec-ci.m4

Description

@XiaoyunWu6666
XiaoyunWu6666
opened on Jun 10, 2021

Describe the bug
DSP panic when TRIG_RELEASE ,in case multiple-pause-resume on
daily-ADL 4496

To Reproduce
TPLG=sof-tgl-nocodec-ci.tplg ~/sof-test/test-case/multiple-pause-resume.sh -r 25

Reproduction Rate
100%

Environment
Kernel Branch: topic/sof-dev
Kernel Commit: 57440285
SOF Branch: main
SOF Commit: 64c5f3a
Topology: sof-tgl-nocodec-ci.tplg (multi-core enabled)

Screenshots or console output
[console]

2021-06-10 04:54:50 UTC [ERROR] Caught kernel log error
===========================>>
[ 6978.023391] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error : DSP panic!
[ 6978.023453] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: status: fw entered - code 00000005
[ 6978.023582] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: runtime exception
[ 6978.023588] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: trace point 00004000
[ 6978.023595] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: panic at :0
[ 6978.023601] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: DSP Firmware Oops
[ 6978.023607] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: Exception Cause: LoadProhibitedCause, A load referenced a page mapped with an attribute that does not permit loads
[ 6978.023613] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: EXCCAUSE 0x0000001c EXCVADDR 0xc0000000 PS       0x00060925 SAR     0x00000000
[ 6978.023622] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: EPC1     0xbe050a21 EPC2     0xbe02d41a EPC3     0x00000000 EPC4    0x00000000
[ 6978.023629] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: EPC5     0x00000000 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[ 6978.023635] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: EPS2     0x00060f20 EPS3     0x00000000 EPS4     0x00000000 EPS5    0x00000000
[ 6978.023641] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00000000 INTERRU 0x00000262
[ 6978.023648] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: stack dump from 0xbe150bc0
[ 6978.023656] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: 0xbe150bc0: 00000000 00000000 00000000 00000000
[ 6978.023663] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: 0xbe150bc4: 7e050a21 be150c40 be0a0700 00000000
[ 6978.023670] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: 0xbe150bc8: f6e94500 7baad15d 000c0800 00000000
[ 6978.023676] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: 0xbe150bcc: 0dead000 00000000 87bd9828 ffff9d8e
[ 6978.023682] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: 0xbe150bd0: c06ee03c ffffffff 8673fc00 ffff9d8e
[ 6978.023688] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: 0xbe150bd4: 88da1200 ffff9d8e b468c4a8 ffffffff
[ 6978.023694] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: 0xbe150bd8: 898d7300 ffff9d8e b46cfbb0 ffffffff
[ 6978.023700] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: 0xbe150bdc: 898d7300 ffff9d8e 00000000 00000000
[ 6978.023705] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: waking up any trace sleepers
[ 6978.023785] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: trace IO error
[ 6978.634361] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc error for 0x60060000 size 12
[ 6978.634430] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at soc_component_trigger on 0000:00:1f.3: -62
[ 6978.634440] kernel:  Port1: ASoC: trigger FE cmd: 3 failed: -62
[ 6978.645412] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc error for 0x60070000 size 12
[ 6978.645432] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at soc_component_trigger on 0000:00:1f.3: -62
[ 6978.645442] kernel:  Port1: ASoC: trigger FE cmd: 4 failed: -62
[ 6978.656132] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc error for 0x60060000 size 12
[ 6978.656168] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at soc_component_trigger on 0000:00:1f.3: -62
[ 6978.656177] kernel:  Port1: ASoC: trigger FE cmd: 3 failed: -62
[ 6978.666692] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc error for 0x60070000 size 12
[ 6978.666713] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at soc_component_trigger on 0000:00:1f.3: -62
[ 6978.666723] kernel:  Port1: ASoC: trigger FE cmd: 4 failed: -62
[ 6979.282302] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc error for 0x60060000 size 12
[ 6979.282385] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at soc_component_trigger on 0000:00:1f.3: -62
[ 6979.282395] kernel:  Port1: ASoC: trigger FE cmd: 3 failed: -62
[ 6979.292703] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc error for 0x60070000 size 12
[ 6979.292723] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at soc_component_trigger on 0000:00:1f.3: -62
[ 6979.292733] kernel:  Port1: ASoC: trigger FE cmd: 4 failed: -62
[ 6979.303212] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc error for 0x60060000 size 12
[ 6979.303248] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at soc_component_trigger on 0000:00:1f.3: -62
[ 6979.303258] kernel:  Port1: ASoC: trigger FE cmd: 3 failed: -62
[ 6979.314187] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc error for 0x60070000 size 12
[ 6979.314207] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at soc_component_trigger on 0000:00:1f.3: -62
[ 6979.314217] kernel:  Port1: ASoC: trigger FE cmd: 4 failed: -62
[ 6979.830007] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc error for 0x60050000 size 12
[ 6979.830054] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at soc_component_trigger on 0000:00:1f.3: -62
[ 6979.830065] kernel:  Port1: ASoC: trigger FE cmd: 0 failed: -62
[ 6979.840428] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc error for 0x60030000 size 12
[ 6979.840462] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at snd_soc_pcm_component_hw_free on 0000:00:1f.3: -62
[ 6979.851082] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc error for 0x80010000 size 12
[ 6979.851102] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed resetting DAI config for SSP1.IN
[ 6989.301163] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc error for 0x80010000 size 12
[ 6989.301184] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed resetting DAI config for SSP0.IN
<<===========================
2021-06-10 04:54:50 UTC [REMOTE_ERROR] pause resume PID 68342 had non-zero exit status
2021-06-10 04:54:50 UTC [REMOTE_ERROR] Starting func_exit_handler(), exit status=1, FUNCNAME stack:
2021-06-10 04:54:50 UTC [REMOTE_ERROR]  die()  @  /home/ubuntu/sof-test/test-case/../case-lib/lib.sh
2021-06-10 04:54:50 UTC [REMOTE_ERROR]  main()  @  /home/ubuntu/sof-test/test-case/multiple-pause-resume.sh:188

[dmesg]
dmesg.txt

Activity

  1. XiaoyunWu6666
    added
    bugSomething isn't working as expected
    on Jun 10, 2021

  2. XiaoyunWu6666 commented on Jun 10, 2021

    @XiaoyunWu6666
    XiaoyunWu6666
    on Jun 10, 2021
    ContributorAuthor

    This error looks similar to #4304 but on different platforms and pipelines.
    So file a new issue here

  3. XiaoyunWu6666
    assigned
    fredoh9
    on Jun 10, 2021
  4. XiaoyunWu6666
    added
    Intel Linux Daily testsThis issue can be found in internal Linux daily tests
    on Jun 10, 2021

  5. mengdonglin commented on Jun 15, 2021

    @mengdonglin
    mengdonglin
    on Jun 15, 2021
    Collaborator

    @fredoh9 Can you reproduce this issue?

  6. mengdonglin
    added
    ADLApplies to Alder Lake platform
    P2Critical bugs or normal features
    P1Blocker bugs or important features
    I2SApplies to I2S bus for codec connection
    and removed
    P2Critical bugs or normal features
    on Jun 15, 2021

  7. fredoh9 commented on Jun 15, 2021

    @fredoh9
    fredoh9
    on Jun 15, 2021
    Contributor

    I'm NOT able to reproduce this issue with my current build.

    • "kernelBranch" : "topic/sof-dev",
    • "kernelCommit" : "57440285",
    • "sofBranch" : "main",
    • "sofCommit" : "73df64444b75"

    I will sync to both tip and try again.

  8. fredoh9 commented on Jun 15, 2021

    @fredoh9
    fredoh9
    on Jun 15, 2021
    Contributor

    I'm NOT able to reproduce this issue with latest tip either. Maybe this is not 100%?

    • "kernelBranch" : "topic/sof-dev",
    • "kernelCommit" : "47c75a5b",
    • "sofBranch" : "main",
    • "sofCommit" : "a68b1af80d1a"

    But what a coincident, I had same(or similar) issue with my TGL Voxel, and I was looking at the issue yesterday.
    Is this something to do with SMART_AMP?

    [    15830666.870946] (          35.104164) c1 smart_amp-test 5.25  ./audio/smart_amp_test.c:299  smart_amp_trigger(), command = 0
[    15830691.506362] (          24.635416) c1 ssp-dai      1.2   /drivers/intel/ssp/ssp.c:842  ssp_trigger() cmd 0
[    15832694.266699] (        2002.760376) c1 wait                         src/lib/wait.c:46   ERROR poll timeout reg 488520 mask 63 val 0 us 937

    slogger_error.txt

  9. fredoh9 commented on Jun 15, 2021

    @fredoh9
    fredoh9
    on Jun 15, 2021
    Contributor

    From the internal test 4496, two arecord(hw:0,0, hw:0,1) was running before hit the problem. Nothing to do with SMART_AMP though. Let me try with the same pipes in manual.

    2021-06-10 04:54:36 UTC [REMOTE_INFO] Port0 to command: arecord -D hw:0,0 -r 48000 -c 2 -f S32_LE -vv -i /dev/null -q
2021-06-10 04:54:36 UTC [REMOTE_INFO] Port1 to command: arecord -D hw:0,1 -r 48000 -c 2 -f S16_LE -vv -i /dev/null -q

    sof-tgl-nocodec-ci

  10. XiaoyunWu6666 commented on Jun 18, 2021

    @XiaoyunWu6666
    XiaoyunWu6666
    on Jun 18, 2021
    ContributorAuthor

    @fredoh9 FYI , not sure whether they are relevant or not.

    in inner daily ADL [4706 model=ADLP_RVP_NOCODEC testcase=multiple-pause-resume-25 ], xrun happens and causes pause push error: connection timed out
    [dmesg log]

    [ 6460.901745] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x60070000: GLB_STREAM_MSG: TRIG_RELEASE
[ 6460.902246] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx succeeded: 0x60070000: GLB_STREAM_MSG: TRIG_RELEASE
[ 6461.403096] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: trigger stream 1 dir 0 cmd 0
[ 6461.403120] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x60050000: GLB_STREAM_MSG: TRIG_STOP
[ 6461.439279] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: trigger stream 5 dir 1 cmd 3
[ 6461.906729] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc timed out for 0x60050000 size 12
[ 6461.906749] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: info: preventing DSP entering D3 state to preserve context
[ 6461.906758] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: status: fw entered - code 00000005
[ 6461.906878] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: unexpected fault 0x00000000 trace 0x00004000
[ 6461.906893] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: hda irq intsts 0x00000000 intlctl 0xc0000083 rirb 00
[ 6461.906900] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: dsp irq ppsts 0x00000000 adspis 0x00000000
[ 6461.906916] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: host status 0x00000000 dsp status 0x00000000 mask 0x00000003
[ 6461.906924] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: waking up any trace sleepers
[ 6461.906950] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0x160]=0x140000 successful
[ 6461.906960] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at soc_component_trigger on 0000:00:1f.3: -110
[ 6461.906971] kernel:  Port1: ASoC: trigger FE cmd: 0 failed: -110
[ 6461.907011] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: free stream 1 dir 0
[ 6461.907019] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: trace IO error

    [sof-logger]

    [   395301920.229613] (     3884852.000000) c3 dw-dma                 src/drivers/dw/dma.c:1102 ERROR dw_dma_get_data_size(): xrun detected
[   395301937.990029] (          17.760416) c3 dai          3.16           src/audio/dai.c:886  ERROR dai_report_xrun(): underrun due to no data available
[   395301955.125445] (          17.135416) c3 dai          3.16           src/audio/dai.c:661  ERROR comp_underrun(): dev->comp.id = 16, source->avail = 192, copy_bytes = 0
[   395302070.281690] (         115.156242) c3 pipe         3.17  ....../pipeline-stream.c:168  ERROR pipeline_copy(): ret = -61, start->comp.id = 16, dir = 1
[   395302089.396273] (          19.114582) c3 pipe         3.17  ..../pipeline-schedule.c:63   ERROR pipeline_task(): xrun recover failed! pipeline will be stopped!
[   395313177.156249] (       11087.759766) c2 dmic-dai     2.0   src/drivers/intel/dmic.c:1557 ERROR dmic_irq_handler(): full fifo B or PDM overrun
  11. XiaoyunWu6666
    changed the title [-][BUG]DSP panic when TRIG_RELEASE on ADLP_RVP_NOCODEC[/-] [+][BUG]DSP panic when TRIG_START/TRIG_RELEASE on ADLP_RVP_NOCODEC[/+] on Jun 23, 2021
  12. XiaoyunWu6666
    changed the title [-][BUG]DSP panic when TRIG_START/TRIG_RELEASE on ADLP_RVP_NOCODEC[/-] [+][BUG][ADL]DSP panic when TRIG_START/TRIG_RELEASE on ADLP_RVP_NOCODEC[/+] on Jun 28, 2021

  13. XiaoyunWu6666 commented on Jul 7, 2021

    @XiaoyunWu6666
    XiaoyunWu6666
    on Jul 7, 2021
    ContributorAuthor

    #4458

  14. fredoh9
    removed
    P1Blocker bugs or important features
    on Jul 12, 2021

  15. fredoh9 commented on Jul 12, 2021

    @fredoh9
    fredoh9
    on Jul 12, 2021
    Contributor

    After switching to unified nocodec tplg, haven't seen this issue, Removed P1

  16. XiaoyunWu6666 commented on Jul 13, 2021

    @XiaoyunWu6666
    XiaoyunWu6666
    on Jul 13, 2021
    ContributorAuthor

    This happened in inner daily 5244 model=TGLU_RVP_NOCODEC_MULTICORE testcase=check-capture-50rounds
    inner daily 5244 model=TGLU_RVP_NOCODEC_MULTICORE testcase=check-capture-50rounds

  17. XiaoyunWu6666
    changed the title [-][BUG][ADL]DSP panic when TRIG_START/TRIG_RELEASE on ADLP_RVP_NOCODEC[/-] [+][BUG][ADL]DSP panic when TRIG_START/TRIG_RELEASE in multiple-pause-resume on ADLP_RVP_NOCODEC[/+] on Jul 13, 2021
  18. mengdonglin
    added
    multicoreIssues observed when not only core#0 is used.
    on Jul 13, 2021
  19. XiaoyunWu6666
    added
    P2Critical bugs or normal features
    on Aug 3, 2021

  20. XiaoyunWu6666 commented on Aug 3, 2021

    @XiaoyunWu6666
    XiaoyunWu6666
    on Aug 3, 2021
    ContributorAuthor

    did not see this in CI daily tests , was there any possible solutions that has solved it?
    I lower the priority , shall we close it?

  21. lgirdwood commented on Aug 3, 2021

    @lgirdwood
    lgirdwood
    on Aug 3, 2021
    Member

    Lets monitor and close if not reproducing in next few days.

  22. mengdonglin
    added
    stress CI multicore tplgUsing stressful multicore CI topology sof-tgl-nocodec-ci.m4
    on Aug 6, 2021

  23. XiaoyunWu6666 commented on Dec 13, 2021

    @XiaoyunWu6666
    XiaoyunWu6666
    on Dec 13, 2021
    ContributorAuthor

    Haven't reproduced it for days , close it

  24. XiaoyunWu6666
    closed this as completedon Dec 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

ADLApplies to Alder Lake platformI2SApplies to I2S bus for codec connectionIntel Linux Daily testsThis issue can be found in internal Linux daily testsP2Critical bugs or normal featuresbugSomething isn't working as expectedmulticoreIssues observed when not only core#0 is used.stress CI multicore tplgUsing stressful multicore CI topology sof-tgl-nocodec-ci.m4

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions