Repository navigation
[BUG] I/O error with DMIC capture pause/release on MTL #6723
Description
Describe the bug
CI observed an Input/output error when check-pause-resume-capture-100 or check-signal-stop-start.sh on MTL platforms.
To Reproduce
TPLG=/lib/firmware/intel/sof-ace-tplg/sof-mtl-rt711-4ch.tplg MODEL=MTLP_RVP_SDW ~/sof-test/test-case/check-pause-resume.sh -c 100 -m capture
Reproduction Rate
100%
Environment
- Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
- Kernel: thesofproject/linux@321028577de9
- SOF: 4a9a99aeb24a
- Name of the topology file
- Topology: sof-ace-tplg/sof-mtl-rt711-4ch.tplg, sof-ace-tplg/sof-hda-generic-4ch.tplg
- Name of the platform(s) on which the bug is observed.
- Platform: MTLP_RVP_SDW, MTLP_RVP_HDA
Screenshots or console output
console output
=== PAUSE ===
(100/100) Wait for 121 ms before pause
(100/100) Wait for 181 ms before resume
declare -- cmd="journalctl_cmd --since=@1670281836"
2022-12-05 23:11:08 UTC [REMOTE_INFO] Entering expect script with:
arecord -D hw:0,10 -r 48000 -c 4 -f S32_LE -vv -i /dev/null -q
spawn arecord -D hw:0,10 -r 48000 -c 4 -f S32_LE -vv -i /dev/null -q
Hardware PCM card 0 'sof-soundwire' device 10 subdevice 0
Its setup is:
stream : CAPTURE
access : RW_INTERLEAVED
format : S32_LE
subformat : STD
channels : 4
rate : 48000
exact rate : 48000 (48000/1)
msbits : 32
buffer_size : 4096
period_size : 1024
period_time : 21333
tstamp_mode : NONE
tstamp_type : MONOTONIC
period_step : 1
avail_min : 1024
period_event : 0
start_threshold : 1
stop_threshold : 4096
silence_threshold: 0
silence_size : 0
boundary : 4611686018427387904
appl_ptr : 0
hw_ptr : 0
#+ | 00%
(1/100) Wait for 142 ms before pause
#+ | 00%
=== PAUSE ===
(1/100) Wait for 107 ms before pause
(1/100) Wait for 104 ms before resume
#+ | 00%
=== PAUSE ===
(2/100) Wait for 189 ms before pause
(2/100) Wait for 181 ms before resume
arecord: pcm_read:2221: read error: Input/output error
2022-12-05 23:11:10 UTC [REMOTE_ERROR] Starting func_exit_handler(), exit status=1, FUNCNAME stack:
2022-12-05 23:11:10 UTC [REMOTE_ERROR] main() @ /home/ubuntu/sof-test/test-case/check-pause-resume.sh
Activity
keqiaozhang commentedon Feb 3, 2023
This issue also happened on MTL-NOCODEC when testing signal-stop-start.
marc-hb commentedon Feb 3, 2023
... since @fredoh9 just fixed signal-stop-start's "green failure" in
[-][BUG] I/O error when check-pause-resume-capture on MTL platforms[/-][+][BUG] I/O error when checking signal stop/start or pause/resume related cases on MTL platforms[/+]keqiaozhang commentedon Feb 20, 2023
This issue also can be reproduced with signal stop/start on MTL platforms. Changing the priority to P1.
39 remaining items
serhiy-katsyuba-intel commentedon Mar 16, 2023
Found a suspicious code in Zephyr dmic.c. In dai_dmic_stop() CIC_CONTROL_SOFT_RESET_BIT bit is set to put DMIC in reset. However, in dai_dmic_start() clearing of CIC_CONTROL_SOFT_RESET_BIT is put into #ifndef CONFIG_SOC_SERIES_INTEL_ACE and so is only done for non MTL platforms.
Tagging @dnikodem
Removing all "#ifndef CONFIG_SOC_SERIES_INTEL_ACE" (there is a second #ifndef for CIC_CONTROL_MIC_MUTE_BIT which seems also not needed/wrong) from dmic.c and running the test could verify the theory.
singalsu commentedon Mar 16, 2023
@dnikodem @serhiy-katsyuba-intel If the unreset is done for MTL it should be done before setting the start bits to use the HW sync start feature.
juimonen commentedon Mar 16, 2023
sorry about polluting this bug with start/stop issue, but it is real in my mtl device and it is caused by this:
I have no idea why it is not causing issues in CI machines, maybe my device is running so on the limit etc.
@serhiy-katsyuba-intel I guess this recent change is exactly limiting for 1 period instead of more data to be pushed. @btian1 FYI.
So I can't say for sure this is a real issue, or just my borked device, or could we face this issue in other scenarios...
ranj063 commentedon Mar 16, 2023
@serhiy-katsyuba-intel @juimonen The real issue is that we're not waiting to ensure that the DMA channel is disabled. The fix for this is in the DW DMA driver here in this PR: zephyrproject-rtos/zephyr#55891
On top of this, there are 2 more things we need to make sure:
abonislawski commentedon Mar 17, 2023
@ranj063 great changes and findings! Thanks for debugging this
lgirdwood commentedon Mar 17, 2023
@ranj063 is the a PR for clearing the DONE bits ?
ranj063 commentedon Mar 17, 2023
@lgirdwood not yet. I will post it today
kv2019i commentedon Mar 22, 2023
@serhiy-katsyuba-intel @ranj063 We now pull in some fixes to dw-dma driver via #7316 . This bug with dmic pause/release does still happen.
One curious data point is that without "e40859f78712 Revert "dma: dw: Do not program SAR/DAR and CTL_HI/LO when using HW LLI"" the dmic pause/release test PASSES on MTL. So this hints the problem is somehow related to state of the the dmic FIFO dma-dw reads from.
ranj063 commentedon Mar 24, 2023
@kv2019i @serhiy-katsyuba-intel this issue is now fixed with the zephyr PR 56209.
keqiaozhang commentedon Mar 29, 2023
We need Zephyr updates in sof main branch. reopening this bug.
keqiaozhang commentedon Apr 3, 2023
This bug is fixed by 7bc3bde. Closing this bug.