Repository navigation
[BUG] empty kernel logs on MAX_WAIT_FW_LOADING failures #1112
Description
Describe the bug
#1059 added the ability to wait for the firmware to be loaded, which solved a number of issues, including some i915 timeout issues.
However sof-test does not collect kernel logs when this fails. This is very problematic because the only way to know why the firmware is not loaded is to look at kernel logs.
To Reproduce
- sudo rm -rf sof-tests/logs/* # do not delete the logs/ directory itself
sof-test/tools/kmod/sof_remove.sh- Run any test.
- wc -l logs///dmesg.txt
=> dmesg.txt is empty
Expected behavior
Kernel logs are collected.
Detail Info
- Branch name and commit hash of the 3 repositories: sof (firmware/topology), linux (kernel driver) and sof-test (test case)
- Kernel: any
- SOF: any
- SOF-TEST: e3d7664
- Test report ID (if you find it from test report)
- ID: any. Recent examples in Fix bad merge linux#4628
- Test DUT Model (or a brief discribtion about the device)
- MODEL: any
- Test TPLG
- TPLG: any
- Test case (what test script and how you run it)
- TESTCASE: any
cc:
Activity
marc-hb commentedon Dec 5, 2023
Seen again in https://sof-ci.01.org/sofpr/PR8550/build552/devicetest/index.html (thesofproject/sof#8550)
marc-hb commentedon Mar 21, 2024
Still a problem:
https://sof-ci.01.org/sofpr/PR8910/build3598/devicetest/index.html
marc-hb commentedon Apr 5, 2024
We had a surge of FW failing to load for various reasons:
We need to fix this, first (and small) part:
marc-hb commentedon Apr 9, 2024
I just noticed #1036 by chance, possibly a duplicate
Explicitly invoke start_test() in every test-case/*.sh
Explicitly invoke start_test() in every test-case/*.sh
Explicitly invoke start_test() in every test-case/*.sh
verify-kernel-boot-log.sh: fix missing boot logs on failure
marc-hb commentedon May 8, 2024
Fix submitted: #1191EDIT: no, this was only for #1036 and verify-kernel-boot-log.sh. Need to test other tests.
verify-kernel-boot-log.sh: fix missing boot logs on failure
marc-hb commentedon May 16, 2024
Lucky us we just got a complete, real-world sample case here where the firmware did not load (topology mismatch)
https://sof-ci.01.org/linuxpr/PR4995/build3008/devicetest/index.html
The recent,
verify-kernel-boot-log.shfixed in #1191 worked great.The other, consecutive tests were not too bad. Only 2 CI TIMEOUTS, all other tests timed out after MAX_WAIT_FW_LOADING.
The lack of kernel logs in consecutive tests is expected after all: nothing new to show.
lib.sh: switch MTL back to default MAX_WAIT_FW_LOADING=3 seconds
marc-hb commentedon May 16, 2024
That's only because MTL still has an unreasonably long MAX_WAIT_FW_LOADING, otherwise there would have not been any CI timeout.
Fixed in:
lib.sh: switch MTL back to default MAX_WAIT_FW_LOADING=3 seconds
marc-hb commentedon May 16, 2024
#1196 merged, good enough.