Skip to content

scheduler: add pending_cnt for task to make pending scheduling possible - #778

Closed
keyonjie wants to merge 2 commits into
thesofproject:masterfrom
keyonjie:scheduler
Closed

keyonjie wants to merge 2 commits into
thesofproject:masterfrom
keyonjie:scheduler

Conversation

@keyonjie

@keyonjie keyonjie commented Dec 24, 2018 •

Copy link
Copy Markdown
Contributor

This series is [4/5] of fixes of #800.

When one instance of a task is queued(queued, pending, running, or
preempted), it is possible that another instance of this same task
scheduling require will come before the previous one is completed. In
today's design, this second instance may be dropped(e.g. for
pipeline_copy() task) and unexpected error will happen.

Here, we add a pending_cnt for the task, accumulate it when needed, and
schedule it when the previous one is finished.

Signed-off-by: Keyon Jie yang.jie@linux.intel.com

@ranj063

ranj063 commented Dec 24, 2018

Copy link
Copy Markdown
Collaborator

@keyonjie im having a hard time imagining a scenario where this would work properly. If a new task instance arrives before the previous instance completes cleanly, we have lost all the context of the previous task, so keeping a pending count to re-schedule this task will only lead to errors in my opinion.

Also, have you verified this is still an actual issue? could you please share some logs to show the issue?

@keyonjie

Copy link
Copy Markdown
Contributor Author

@keyonjie im having a hard time imagining a scenario where this would work properly. If a new task instance arrives before the previous instance completes cleanly, we have lost all the context of the previous task, so keeping a pending count to re-schedule this task will only lead to errors in my opinion.

Also, have you verified this is still an actual issue? could you please share some logs to show the issue?

  1. this is not for suspend/resume, it is for simultaneous pipelines, you can easily see that issues(xrun, let me get some logs and attach) with simultaneous pipelines.
  2. we don't have too much context for task, both for pipeline and IPC, I think the deadline is the only one context we need to save, I have already add handling to deadline in this patch.

@lgirdwood

Copy link
Copy Markdown
Member

@keyonjie lets have some logs of the issue before we commit any speculative changes.

@wenqingfu

Copy link
Copy Markdown
  1. it is for simultaneous pipelines, you can easily see that issues(xrun, let me get some logs and attach) with simultaneous pipelines.

If your PR helps with any Issue, please name it in the PR description, so that that Issue's problem statement help people understand what problem end users will run into without your PR.

@keyonjie

Copy link
Copy Markdown
Contributor Author

this PR helps for #743
I can't say it fix that, but it reduce the possibility of it.

@keyonjie
keyonjie force-pushed the scheduler branch 2 times, most recently from de1c426 to e44f032 Compare December 29, 2018 07:49
@keyonjie

Copy link
Copy Markdown
Contributor Author

@keyonjie lets have some logs of the issue before we commit any speculative changes.

@lgirdwood I captured logs like "task 0xXXX already WIP, schedule it later..." from time to time(e.g. happens every 100s~200s playback) where we will drop a schedule copy, then Xrun happens.

I captured this weeks back with messed debugging code changes, I didn't attach the logs here as it may be confusing. Let me try to re-do that later and attach logs.

@libinyang libinyang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not similar with schedule. Just give some comments purely based on the code.

Comment thread src/arch/xtensa/include/arch/task.h Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why remove check state? Even in TASK_STATE_FREE, TASK_STATE_CANEL, TAKS_STATE_INIT and so on without our any cnt > 0?

Comment thread src/lib/schedule.c Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spin_unlock_irq()

Comment thread src/lib/schedule.c Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pending_cnt may be wrong now.
Let's image if pending_cnt is 2, and the state is PENDING. schedule_task() will be called. In _schedule_task(), the pending_cnt will be added again. But in this case, the cnt should not be added, right?

Comment thread src/audio/pipeline.c Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it should not be named pipeline_schedule_copy_idle()?
And this simple code may impact a lot. Please do a stress test, especially with encoding/decoding and other tasks together. I'm not confident on the side effect of this code.

When one instance of a task is queued(queued, pending, running, or
preempted), it is possible that another instance of this same task
scheduling reqire will come before the previous one is completed. In
today's design, this second instance may be dropped(e.g. for
pipeline_copy() task) and unexpected error will happen.

Here, we add a pending_cnt for the task, accumulate it when needed, and
schedule it when the previous one is finished.

Signed-off-by: Keyon Jie <yang.jie@linux.intel.com>
Signed-off-by: Keyon Jie <yang.jie@linux.intel.com>
@keyonjie

keyonjie commented Jan 23, 2019 •

Copy link
Copy Markdown
Contributor Author

we are seeing error logs in sof-logger without the fix:

    0      1      unknown         651246064.895833         6.041667 /home/keyon/hdd/github/keyonjie/sof/src/lib/schedule.c:362  unexpected task state 5 at task completion
    0      1      unknown         651246070.937500         6.041667 /home/keyon/hdd/github/keyonjie/sof/src/lib/schedule.c:362  unexpected task state 5 at task completion
    0      1      unknown         651246077.135417         6.197917 /home/keyon/hdd/github/keyonjie/sof/src/lib/schedule.c:362  unexpected task state 5 at task completion
    0      1      unknown         651246083.177083         6.041667 /home/keyon/hdd/github/keyonjie/sof/src/lib/schedule.c:362  unexpected task state 5 at task complet

@lgirdwood

Copy link
Copy Markdown
Member

@keyonjie can you confirm the intention here is to allow a task to be scheduled twice ? Wouldn't this mean that it was being rescheduled before it was finished ?

@keyonjie

Copy link
Copy Markdown
Contributor Author

@keyonjie can you confirm the intention here is to allow a task to be scheduled twice ? Wouldn't this mean that it was being rescheduled before it was finished ?

The intention here is to allow a task(e.g. schedule_copy for Pipeline 1) can be add to list at the second time(before the first time done) and be scheduled later, it(the 2nd one) won't be rescheduled before the 1st one was finished.

Without this, the 2nd task might be dropped, or impact the 1st instance which is under scheduling.

@lgirdwood

Copy link
Copy Markdown
Member

@keyonjie ok, but my question is why is Pipe1 being rescheduled before it's completed ?This implies there is not enough free cycles to complete the pipe1 processing in this case ?

@keyonjie

Copy link
Copy Markdown
Contributor Author

@keyonjie ok, but my question is why is Pipe1 being rescheduled before it's completed ?This implies there is not enough free cycles to complete the pipe1 processing in this case ?

@lgirdwood according to my test, some other cases such as pipeline_schedule_idle(not scheduled at once) might also hit to this.

@lgirdwood

Copy link
Copy Markdown
Member

@keyonjie ok, is this only coming from idle scheduling or from regular runtime pipeline scheduling. If it's idle only I'm tempted just to wait for scheduler updates coming from @tlauda and @mmaka1. @mmaka1 @tlauda do you guys agree ?

@keyonjie

Copy link
Copy Markdown
Contributor Author

@keyonjie ok, is this only coming from idle scheduling or from regular runtime pipeline scheduling. If it's idle only I'm tempted just to wait for scheduler updates coming from @tlauda and @mmaka1. @mmaka1 @tlauda do you guys agree ?

@lgirdwood I only found it from idle scheduling till now and no bandwidth to try it more now, I am fine to close this PR and waiting for scheduler updates if it will fix this issue.

@lgirdwood

Copy link
Copy Markdown
Member

@keyonjie ok, lets close now and keep the issue as under investigation

@keyonjie

Copy link
Copy Markdown
Contributor Author

OK, closing it now.

@keyonjie keyonjie closed this Feb 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants