Task priority being ignored #1078

Activity

1. spirali commented on Mar 16, 2026

   

   spirali

   on Mar 16, 2026

   Collaborator

   More actions

   This seems like a bug. Can you share with us the resource requests of these tasks? Or ideally the part of job definition file?
2. kyleabbott commented on Mar 16, 2026

   

   kyleabbott

   on Mar 16, 2026

   Author

   More actions

   Here is hq task info for a couple of randomly selected tasks:

   From new job:
   +--------------+----------------------------------------------------------------------------------+
   | Task ID      | 24039                                                                            |
   | State        | RUNNING                                                                          |
   | Resources    | cpus: 1 compact                                                                  |
   | Priority     | -1                                                                               |
   
   From earlier submitted job:
   +--------------+-------------------------------------------------------------------------------+
   | Task ID      | 42                                                                            |
   | State        | WAITING                                                                       |
   | Resources    | cpus: 1 compact                                                               |
   | Priority     | 0                                                                             |
   
   

   the JDFs are very simple:

   [[task]]
   id = 24039
   command = ["bin/foo", "--bar", ...]
   cwd = "/path/to/stuff"
   priority = -1
   
   [task.env]
   ENV_VAR_A = "1"
   ENV_VAR_B = "1"
   

   And the priority == 0 job above looks the same, except the "priority" line is omitted from the JDF.
3. spirali commented on Mar 20, 2026

   

   spirali

   on Mar 20, 2026

   Collaborator

   More actions

   I was able replicate some problems with priorities in v0.25. I will look on it if it make sense to fix it or wait for new scheduler that complete redo scheduler (it should be merged in few days).
4. kyleabbott commented on Mar 23, 2026

   

   kyleabbott

   on Mar 23, 2026

   Author

   More actions

   I believe at least one cause of lower priority tasks being ran while there are higher priority tasks waiting is in the case that there are multiple workers that all have more tasks assigned to them than they have CPUs. If my understanding is correct, the existing scheduler does not invoke the rebalance code path if there is not at least a single underloaded worker. In this case it's possible for the higher priority tasks to be assigned to a subset of workers, while other workers only have lower priority tasks assigned. I am not sure if this 100% explains the issue I have been seeing, but I believe it at least contributing... In one case I noticed only a small constant number of higher priority tasks running, which happened to exactly match the number of CPUs of a single worker. Upon inspection indeed all of that job's tasks were assigned to that single worker for some reason, while the other workers were packed with lower priority tasks. No rebalancing was taking place because all workers were overloaded.

   Not sure how the new scheduling algorithm would deal with this scenario?
5. spirali commented on Mar 25, 2026

   

   spirali

   on Mar 25, 2026

   Collaborator

   More actions

   You are right that something like this is probably happening and this was the source of main problems with priorities and all changes was mostly about patching specific situations.

   Nevertheless, I have merged new scheduler in the main (#1080), so in tomorrows nightly it should be there. Can you please check if your problem persists?

   The new scheduler completely removes the concept of underloaded/overloaded workers and basically do targeted scheduling to workers (it still supports prefilling tasks to workers, but only when it is safe wrt priorities and proactively retracts tasks when a high priority task is submitted later). I am now writing a paper about the new scheduler, so it will be hopefully documented one day:)

   👍React with 👍1Reacted by kyleabbott
6. kyleabbott commented on Mar 27, 2026

   

   kyleabbott

   on Mar 27, 2026

   Author

   More actions

   You are right that something like this is probably happening and this was the source of main problems with priorities and all changes was mostly about patching specific situations.

   Nevertheless, I have merged new scheduler in the main (#1080), so in tomorrows nightly it should be there. Can you please check if your problem persists?

   The new scheduler completely removes the concept of underloaded/overloaded workers and basically do targeted scheduling to workers (it still supports prefilling tasks to workers, but only when it is safe wrt priorities and proactively retracts tasks when a high priority task is submitted later). I am now writing a paper about the new scheduler, so it will be hopefully documented one day:)

   I will test the priorities with the new scheduler, and I look forward to reading the paper :)