Repository navigation
How to ask for exlusive use of a GPU ? #52
Description
Issue
I would like to use the lsf-python-api to submit a LSF job which requests exclusive use of hardware resources.
For example, here's how I request 4 cores, 1 GPU, and 1000 MB of RAM with a 1h walltime:
import pythonlsf.lsf as lsf
import os
if lsf.lsb_init("test") > 0:
raise ValueError("Unable to initialise LSF environment")
submitreq = lsf.submit()
submitreq.options = 0
submitreq.options2 = 0
submitreq.options3 = 0
submitreq.options4 = 0
submitreq.beginTime = 0
submitreq.termTime = 0
submitreq.resReq = "rusage[mem=1000:ngpus_physical=1.00] span[ptile=1] affinity[core(4,exclusive=(core,alljobs))*1]"
submitreq.outFile = os.path.join(os.getcwd(), "my-stdout.txt")
submitreq.queue = "x86_1h"
submitreq.options = lsf.SUB_OUT_FILE | lsf.SUB_QUEUE | lsf.SUB_RES_REQ
submitreq.options2 = lsf.SUB2_OVERWRITE_OUT_FILE
submitreq.cwd = os.getcwd()
submitreq.options3 = lsf.SUB3_CWD
submitreq.command = "nvidia-smi"
submitreply = lsf.submitReply()
job_id = lsf.lsb_submit(submitreq, submitreply)
print("Job id is", job_id)Observed behaviour
Running the above prints a job id and when I use bjobs -l <id> I see this at the bottom of the printout:
GPU REQUIREMENT DETAILS:
Combined: mode=shared:mps=no:j_exclusive=no:gvendor=nvidia
Effective: mode=shared:mps=no:j_exclusive=no:gvendor=nvidia
Expected behaviour
If I run an equivalent bsub command I have the option of setting the gpu mode like so:
bsub <other fields....> -gpu num=1:mode=exclusive_process
Then when I look at the bjobs -l <id> information I see this printout:
Combined: num=1:mode=exclusive_process:mps=no:j_exclusive=yes:gvendor=nvidia
Effective: num=1:mode=exclusive_process:mps=no:j_exclusive=yes:gvendor=nvidia
Question
How can I get my python code to ask for exclusive use of the requested GPUs ?
Metadata
Metadata
Assignees
Labels
No labels
Activity
liyancn commentedon Jun 9, 2023
#54
VassilisVassiliadis commentedon Jun 9, 2023
Hi @liyancn I just tested the example you provided and after making a small change (#58) it worked, thank you!