Repository navigation
How to run a job with 'qsub' arguments or script flags #61
Description
When I run a script on an HPC with a scheduler such as SGE, my script might contain flags like these:
#!/bin/bash
#$ -pe openmpi 32
#$ -A TensorFlow
#$ -N rqsub_tile
#$ -cwd
#$ -S /bin/bash
#$ -q gpu0.q
#$ -l excl=true
run_script.sh Or, I might run a qsub command like this:
$ qsub -wd $PWD -o :${qsub_logdir}/ -e :${qsub_logdir}/ -j y -N "$job_name" -pe threaded 6-18 -l mem_free=10G -l mem_token=10G run_script.shI've spent a lot of time reading the docs, running through the examples, and Google'ing, but I cant find anything that actually shows how to use these parameters with this Python DRMAA library. Is it described somewhere? It sounds like something that might be part of the JobTemplate described here, but the docs do not mention this, or tell you much at all really.
Activity
jakirkham commentedon Mar 12, 2018
So DRMAA would be used to set the working directory (
workingDirectory), stdout/stderr log filenames (outputPath/errorPath), job name (jobName), and command to be run (remoteCommand). The-j yis redundant for DRMAA; so, would be dropped. The rest don't have a corresponding DRMAA parameter, but can be passed in usingnativeSpecification, which is just a catchall string parameter for anything thatqsub(or similar on other clusters) would need.As a simple example, you might find this code interesting. It creates a light CLI around DRMAA and allows submitting jobs without much of the boiler plate that
qsubneeds.stevekm commentedon Mar 12, 2018
Thanks. Found a mention of it here:
jakirkham commentedon Mar 31, 2018
Glad to hear that helped. Good to close?