htcondor: can a python executable started via condor_submit access all the values of its condor descriptors?

Question

Here a trivial submission file:

executable = /path/to/myexecutable
error = test.err
output = test.out
log = test.log
request_memory = 1024
request_cpus = 1
queue

myexecutable is a python executable

#!/usr/bin/env python                                                                                                                                                                                                             

import htcondor

# here some code that retrieves the 
# value of a descriptor, e.g. 1024 for 'request_memory'

Once the executable started by condor (e.g. via condor_submit) is there some ways to retrieve the values of the descriptors provided in the submission file ? (including the values of the expanded macros, so directly parsing the file would not do it)

Answer 1

I recommend not using the htcondor python bindings inside of an executable.

I think the most straightforward is to use the arguments command in the submit file to pass information from the submit file to the executable.

Here is how to pass the value of request_memory in your example. The submit file:

executable = /path/to/myexecutable
arguments = $(request_memory)
error = test.err
output = test.out
log = test.log
request_memory = 1024
request_cpus = 1
queue

The executable:

#!/usr/bin/env python                                                                                                                                                                                                             

import sys

# catch the argument
my_memory = sys.argv[1]

# Value of 'my_memory' will match the 'request_memory' from the submit file
print(my_memory)

Note that the variable name itself (request_memory) is not passed, only its value, so you need to make sure the order of your arguments in the submit file matches the order you in which you catch the arguments.

Update: Realized the title was asking for all descriptors.

I don't recommend this, but in the executable's directory should be a file .job.ad that should contain all of that job's ClassAds that existed at the start of the job execution.

The reason I don't recommend it is that the submit descriptors do not necessarily match the name of their corresponding ClassAd. For your example, the submit descriptor request_memory corresponds to the ClassAd RequestMemory.

For your example, this looks like

>>> with open('.job.ad', 'r') as f:
...    for line in f.readlines():
...        if 'RequestMemory' in line:
...            print(line)
RequestMemory = 1024