job.configuration.interface package

Submodules

job.configuration.interface.exceptions module

Defines exceptions that can occur when interacting with a job interface

exception job.configuration.interface.exceptions.InvalidEnvironment

Bases: exceptions.Exception

Exception indicating that the provided definition of a job interface was invalid

exception job.configuration.interface.exceptions.InvalidInterfaceDefinition

Bases: exceptions.Exception

Exception indicating that the provided definition of a job interface was invalid

job.configuration.interface.job_interface module

Defines the interface for executing a job

class job.configuration.interface.job_interface.JobInterface(definition, do_validate=True)

Bases: object

Represents the interface for executing a job

add_output_to_connection(output_name, job_conn, input_name)

Adds the given output from the interface as a new input to the given job connection

Parameters:
add_workspace_to_data(job_data, workspace_id)

Adds the given workspace ID to the given job data for every output in this job interface

Parameters:
static convert_interface(interface)

Convert a previous Job interface schema to the 1.4 schema

Parameters:interface (dict) – The previous interface
Returns:converted interface
Return type:dict
fully_populate_command_argument(job_data, job_environment, job_exe_id)

Return a fully populated command arguments string. If pre-steps are necessary (see are_pre_steps_needed), they should be run before this. populated with information from the job_data, job_environment, job_input_dir, and job_output_dir.This gets the properties and input_files from the job_data, the shared_resources from the job_environment, and ${input} ${output_dir} from the work_dir. Throws a job.configuration.interface.exceptions.InvalidEnvironment if the necessary pre-steps have not been performed

Parameters:
get_command()

Gets the command :return: the command :rtype: str

get_command_args()

Gets the command arguments :return: the command args :rtype: str

get_dict()

Returns the internal dictionary that represents this job interface

Returns:The internal dictionary
Return type:dict
get_file_output_names()

Returns the output parameter names for all file outputs

Returns:The file output parameter names
Return type:[string]
get_injected_command_args(values, env_vars)

Gets the command arguments with values injected

Parameters:
  • values ({str, str}) – Input values to replace named placeholders in command value
  • env_vars ({}) – Incoming environment variables
Returns:

the command args
Return type:

str
get_mounts()

Gets the mounts defined for the job

Returns:the mounts for a job
Return type:list()
get_settings()

Gets the settings for the job

Returns:the settings object
Return type:dict
perform_post_steps(job_exe, job_data, stdoutAndStderr)

Stores the files and deletes any working directories

Parameters:
Returns:

A tuple of the job results and the results manifest generated by the job execution
Return type:

(job.configuration.results.job_results.JobResults, job.configuration.results.results_manifest.results_manifest.ResultsManifest)
perform_pre_steps(job_data)

Performs steps prep work before a job can actually be run. This includes downloading input files. This returns the command that should be executed for these parameters. :param job_data: The job data :type job_data: job.configuration.data.job_data.JobData :param job_environment: The job environment :type job_environment: dict

populate_command_argument_properties(job_data)

Return the command arguments string, populated with the properties from the job_data.

Parameters:job_data (job.configuration.data.job_data.JobData) – The job data
Returns:command arguments for the given properties
Return type:str
populate_command_argument_settings(command_arguments, exe_configuration, job_type)

Return the command arguments string, populated with the settings from the exe_configuration.

Parameters:
Returns:

command arguments with the settings populated
Return type:

str
populate_env_vars_arguments(param_replacements)

Populates the environment variables with the requested values.

Parameters:param_replacements (dict) – The parameter replacements
Returns:env_vars populated with values
Return type:dict
static replace_command_parameters(command_arguments, param_replacements)

find all occurrences of a parameter with a given name in the command_arguments string and replace it with the param value. If the parameter replacement string in the command uses a custom output ( ${-f :foo}). The parameter will be replaced with the string preceding the colon and the given param value will be appended.

Parameters:
  • command_arguments (string) – The command_arguments that you want to perform the replacement on
  • param_replacements (dict) – The parameters you are searching for
Returns:

The string with all replacements made
Return type:

str
validate_connection(job_conn)

Validates the given job connection to ensure that the connection will provide sufficient data to run a job with this interface

Parameters:job_conn (job.configuration.data.job_connection.JobConnection) – The job data
Returns:A list of warnings discovered during validation.
Return type:list[job.configuration.data.job_data.ValidationWarning]

:raises job.configuration.data.exceptions.InvalidConnection: If there is a configuration problem.

validate_data(job_data)

Ensures that the job_data matches the job_interface description

Parameters:job_data (job.configuration.data.job_data.JobData) – The job data
Returns:A list of warnings discovered during validation.
Return type:list[job.configuration.data.job_data.ValidationWarning]

:raises job.configuration.data.exceptions.InvalidData: If there is a configuration problem.

validate_populated_mounts(exe_configuration)

Ensures that all required mounts are defined in the execution configuration

Parameters:exe_configuration (job.execution.configuration.json.exe_config.ExecutionConfiguration) – The execution configuration
validate_populated_settings(exe_configuration)

Ensures that all required settings are defined in the execution configuration

Parameters:exe_configuration (job.execution.configuration.json.exe_config.ExecutionConfiguration) – The execution configuration
validate_workspace_for_outputs(exe_configuration)

Ensures that all required output workspaces are defined in the execution configuration

Parameters:exe_configuration (job.execution.configuration.json.exe_config.ExecutionConfiguration) – The execution configuration

:raises job.configuration.data.exceptions.InvalidConfiguration: If there is a configuration problem.

job.configuration.interface.job_interface_1_0 module

Defines the interface for executing a job

class job.configuration.interface.job_interface_1_0.JobInterface(definition)

Bases: object

Represents the interface for executing a job

add_output_to_connection(output_name, job_conn, input_name)

Adds the given output from the interface as a new input to the given job connection

Parameters:
add_workspace_to_data(job_data, workspace_id)

Adds the given workspace ID to the given job data for every output in this job interface

Parameters:
static convert_interface(interface)

Convert the previous Job interface schema to the 1.0 schema

Parameters:interface (dict) – The previous interface
Returns:converted interface
Return type:dict
fully_populate_command_argument(job_data, job_environment, job_exe_id)

Return a fully populated command arguments string. If pre-steps are necessary (see are_pre_steps_needed), they should be run before this. populated with information from the job_data, job_environment, job_input_dir, and job_output_dir.This gets the properties and input_files from the job_data, the shared_resources from the job_environment, and ${input} ${output_dir} from the work_dir. Throws a job.configuration.interface.exceptions.InvalidEnvironment if the necessary pre-steps have not been performed

Parameters:
get_command()

Gets the command :return: the command :rtype: str

get_dict()

Returns the internal dictionary that represents this job interface

Returns:The internal dictionary
Return type:dict
get_file_output_names()

Returns the output parameter names for all file outputs

Returns:The file output parameter names
Return type:list of str
perform_post_steps(job_exe, job_data, stdoutAndStderr)

Stores the files and deletes any working directories

Parameters:
Returns:

A tuple of the job results and the results manifest generated by the job execution
Return type:

(job.configuration.results.job_results.JobResults, job.configuration.results.results_manifest.results_manifest.ResultsManifest)
perform_pre_steps(job_data)

Performs steps prep work before a job can actually be run. This includes downloading input files. This returns the command that should be executed for these parameters. :param job_data: The job data :type job_data: job.configuration.data.job_data.JobData :param job_environment: The job environment :type job_environment: dict

populate_command_argument_properties(job_data)

Return the command arguments string, populated with the properties from the job_data.

Parameters:job_data (job.configuration.data.job_data.JobData) – The job data
Returns:command arguments for the given properties
Return type:str
validate_connection(job_conn)

Validates the given job connection to ensure that the connection will provide sufficient data to run a job with this interface

Parameters:job_conn (job.configuration.data.job_connection.JobConnection) – The job data
Returns:A list of warnings discovered during validation.
Return type:list[job.configuration.data.job_data.ValidationWarning]

:raises job.configuration.data.exceptions.InvalidConnection: If there is a configuration problem.

validate_data(job_data)

Ensures that the job_data matches the job_interface description

Parameters:job_data (job.configuration.data.job_data.JobData) – The job data
Returns:A list of warnings discovered during validation.
Return type:list[job.configuration.data.job_data.ValidationWarning]

:raises job.configuration.data.exceptions.InvalidData: If there is a configuration problem.

job.configuration.interface.job_interface_1_1 module

Defines the interface for executing a job

class job.configuration.interface.job_interface_1_1.JobInterface(definition)

Bases: job.configuration.interface.job_interface_1_0.JobInterface

Represents the interface for executing a job

static convert_interface(interface)

Convert the previous Job interface schema to the 1.1 schema

Parameters:interface (dict) – The previous interface
Returns:converted interface
Return type:dict

job.configuration.interface.job_interface_1_2 module

Defines the interface for executing a job

class job.configuration.interface.job_interface_1_2.JobInterface(definition)

Bases: job.configuration.interface.job_interface_1_1.JobInterface

Represents the interface for executing a job

static convert_interface(interface)

Convert the previous Job interface schema to the 1.2 schema

Parameters:interface (dict) – The previous interface
Returns:converted interface
Return type:dict
populate_command_argument_settings(command_arguments, job_configuration)

Return the command arguments string, populated with the settings from the job_configuration.

Parameters:
Returns:

command arguments with the settings populated
Return type:

str
populate_env_vars_arguments(job_configuration)

Populates the environment variables with the requested values.

Parameters:job_configuration (job.execution.configuration.json.exe_config.ExecutionConfiguration) – The job configuration
Returns:env_vars populated with values
Return type:dict
validate_populated_settings(job_exe, job_configuration)

Ensures that all required settings are defined in the job_configuration

Parameters:

job.configuration.interface.job_interface_1_3 module

Defines the interface for executing a job

class job.configuration.interface.job_interface_1_3.JobInterface(definition)

Bases: object

Represents the interface for executing a job

add_output_to_connection(output_name, job_conn, input_name)

Adds the given output from the interface as a new input to the given job connection

Parameters:
add_workspace_to_data(job_data, workspace_id)

Adds the given workspace ID to the given job data for every output in this job interface

Parameters:
static convert_interface(interface)

Convert a previous Job interface schema to the 1.3 schema

Parameters:interface (dict) – The previous interface
Returns:converted interface
Return type:dict
fully_populate_command_argument(job_data, job_environment, job_exe_id)

Return a fully populated command arguments string. If pre-steps are necessary (see are_pre_steps_needed), they should be run before this. populated with information from the job_data, job_environment, job_input_dir, and job_output_dir.This gets the properties and input_files from the job_data, the shared_resources from the job_environment, and ${input} ${output_dir} from the work_dir. Throws a job.configuration.interface.exceptions.InvalidEnvironment if the necessary pre-steps have not been performed

Parameters:
get_command()

Gets the command :return: the command :rtype: str

get_dict()

Returns the internal dictionary that represents this job interface

Returns:The internal dictionary
Return type:dict
get_file_output_names()

Returns the output parameter names for all file outputs

Returns:The file output parameter names
Return type:list of str
perform_post_steps(job_exe, job_data, stdoutAndStderr)

Stores the files and deletes any working directories

Parameters:
Returns:

A tuple of the job results and the results manifest generated by the job execution
Return type:

(job.configuration.results.job_results.JobResults, job.configuration.results.results_manifest.results_manifest.ResultsManifest)
perform_pre_steps(job_data)

Performs steps prep work before a job can actually be run. This includes downloading input files. This returns the command that should be executed for these parameters. :param job_data: The job data :type job_data: job.configuration.data.job_data.JobData :param job_environment: The job environment :type job_environment: dict

populate_command_argument_properties(job_data)

Return the command arguments string, populated with the properties from the job_data.

Parameters:job_data (job.configuration.data.job_data.JobData) – The job data
Returns:command arguments for the given properties
Return type:str
populate_command_argument_settings(command_arguments, job_configuration, job_type)

Return the command arguments string, populated with the settings from the job_configuration.

Parameters:
Returns:

command arguments with the settings populated
Return type:

str
populate_env_vars_arguments(job_configuration, job_type)

Populates the environment variables with the requested values.

Parameters:
Returns:

env_vars populated with values
Return type:

dict
validate_connection(job_conn)

Validates the given job connection to ensure that the connection will provide sufficient data to run a job with this interface

Parameters:job_conn (job.configuration.data.job_connection.JobConnection) – The job data
Returns:A list of warnings discovered during validation.
Return type:list[job.configuration.data.job_data.ValidationWarning]

:raises job.configuration.data.exceptions.InvalidConnection: If there is a configuration problem.

validate_data(job_data)

Ensures that the job_data matches the job_interface description

Parameters:job_data (job.configuration.data.job_data.JobData) – The job data
Returns:A list of warnings discovered during validation.
Return type:list[job.configuration.data.job_data.ValidationWarning]

:raises job.configuration.data.exceptions.InvalidData: If there is a configuration problem.

validate_populated_settings(job_configuration)

Ensures that all required settings are defined in the job_configuration

Parameters:job_configuration (job.execution.configuration.json.exe_config.ExecutionConfiguration) – The job configuration

job.configuration.interface.scale_file module

Defines a class for representing the meta-data description of a file or group of files in a job interface

class job.configuration.interface.scale_file.ScaleFileDescription

Bases: object

Represents the meta-data description of a file or group of files in a job interface

add_allowed_media_type(media_type)

Adds an allowed media type to the file description

Parameters:media_type (str) – The allowed media type
is_media_type_allowed(media_type)

Indicates whether the given media type is allowed by this file description

Parameters:media_type (str) – The media type
Returns:True if the media type is allowed, False otherwise
Return type:bool

Module contents