job.execution.configuration.json package

Submodules

job.execution.configuration.json.exe_config module

Defines the JSON schema for describing the execution configuration

class job.execution.configuration.json.exe_config.ExecutionConfiguration(configuration=None, do_validate=True)

Bases: object

Represents a job execution configuration

add_to_task(task_type, args=None, docker_params=None, env_vars=None, mount_volumes=None, resources=None, settings=None, wksp_volumes=None, workspaces=None)

Adds the given parameters to the task with the given type. The task with the given type must already exist. A mount volume of None indicates a required mount that is missing. A setting value of None indicates a required setting that is missing.

Parameters:
  • task_type (string) – The task type to add the parameters to
  • args (string) – The command arguments for the task
  • docker_params (list()) – The Docker parameters
  • env_vars (dict) – A dict of env var names and values to add to the task
  • mount_volumes (dict) – The mount volumes stored by mount name (a volume may be None)
  • resources (node.resources.node_resources.NodeResources) – The resources
  • settings (dict) – The setting names and values (a value may be None)
  • wksp_volumes (dict) – The workspace volumes stored by workspace name
  • workspaces (dict) – The workspaces stored by name
create_copy()

Creates and returns a copy of this configuration

Returns:The copied configuration
Return type:job.execution.configuration.json.exe_config.ExecutionConfiguration
create_tasks(task_types)

Makes sure that tasks with the given types are created and in the given order. If an already existing task type is not included in the given list, it will be removed.

Parameters:task_types (list()) – The list of task types
get_args(task_type)

Returns the command arguments for the given task type, None if the task type doesn’t exist

Parameters:task_type (string) – The task type
Returns:The command arguments, possibly None
Return type:string
get_dict()

Returns the internal dictionary that represents this execution configuration

Returns:The internal dictionary
Return type:dict
get_docker_params(task_type)

Returns the Docker parameters for the given task type

Parameters:task_type (string) – The task type
Returns:The list of Docker parameters
Return type:list()
get_env_vars(task_type)

Returns the environment variables for the given task type

Parameters:task_type (string) – The task type
Returns:The dict of environment variables
Return type:dict
get_input_workspace_names()

Returns a list of the names of all input workspaces

Returns:The list of the names of all input workspaces
Return type:list()
get_mounts(task_type)

Returns the mounts for the given task type

Parameters:task_type (string) – The task type
Returns:The dict of mount names mapping to volume names
Return type:dict
get_named_docker_volumes()

Returns the names of all (non-host) Docker volumes

Returns:The list of all named Docker volumes
Return type:list()
get_output_workspace_names()

Returns a list of the names of all output workspaces

Returns:The list of the names of all output workspaces
Return type:list()
get_resources(task_type)

Returns the resources for the given task type, None if the task type doesn’t exist

Parameters:task_type (string) – The task type
Returns:The task resources, possibly None
Return type:node.resources.node_resources.NodeResources
get_settings(task_type)

Returns the settings for the given task type

Parameters:task_type (string) – The task type
Returns:The dict of settings
Return type:dict
get_task_id(task_type)

Returns the task ID for the given task type, None if the task type doesn’t exist

Parameters:task_type (string) – The task type
Returns:The task ID, possibly None
Return type:string
get_task_types()

Returns all task types in the configuration in order

Returns:The ordered list of task types
Return type:list()
get_volumes(task_type)

Returns the Docker volumes for the given task type

Parameters:task_type (string) – The task type
Returns:The dict of Docker volumes stored by volume name
Return type:dict
get_workspaces(task_type)

Returns the workspaces for the given task type

Parameters:task_type (string) – The task type
Returns:The list of workspaces
Return type:[job.execution.configuration.workspace.TaskWorkspace]
set_input_files(input_files)

Sets the given input files in the configuration

Parameters:input_files (dict) – A dict where data input name maps to a list of input files
set_output_workspaces(output_workspaces)

Sets the given output workspaces in the configuration

Parameters:output_workspaces (dict) – A dict where job output parameters map to output workspace name
set_task_ids(cluster_id)

Sets the IDs for all of the tasks

Parameters:cluster_id (string) – The cluster ID for the job execution

job.execution.configuration.json.exe_config_1_0 module

Defines the JSON schema for describing the job configuration

class job.execution.configuration.json.exe_config_1_0.ExecutionConfiguration(configuration=None)

Bases: object

Represents a job configuration

add_job_task_workspace(name, mode)

Adds a needed workspace to this job’s job task

Parameters:
  • name (string) – The name of the workspace
  • mode (string) – The mode of the workspace, either MODE_RO or MODE_RW
add_post_task_workspace(name, mode)

Adds a needed workspace to this job’s post task

Parameters:
  • name (string) – The name of the workspace
  • mode (string) – The mode of the workspace, either MODE_RO or MODE_RW
add_pre_task_workspace(name, mode)

Adds a needed workspace to this job’s pre task

Parameters:
  • name (string) – The name of the workspace
  • mode (string) – The mode of the workspace, either MODE_RO or MODE_RW
configure_workspace_docker_params(job_exe, workspaces, docker_volumes)

Configures the Docker parameters needed for each workspace in the job execution tasks. The given job execution must have been set to RUNNING status.

Parameters:
  • job_exe (job.models.JobExecution) – The job execution model (must not be queued) with related job and job_type fields
  • workspaces ({string: storage.models.Workspace}) – A dict of all workspaces stored by name
  • docker_volumes ([string]) – A list to add Docker volume names to
Raises:

Exception – If the job execution is still queued
get_dict()

Returns the internal dictionary that represents this job configuration

Returns:The internal dictionary
Return type:dict
get_job_task_workspaces()

Returns the workspaces needed for the job task

Returns:The job task workspaces
Return type:[job.execution.configuration.json.exe_config_1_0.TaskWorkspace]
get_post_task_workspaces()

Returns the workspaces needed for the post task

Returns:The post task workspaces
Return type:[job.execution.configuration.json.exe_config_1_0.TaskWorkspace]
get_pre_task_workspaces()

Returns the workspaces needed for the pre task

Returns:The pre task workspaces
Return type:[job.execution.configuration.json.exe_config_1_0.TaskWorkspace]
class job.execution.configuration.json.exe_config_1_0.TaskWorkspace(name, mode)

Bases: object

Represents a workspace needed by a job task

job.execution.configuration.json.exe_config_1_1 module

Defines the JSON schema for describing the job configuration

class job.execution.configuration.json.exe_config_1_1.ExecutionConfiguration(configuration=None)

Bases: job.execution.configuration.json.exe_config_1_0.ExecutionConfiguration

Represents a job configuration

add_job_task_setting(name, value)

Adds a setting name/value to this job’s job task

Parameters:
  • name (string) – The setting name to add
  • value (string) – The setting value to add
add_post_task_setting(name, value)

Adds a setting name/value to this job’s post task

Parameters:
  • name (string) – The setting name to add
  • value (string) – The setting value to add
add_pre_task_setting(name, value)

Adds a setting name/value to this job’s pre task

Parameters:
  • name (string) – The setting name to add
  • value (string) – The setting value to add
static convert_configuration(configuration)

Convert the previous Job configuration schema to the 1.1 schema

Parameters:configuration (dict) – The previous configuration
Returns:converted configuration
Return type:dict

Module contents