recipe.seed package

Submodules

recipe.seed.recipe_connection module

Defines connections that will provide data to execute recipes

class recipe.seed.recipe_connection.RecipeConnection

Bases: object

Represents a connection that will provide data to execute recipes. This class contains the necessary description needed to ensure the data provided by the connection will be sufficient to execute the given recipe.

add_input_file(file_name, multiple, media_types, optional)

Adds a new file parameter to this connection

Parameters:
  • file_name (str) – The file parameter name
  • multiple (bool) – Whether the file parameter provides multiple files (True)
  • media_types (list of str) – The possible media types of the file parameter (unknown if None or [])
  • optional (bool) – Whether the file parameter is optional and may not be provided (True)
add_property(property_name)

Adds a new property parameter to this connection

Parameters:property_name (str) – The property parameter name
add_workspace()

Indicates that this connection provides a workspace for storing output files

has_workspace()

Indicates whether this connection provides a workspace for storing output files

Returns:True if this connection provides a workspace, False otherwise
Return type:bool
validate_input_files(files)

Validates the given file parameters to make sure they are valid with respect to the recipe definition.

Parameters:files (dict of str -> tuple(bool, bool, job.configuration.interface.scale_file.ScaleFileDescription)) – Dict of file parameter names mapped to a tuple with three items: whether the parameter is required (True), if the parameter is for multiple files (True), and the description of the expected file meta-data
Returns:A list of warnings discovered during validation.
Return type:list[recipe.configuration.data.recipe_data.ValidationWarning]
:raises recipe.configuration.data.exceptions.InvalidRecipeConnection: If there is a configuration
problem
validate_input_json(property_names)

Validates the given property names to make sure all properties exist if they are required.

Parameters:property_names (dict of str -> bool) – Dict of property names mapped to a bool indicating if they are required
Returns:A list of warnings discovered during validation.
Return type:list[recipe.configuration.data.recipe_data.ValidationWarning]
:raises recipe.configuration.data.exceptions.InvalidRecipeConnection: If there is a configuration
problem
class recipe.seed.recipe_connection.SeedRecipeConnection

Bases: object

Represents a connection that will provide data to execute jobs. This class contains the necessary description needed to ensure the data provided by the connection will be sufficient to execute the given job.

add_input_file(file_name, multiple, media_types, optional, partial)

Adds a new file parameter to this connection

Parameters:
  • file_name (str) – The file parameter name
  • multiple (bool) – Whether the file parameter provides multiple files (True)
  • media_types (list of str) – The possible media types of the file parameter (unknown if None or [])
  • optional (bool) – Whether the file parameter is optional and may not be provided (True)
  • partial (bool) – Flag indicating if the parameter only requires a small portion of the file
add_property(property_name)

Adds a new property parameter to this connection

Parameters:property_name (str) – The property parameter name
add_workspace()

Indicates that this connection provides a workspace for storing output files

has_workspace()

Indicates whether this connection provides a workspace for storing output files

Returns:True if this connection provides a workspace, False otherwise
Return type:bool
validate_input_files(files)

Validates the given file parameters to make sure they are valid with respect to the job interface.

Parameters:files ([job.seed.types.SeedInputFiles]) – List of file inputs
Returns:A list of warnings discovered during validation.
Return type:list[job.configuration.data.job_data.ValidationWarning]

:raises job.configuration.data.exceptions.InvalidConnection: If there is a configuration problem.

validate_properties(property_names)

Validates the given property names to make sure all properties exist if they are required.

Parameters:property_names (dict of str -> bool) – Dict of property names mapped to a bool indicating if they are required
Returns:A list of warnings discovered during validation.
Return type:list[job.configuration.data.job_data.ValidationWarning]

:raises job.configuration.data.exceptions.InvalidConnection: If there is a configuration problem.

recipe.seed.recipe_data module

Defines the data needed for executing a recipe

class recipe.seed.recipe_data.RecipeData(data=None)

Bases: object

Represents the data needed for executing a recipe. Data includes details about the data inputs, links needed to connect shared resources to resource instances in Scale, and details needed to store all resulting output.

add_file_input(name, file_id)

Adds a new file parameter to this job data.

Parameters:
  • data (dict) – The files parameter dict
  • add_to_internal (bool) – Whether we should add to private data dict. Unneeded when used from __init__
add_input_to_data(recipe_input_name, job_data, job_input_name)

Adds the given input from the recipe data as a new input to the given job data

Parameters:
  • recipe_input_name (str) – The name of the recipe data input to add to the job data
  • job_data (job.data.job_data.JobData) – The job data
  • job_input_name (str) – The name of the job data input to add
get_dict()

Returns the internal dictionary that represents this job data

Returns:The internal dictionary
Return type:dict
get_input_file_ids()

Returns a set of scale file identifiers for each file in the job input data.

Returns:Set of scale file identifiers
Return type:{int}
get_input_file_ids_by_input()

Returns the list of file IDs for each input that holds files

Returns:Dict where each file input name maps to its list of file IDs
Return type:dict
get_input_file_info()

Returns a set of scale file identifiers and input names for each file in the job input data.

Returns:Set of scale file identifiers and names
Return type:set[tuple]
get_workspace_id()

Returns the workspace ID in the recipe data

Returns:The workspace ID
Return type:int
set_workspace_id(workspace_id)

Set the workspace ID in the recipe data.

Parameters:workspace_id (int) – The new workspace ID

:raises recipe.configuration.data.exceptions.InvalidRecipeData: If the workspace ID is an invalid type

validate_input_files(files)

Validates the given file parameters to make sure they are valid with respect to the job interface.

Parameters:files (dict of str -> tuple(bool, bool, job.configuration.interface.scale_file.ScaleFileDescription)) – Dict of file parameter names mapped to a tuple with three items: whether the parameter is required (True), if the parameter is for multiple files (True), and the description of the expected file meta-data
Returns:A list of warnings discovered during validation.
Return type:[job.configuration.data.job_data.ValidationWarning]

:raises job.configuration.data.exceptions.InvalidRecipeData: If there is a configuration problem.

validate_input_json(input_json)

Validates the given property names to ensure they are all populated correctly and exist if they are required.

Parameters:input_json ([job.seed.types.SeedInputJson]) – List of Seed input json fields
Returns:A list of warnings discovered during validation.
Return type:[job.configuration.data.job_data.ValidationWarning]

:raises job.configuration.data.exceptions.InvalidRecipeData: If there is a configuration problem.

validate_workspace()

Validates the given file parameters to make sure they are valid with respect to the job interface

Returns:A list of warnings discovered during validation.
Return type:list[recipe.configuration.data.recipe_data.ValidationWarning]

:raises recipe.configuration.data.exceptions.InvalidRecipeData: If the workspace is missing or invalid

class recipe.seed.recipe_data.ValidationWarning(key, details)

Bases: object

Tracks job data configuration warnings during validation that may not prevent the job from working.

recipe.seed.recipe_definition module

Defines the class for managing a recipe definition

class recipe.seed.recipe_definition.RecipeDefinition(definition)

Bases: object

Represents the definition for a recipe. The definition includes the recipe inputs, the jobs that make up the recipe, and how the inputs and outputs of those jobs are connected together.

get_dict()

Returns the internal dictionary that represents this recipe definition

Returns:The internal dictionary
Return type:dict
get_graph()

Returns the recipe graph for this definition

Returns:The recipe graph
Return type:recipe.handlers.graph.RecipeGraph
get_job_type_keys()

Returns a set of tuples that represent keys for each job in the recipe

Returns:Set of referenced job types as a tuple of (name, version)
Return type:set[(str, str)]
get_job_type_map()

Returns a mapping of job name to job type for each job in the recipe

Returns:Dictionary with the recipe job name of each job mapping to its job type
Return type:dict of str -> job.models.JobType
get_job_types(lock=False)

Returns a set of job types for each job in the recipe

Parameters:lock (bool) – Whether to obtain select_for_update() locks on the job type models
Returns:Set of referenced job types
Return type:set[job.models.JobType]
get_jobs_to_create()

Returns the list of job names and types to create for the recipe, in the order that they should be created

Returns:List of tuples with each job’s name and type
Return type:[(str, job.models.JobType)]
validate_connection(recipe_conn)

Validates the given recipe connection to ensure that the connection will provide sufficient data to run a recipe with this definition

Parameters:recipe_conn (recipe.configuration.data.recipe_connection.LegacyRecipeConnection) – The recipe definition
Returns:A list of warnings discovered during validation
Return type:list[recipe.configuration.data.recipe_data.ValidationWarning]
:raises recipe.configuration.data.exceptions.InvalidRecipeConnection: If there is a configuration
problem
validate_data(recipe_data)

Validates the given data against the recipe definition

Parameters:recipe_data (recipe.seed.recipe_data.RecipeData) – The recipe data
Returns:A list of warnings discovered during validation.
Return type:list[recipe.configuration.data.recipe_data.ValidationWarning]

:raises recipe.configuration.data.exceptions.InvalidRecipeData: If there is a configuration problem

validate_job_interfaces()

Validates the interfaces of the recipe jobs in the definition to ensure that all of the input and output connections are valid

Returns:A list of warnings discovered during validation.
Return type:list[job.configuration.data.job_data.ValidationWarning]
:raises recipe.configuration.definition.exceptions.InvalidDefinition:
If there are any invalid job connections in the definition

Module contents