product package

Submodules

product.apps module

Defines the application configuration for the product application

class product.apps.ProductConfig(app_name, app_module)

Bases: django.apps.config.AppConfig

Configuration for the product application

label = u'product'
name = u'product'
ready()

Registers the product implementations with other applications.

verbose_name = u'Product'

product.models module

Defines the database model for product files

Bases: django.db.models.base.Model

Represents an ancestry link between two files where the ancestor resulted in the descendant through a series of one or more job executions. A direct ancestry link (parent to child) is formed when the parent is passed as input to a job execution and the child is created as an output of that job execution.

Parameters:
  • ancestor (django.db.models.ForeignKey) – An ancestor file from which the other file is descended
  • descendant (django.db.models.ForeignKey) – A product file that descended from the ancestor file
  • job_exe (django.db.models.ForeignKey) – The job execution that caused this link to be formed
  • job (django.db.models.ForeignKey) – The job that caused this link to be formed
  • recipe (django.db.models.ForeignKey) – The recipe that caused this link to be formed. Note that not all jobs are created from a recipe and so this field could be None.
  • batch (django.db.models.ForeignKey) – The batch associated with this link’s job/recipe, possibly None
  • ancestor_job (django.db.models.ForeignKey) – The higher level job that took the ancestor as input and indirectly caused this link to be formed. Note that this field will be null for directly created files, where the ancestor is a parent that was fed as input to a job and the descendant is a child produced as output by that same job.
  • ancestor_job_exe (django.db.models.ForeignKey) – The higher level job execution that took the ancestor as input and indirectly caused this link to be formed. Note that this field will be null for directly created files, where the ancestor is a parent that was fed as input to a job execution and the descendant is a child produced as output by that same job execution.
  • created (django.db.models.DateTimeField) – When the file link was created
exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

ancestor

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

ancestor_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

ancestor_job

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

ancestor_job_exe

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

ancestor_job_exe_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

ancestor_job_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

batch

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

batch_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

created

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

descendant

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

descendant_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

get_next_by_created(*moreargs, **morekwargs)
get_previous_by_created(*moreargs, **morekwargs)
id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

job

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

job_exe

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

job_exe_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

job_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <product.models.FileAncestryLinkManager object>
recipe

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

recipe_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

class product.models.FileAncestryLinkManager

Bases: django.db.models.manager.Manager

Provides additional methods for handling ancestry links for files used in jobs

Creates the appropriate file ancestry links for the given parent and child files. All database changes are made in an atomic transaction.

Parameters:
  • parent_ids (set of int) – Set of parent file IDs
  • child_ids (set of int) – Set of child file IDs. Passing None can be used to link input files to jobs and recipes without any derived products.
  • job (job.models.Job) – The job that is creating the file links
  • job_exe_id (int) – The job execution that is creating the file links
get_source_ancestor_ids(file_ids)

Returns a list of the source file ancestor IDs for the given file IDs. This will include any of the given files that are source files themselves.

Parameters:file_ids (list()) – The file IDs
Returns:The list of ancestor source file IDs
Return type:list()
get_source_ancestors(file_ids)

Returns a list of the source file ancestors for the given file IDs. This will include any of the given files that are source files themselves.

Parameters:file_ids ([int]) – The file IDs
Returns:The list of ancestor source files
Return type:list[storage.models.ScaleFile]
class product.models.ProductFile(*args, **kwargs)

Bases: storage.models.ScaleFile

Represents a product file that has been created by Scale. This is a proxy model of the storage.models.ScaleFile model. It has the same set of fields, but a different manager that provides functionality specific to product files.

exception DoesNotExist

Bases: storage.models.DoesNotExist

exception MultipleObjectsReturned

Bases: storage.models.MultipleObjectsReturned

VALID_TIME_FIELDS = [u'source', u'data', u'last_modified']
classmethod create()

Creates a new product file

Returns:The new product file
Return type:product.models.ProductFile
objects = <product.models.ProductFileManager object>
class product.models.ProductFileManager(*args, **kwargs)

Bases: django.contrib.gis.db.models.manager.GeoManager

Provides additional methods for handling product files

filter_products(started=None, ended=None, time_field=None, job_type_ids=None, job_type_names=None, job_ids=None, is_published=None, is_superseded=None, file_name=None, job_output=None, recipe_ids=None, recipe_type_ids=None, recipe_job=None, batch_ids=None, order=None)

Returns a query for product models that filters on the given fields. The returned query includes the related workspace, job_type, and job fields, except for the workspace.json_config field. The related countries are set to be pre-fetched as part of the query.

Parameters:
  • started (datetime.datetime) – Query product files updated after this amount of time.
  • ended (datetime.datetime) – Query product files updated before this amount of time.
  • time_field (string) – The time field to use for filtering.
  • job_type_ids ([int]) – Query product files produced by jobs with the given type identifier.
  • job_type_names ([string]) – Query product files produced by jobs with the given type name.
  • job_ids ([int]) – Query product files produced by a given job id
  • is_published (bool) – Query product files flagged as currently exposed for publication.
  • is_superseded (bool) – Query product files that have/have not been superseded.
  • file_name (str) – Query product files with the given file name.
  • job_output (str) – Query product files with the given job output
  • recipe_ids ([int]) – Query product files produced by a given recipe id
  • recipe_job (str) – Query product files produced by a given recipe name
  • recipe_type_ids ([int]) – Query product files produced by a given recipe types
  • batch_ids ([int]) – Query product files produced by batches with the given identifiers.
  • order ([string]) – A list of fields to control the sort order.
Returns:

The product file query
Return type:

django.db.models.QuerySet
get_details(product_id)

Gets additional details for the given product model

Parameters:product_id (int) – The unique identifier of the product.
Returns:The product file model with related workspace
Return type:storage.models.ScaleFile

:raises storage.models.ScaleFile.DoesNotExist: If the file does not exist

get_product_sources(product_file_id, started=None, ended=None, time_field=None, is_parsed=None, file_name=None, order=None)

Returns a query for the list of sources that produced the given product file ID.

Parameters:
  • product_file_id (int) – The product file ID.
  • started (datetime.datetime) – Query source files updated after this amount of time.
  • ended (datetime.datetime) – Query source files updated before this amount of time.
  • time_field (string) – The time field to use for filtering.
  • is_parsed (bool) – Query source files flagged as successfully parsed.
  • file_name (str) – Query source files with the given file name.
  • order ([string]) – A list of fields to control the sort order.
Returns:

The list of source files that match the time range.
Return type:

list[storage.models.ScaleFile]
get_products(started=None, ended=None, time_field=None, job_type_ids=None, job_type_names=None, job_ids=None, is_published=None, file_name=None, job_output=None, recipe_ids=None, recipe_type_ids=None, recipe_job=None, batch_ids=None, order=None)

Returns a list of product files within the given time range.

Parameters:
  • started (datetime.datetime) – Query product files updated after this amount of time.
  • ended (datetime.datetime) – Query product files updated before this amount of time.
  • time_field (string) – The time field to use for filtering.
  • job_type_ids ([int]) – Query product files produced by jobs with the given type identifier.
  • job_type_names ([string]) – Query product files produced by jobs with the given type name.
  • job_ids ([int]) – Query product files produced by a given job id
  • is_published (bool) – Query product files flagged as currently exposed for publication.
  • file_name (str) – Query product files with the given file name.
  • job_output (str) – Query product files with the given job output
  • recipe_ids ([int]) – Query product files produced by a given recipe id
  • recipe_job (str) – Query product files produced by a given recipe name
  • recipe_type_ids ([int]) – Query product files produced by a given recipe types
  • batch_ids ([int]) – Query product files produced by batches with the given identifiers.
  • order ([string]) – A list of fields to control the sort order.
Returns:

The list of product files that match the time range.
Return type:

list[storage.models.ScaleFile]
populate_source_ancestors(products)

Populates each of the given products with its source file ancestors in a field called “source_files”

Parameters:products (list[storage.models.ScaleFile]) – List of products
publish_products(*args, **kwargs)

Publishes all of the products produced by the given job execution. All database changes will be made in an atomic transaction.

Parameters:
  • job_exe_id (int) – The job execution ID
  • job (job.models.Job) – The locked job model
  • when (datetime.datetime) – When the products were published
unpublish_products(job_ids, when)

Unpublishes all of the published products created by the given job IDs

Parameters:
  • job_ids (list()) – The job IDs
  • when (datetime.datetime) – When the products were unpublished
unpublish_products_old(root_job_id, when)

Unpublishes all of the published products created by the superseded jobs with the given root ID :param root_job_id: The root superseded job ID :type root_job_id: int :param when: When the products were unpublished :type when: datetime.datetime

upload_files(file_entries, input_file_ids, job_exe, workspace)

Uploads the given local product files into the workspace.

Parameters:
Returns:

The list of the saved product models
Return type:

[storage.models.ScaleFile]

product.serializers module

Defines the serializers for products

class product.serializers.ProductFileBaseSerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: storage.serializers.ScaleFileSerializerV6

Converts product file model fields to REST output

class product.serializers.ProductFileDetailsSerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: product.serializers.ProductFileSerializer

Converts product file model fields to REST output

class product.serializers.ProductFileSerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: product.serializers.ProductFileBaseSerializer

Converts product file model fields to REST output

class JobTypeBaseSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: util.rest.ModelIdSerializer

Converts job type model fields to REST output

class product.serializers.ProductFileUpdateField(read_only=False, write_only=False, required=None, default=<class rest_framework.fields.empty>, initial=<class rest_framework.fields.empty>, source=None, label=None, help_text=None, style=None, error_messages=None, validators=None, allow_null=False)

Bases: rest_framework.fields.Field

Field for displaying the update information for a product file

to_representation(value)

Converts the model to its update information

Parameters:value (product.models.ProductFile) – the product file model
Return type:dict
Returns:the dict with the update information
type_label = u'update'
type_name = u'UpdateField'
class product.serializers.ProductFileUpdateSerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: product.serializers.ProductFileSerializer

Converts product file updates to REST output

class SourceFileBaseSerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: storage.serializers.ScaleFileSerializerV6

Converts source file model fields to REST output

product.types module

class product.types.ProductFileMetadata(output_name, local_path, media_type=None, remote_path=None, data_start=None, data_end=None, geojson=None, source_started=None, source_ended=None, source_sensor_class=None, source_sensor=None, source_collection=None, source_task=None, url=None, job_name=None, job_version=None, package_version=None)

Bases: object

Module contents