storage package

Subpackages

Submodules

storage.apps module

Defines the configuration for the storage application

class storage.apps.StorageConfig(app_name, app_module)

Bases: django.apps.config.AppConfig

Configuration for the storage app

label = u'storage'
name = u'storage'
ready()

Registers components related to storage

verbose_name = u'Storage'

storage.container module

Defines the methods for handling file systems in the container’s local volume

storage.container.get_workspace_volume_path(name)

Returns the absolute local path within the container onto which this workspace broker’s container volume is mounted

Parameters:name (string) – The name of the workspace
Returns:The absolute local path of the workspace’s mount
Return type:string

storage.delete_files_job module

Defines the functions necessary to delete a file from a workspace

storage.delete_files_job.delete_files(files, volume_path, broker)

Deletes the given files within a workspace.

Parameters:
  • files ([collections.namedtuple]) – List of named tuples containing path and ID of the file to delete.
  • volume_path (string) – Absolute path to the local container location onto which the volume file system was mounted, None if this broker does not use a container volume
  • broker (storage.brokers.broker.Broker) – The storage broker

storage.exceptions module

Defines the exceptions related to files and storage methods

exception storage.exceptions.ArchivedWorkspace

Bases: exceptions.Exception

Exception indicating an attempt was made to store or retrieve a file with an archived (no longer active) workspace

exception storage.exceptions.DeletedFile(file_name)

Bases: error.exceptions.ScaleError

Error class indicating an attempt was made to retrieve a deleted file (a file whose is_deleted flag is true in the database)

get_log_message()

See error.exceptions.ScaleError.get_log_message()

exception storage.exceptions.InvalidDataTypeTag

Bases: exceptions.Exception

Exception indicating an attempt to add an invalid data type tag to a file

exception storage.exceptions.MissingFile(file_name)

Bases: error.exceptions.ScaleError

Error class indicating an attempt was made to retrieve a missing file

get_log_message()

See error.exceptions.ScaleError.get_log_message()

exception storage.exceptions.MissingVolumeMount

Bases: exceptions.Exception

Exception indicating that a workspace required a volume file system mount in order to perform an operation and the required volume mount was missing

storage.geospatial_utils module

Defines functions for geospatial information

storage.geospatial_utils.get_center_point(geom)

Returns a center point for the given geometry object.

Parameters:geom (GEOSGeometry) – The geometry
Return type:Point
Returns:the center point
storage.geospatial_utils.parse_geo_json(geo_json)

Parses GeoJSON and returns a geometry object and metadata.

Parameters:geo_json (dict) – The geo json to parse
Return type:GEOSGeometry, dict
Returns:the geometry and metadata
storage.geospatial_utils.parse_geo_json_file(geo_json_path)

Parses GeoJSON from a file and returns a geometry object and metadata.

Parameters:geo_json_path (str) – The absolute file path of the GeoJSON file
Return type:GEOSGeometry, dict
Returns:the geometry and metadata

storage.media_type module

Defines functions for determining the media type of a file

storage.media_type.get_media_type(file_name)

Returns the media type of a file based upon the file’s name

Parameters:file_name (string) – The name of the file
Returns:The media type of the file
Return type:string

storage.models module

Defines the database models for file information and workspaces

class storage.models.CountryData(*args, **kwargs)

Bases: django.db.models.base.Model

Represents country borders and official abbreviations

Parameters:
  • name (django.db.models.CharField) – The full name of the country
  • fips (django.db.models.CharField) – FIPS digraph for the country
  • gmi (django.db.models.CharField) – gmi trigraph for the country
  • iso2 (django.db.models.CharField) – ISO digraph for the country
  • iso3 (django.db.models.CharField) – ISO trigraph for the country
  • iso_num (django.db.models.IntegerField) – ISO number for the country
  • border (django.contrib.gis.geos.geometry.GEOSGeometry) – Border geometry of this country
  • effective (django.db.models.DateTimeField) – When the country information including the border are effective.
  • is_deleted (django.db.models.BooleanField) – Whether the country has been deleted or not
  • created (django.db.models.DateTimeField) – When the country model was created
  • deleted (django.db.models.DateTimeField) – When the country model was deleted
  • last_modified (django.db.models.DateTimeField) – When the country model was last modified
exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

border
created

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

deleted

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

effective

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

fips

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

get_next_by_created(*moreargs, **morekwargs)
get_next_by_effective(*moreargs, **morekwargs)
get_next_by_last_modified(*moreargs, **morekwargs)
get_previous_by_created(*moreargs, **morekwargs)
get_previous_by_effective(*moreargs, **morekwargs)
get_previous_by_last_modified(*moreargs, **morekwargs)
gmi

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

is_deleted

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

iso2

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

iso3

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

iso_num

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

last_modified

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

name

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <storage.models.CountryDataManager object>
scalefile_set

Accessor to the related objects manager on the forward and reverse sides of a many-to-many relation.

In the example:

class Pizza(Model):
    toppings = ManyToManyField(Topping, related_name='pizzas')

pizza.toppings and topping.pizzas are ManyToManyDescriptor instances.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

class storage.models.CountryDataManager

Bases: django.db.models.manager.Manager

Provides additional methods for handling country data

get_effective(target_date, name=None, iso2=None)

Get the country data entry for a name or iso2 abbreviation and target date such that this is the most recent entry whose effective date is before the target.

Parameters:
  • target_date (datetime.datetime) – The target date
  • name (string) – The name of the country. Mutually exclusive with iso2. One of these is required.
  • iso2 (string) – The iso2 abbreviation of the country. Mutually exclusive with name. One of these is required.
Returns:

A query set
Return type:

django.db.models.query.QuerySet
get_intersects(geom, target_date)

Get the countries whose borders intersect the specified geometry and whose effective date is before the target.

Parameters:
  • geom (django.contrib.gis.geos.geometry.GEOSGeometry) – The geometry (point, poly, etc.) to search.
  • target_date (datetime.datetime) – The target date
Returns:

A dict of intersected countries mapped to entities
Return type:

dict
update_border(*args, **kwargs)

Updates the country border geometry for an existing country, adding a new entry for the new effective date.

Parameters:
  • name (string) – The name of an existing country
  • border (GEOSGeometry or dict) – The new border geometry. Either GEOSGeometry or geojson (which will be converted to GEOSGeometry)
  • effective (datetime.datetime) – The effective date for the new border. If None, now() will be used
class storage.models.PurgeResults(*args, **kwargs)

Bases: django.db.models.base.Model

Represents the results from purge operations

Parameters:
  • source_file_id (django.db.models.PositiveIntegerField) – The ID of the source file purged
  • trigger_event (django.db.models.ForeignKey) – The event that triggered the creation of the purge process
  • num_jobs_deleted (django.db.models.PositiveIntegerField) – The number of jobs deleted as part of the purge process
  • num_recipes_deleted (django.db.models.PositiveIntegerField) – The number of recipes deleted as part of the purge process
  • num_products_deleted (django.db.models.PositiveIntegerField) – The number of products deleted as part of the purge process
  • purge_started (django.db.models.DateTimeField) – The datetime that the purge process began
  • purge_completed (django.db.models.DateTimeField) – The datetime that the purge process completed
  • force_stop_purge (django.db.models.BooleanField) – Stops new messages from being generated if False
exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

force_stop_purge

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

get_next_by_purge_started(*moreargs, **morekwargs)
get_previous_by_purge_started(*moreargs, **morekwargs)
id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

num_jobs_deleted

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

num_products_deleted

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

num_recipes_deleted

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <django.db.models.manager.Manager object>
purge_completed

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

purge_started

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

source_file_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

trigger_event

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

trigger_event_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

class storage.models.ScaleFile(*args, **kwargs)

Bases: django.db.models.base.Model

Represents a file that is stored within a Scale workspace

Parameters:
  • file_name (django.db.models.CharField) – The name of the file
  • file_type (django.db.models.CharField) – The (Scale) type of the file
  • media_type (django.db.models.CharField) – The IANA media type of the file
  • file_size (django.db.models.BigIntegerField) – The size of the file in bytes
  • data_type_tags (django.db.models.ArrayField) – An array of data type “tags” for the file
  • file_path (django.db.models.CharField) – The relative path of the file in its workspace
  • workspace (django.db.models.ForeignKey) – The workspace that stores this file
  • is_deleted (django.db.models.BooleanField) – Whether the file has been deleted or not
  • uuid (django.db.models.CharField) – A universally unique identifier for the source record. It ensures that subsequent updates of the record will result in the same UUID, which can then be used as a stable permanent link in applications.
  • created (django.db.models.DateTimeField) – When the file model was created
  • deleted (django.db.models.DateTimeField) – When the file was deleted
  • last_modified (django.db.models.DateTimeField) – When the file model was last modified
  • data_started (django.db.models.DateTimeField) – The start time of the data in this file
  • data_ended (django.db.models.DateTimeField) – The end time of the data in this file
  • geometry (django.contrib.gis.db.models.GeometryField) – Geometry representing the data in this file
  • center_point (django.contrib.gis.db.models.PointField) – The center point of this file geometry
  • meta_data (django.contrib.postgres.fields.JSONField) – JSON meta-data about this file
  • countries (django.db.models.ManyToManyField of storage.models.CountryData) – List of countries represented in this file as indicated by the file’s geometry.
  • is_parsed (django.db.models.BooleanField) – Whether the source file has been parsed or not
  • parsed (django.db.models.DateTimeField) – When the source file was parsed
  • job_exe (django.db.models.ForeignKey) – The job execution that created this product
  • job (django.db.models.ForeignKey) – The job that created this product
  • job_type (django.db.models.ForeignKey) – The type of the job that created this product
  • job_output (django.db.models.CharField) – The output name from the job interface
  • recipe (django.db.models.ForeignKey) – The identifier of a recipe this file is associated with
  • recipe_node (django.db.models.CharField) – The name of the node within the recipe
  • recipe_node – The name of the job within the recipe
  • recipe_type (django.db.models.ForeignKey) – The type of recipe that the file is associated with
  • batch (django.db.models.ForeignKey) – The identifier of the batch this file is associated with
  • source_started (django.db.models.DateTimeField) – The earliest start time from all source files
  • source_ended (django.db.models.DateTimeField) – The latest end time from all source files
  • source_sensor_class (django.db.models.CharField) – The class of sensor used to produce the source file.
  • source_sensor (django.db.models.CharField) – The specific identifier of the sensor used to produce the source file.
  • source_collection (django.db.models.CharField) – The collection of the source file.
  • source_task (django.db.models.CharField) – The task that produced the source file.
  • has_been_published (django.db.models.BooleanField) – Whether this product has ever been published. A product becomes published when its job execution completes successfully. A product that has been published will appear in the API call to retrieve product updates.
  • is_published (django.db.models.BooleanField) – Whether this product is currently published. A published product has had its job execution complete successfully and has not been unpublished.
  • is_superseded (django.db.models.BooleanField) – Whether this product has been superseded by another product with the same UUID
  • published (django.db.models.DateTimeField) – When this product was published (its job execution was completed)
  • unpublished (django.db.models.DateTimeField) – When this product was unpublished
  • superseded (django.db.models.DateTimeField) – When this product was superseded
ALPHABETIZE_FIELDS = [u'file_name', u'file_type', u'media_type', u'file_path']
exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

FILE_TYPES = ((u'SOURCE', u'SOURCE'), (u'PRODUCT', u'PRODUCT'))
exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

VALID_TIME_FIELDS = [u'source', u'data', u'last_modified']
add_data_type_tag(tag)

Adds a new data type tag to the file.

Parameters:tag (string) – The data type tag to add
ancestors

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

batch

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

batch_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

center_point
countries

Accessor to the related objects manager on the forward and reverse sides of a many-to-many relation.

In the example:

class Pizza(Model):
    toppings = ManyToManyField(Topping, related_name='pizzas')

pizza.toppings and topping.pizzas are ManyToManyDescriptor instances.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

created

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

data_ended

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

data_started

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

data_type_tags

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

datasetfile_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

deleted

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

descendants

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

file_name

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

file_path

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

file_size

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

file_type

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

geometry
get_data_type_tags()

Returns the set of data type tags associated with this file

Returns:The set of data type tags
Return type:{string}
get_file_type_display(*moreargs, **morekwargs)
get_next_by_created(*moreargs, **morekwargs)
get_next_by_last_modified(*moreargs, **morekwargs)
get_previous_by_created(*moreargs, **morekwargs)
get_previous_by_last_modified(*moreargs, **morekwargs)
has_been_published

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

ingest_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

is_deleted

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

is_parsed

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

is_published

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

is_superseded

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

job

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

job_exe

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

job_exe_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

job_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

job_output

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

job_type

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

job_type_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

jobinputfile_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

last_modified

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

media_type

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

meta_data

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <storage.models.ScaleFileManager object>
parsed

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

published

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

recipe

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

recipe_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

recipe_node

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

recipe_type

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

recipe_type_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

recipeinputfile_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

set_basic_fields(file_name, file_size, media_type=None, data_type_tags=None)

Sets the basic fields for the Scale file

Parameters:
  • file_name (string) – The name of the file
  • file_size (long) – The size of the file in bytes
  • media_type (string) – The IANA media type of the file
  • data_type (set) – The set of data type tags for the file
set_countries()

Clears the countries list then recreates it from the CountryData table. If no geometry is available, this will remain empty. The country border effective date will use (in order or preference) data_started, data_ended, or created.

set_deleted()

Marks the current file as deleted and updates the corresponding fields.

source_collection

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

source_ended

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

source_sensor

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

source_sensor_class

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

source_started

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

source_task

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

superseded

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

unpublished

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

update_uuid(*args)

Computes and sets a new UUID value for this file by hashing the given arguments.

Parameters:args ([string]) – One or more input objects to hash.
Returns:The generated unique identifier.
Return type:string
url

Gets the absolute URL used to download this file.

Note that this property is only supported if the associated workspace supports HTTP downloads.

Returns:The file download URL.
Return type:string
uuid

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

workspace

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

workspace_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

class storage.models.ScaleFileManager

Bases: django.db.models.manager.Manager

Provides additional methods for handling Scale files

delete_files(files)

Deletes the given files from the remove storage system. Each ScaleFile model should have its related workspace field populated. This method will update the ScaleFile model and save the changes in the database.

Parameters:files ([storage.models.ScaleFile]) – List of files to delete

:raises storage.exceptions.ArchivedWorkspace: If one of the files has a workspace that is archived :raises storage.exceptions.MissingRemoteMount: If a required mount location is missing

download_files(file_downloads)

Downloads the given files to the given local file system paths. Each ScaleFile model should have its related workspace field populated.

Parameters:file_downloads ([storage.brokers.broker.FileDownload]) – List of files to download

:raises storage.exceptions.ArchivedWorkspace: If one of the files has a workspace that is archived :raises storage.exceptions.DeletedFile: If one of the files is deleted :raises storage.exceptions.MissingRemoteMount: If a required mount location is missing

filter_files(data_started=None, data_ended=None, created_started=None, created_ended=None, source_started=None, source_ended=None, source_sensor_classes=None, source_sensors=None, source_collections=None, source_tasks=None, mod_started=None, mod_ended=None, job_type_ids=None, job_type_names=None, job_ids=None, is_published=None, is_superseded=None, file_names=None, job_outputs=None, recipe_ids=None, recipe_type_ids=None, recipe_nodes=None, batch_ids=None, order=None, countries=None)

Returns a query for product models that filters on the given fields. The returned query includes the related workspace, job_type, and job fields, except for the workspace.json_config field. The related countries are set to be pre-fetched as part of the query.

Parameters:
  • data_started (datetime.datetime) – Query files where data started after this time.
  • data_ended (datetime.datetime) – Query files where data ended before this time.
  • created_started (datetime.datetime) – Query files created after this time.
  • created_ended (datetime.datetime) – Query files created before this time.
  • source_started (datetime.datetime) – Query files where source collection started after this time.
  • source_ended (datetime.datetime) – Query files where source collection ended before this time.
  • source_sensor_classes (list()) – Query files with the given source sensor class.
  • source_sensor (list()) – Query files with the given source sensor.
  • source_collection (list()) – Query files with the given source class.
  • source_tasks (list()) – Query files with the given source tasks.
  • mod_started (datetime.datetime) – Query files where the last modified date is after this time.
  • mod_ended (datetime.datetime) – Query files where the last modified date is before this time.
  • job_type_ids (list()) – Query files with jobs with the given type identifier.
  • job_type_names (list()) – Query files with jobs with the given type name.
  • job_ids (list()) – Query files with a given job id
  • is_published (bool) – Query files flagged as currently exposed for publication.
  • is_superseded (bool) – Query files that have/have not been superseded.
  • file_names (list()) – Query files with the given file names.
  • job_outputs (list()) – Query files with the given job outputs
  • recipe_ids (list()) – Query files with a given recipe id
  • recipe_nodes (list()) – Query files with a given recipe nodes
  • recipe_type_ids (list()) – Query files with the given recipe types
  • batch_ids (list()) – Query files with batches with the given identifiers.
  • order (list()) – A list of fields to control the sort order.
  • countries (list()) – A List of country codes (iso3) to filter by
Returns:

The product file query
Return type:

django.db.models.QuerySet
filter_files_v6(started=None, ended=None, time_field=None, file_name=None)

Returns a query for Scale files that is filtered on the given fields.

Parameters:
  • started (datetime.datetime) – Query Scale files updated after this amount of time.
  • ended (datetime.datetime) – Query Scale files updated before this amount of time.
  • time_field (string) – The time field to use for filtering.
  • file_name (str) – Query Scale files with the given file name.
Returns:

The Scale file query
Return type:

django.db.models.QuerySet
get_details(file_id)

Returns the file for the given ID with all detail fields included.

There are currently no additional fields included.

Parameters:file_id (int) – The unique identifier of the workspace.
Returns:The file with all detail fields included.
Return type:storage.models.ScaleFile
get_files_for_job_summary(file_ids)

Returns the file models with the given IDs. Each scale_file model only contains the needed fields for calculating summary data for a job’s inputs. The returned list is a queryset iterator, so only access it once.

Parameters:file_ids (list()) – The file IDs
Returns:The scale_file models that match the given IDs
Return type:list()
get_files_for_queued_jobs(file_ids)

Returns the file models with the given IDs. Each scale_file model only contains the needed fields for configuring queued jobs. The returned list is a queryset iterator, so only access it once.

Parameters:file_ids (list()) – The file IDs
Returns:The scale_file models that match the given IDs
Return type:list()
move_files(file_moves)

Moves the given files to the new file system paths. Each ScaleFile model should have its related workspace field populated. This method will update the file_path field in each ScaleFile model to the new path (it may also change other ScaleFile fields) and save the changes in the database.

Parameters:file_moves ([storage.brokers.broker.FileMove]) – List of files to move

:raises storage.exceptions.ArchivedWorkspace: If one of the files has a workspace that is archived :raises storage.exceptions.DeletedFile: If one of the files is deleted :raises storage.exceptions.MissingRemoteMount: If a required mount location is missing

upload_files(workspace, file_uploads)

Uploads the given files from the given local file system paths into the given workspace. Each ScaleFile model should have its file_path field populated with the relative location where the file should be stored within the workspace. This method will update the workspace and other fields (including possibly changing file_path) in each ScaleFile model and will save the models to the database.

Parameters:
Returns:

The list of saved file models
Return type:

[storage.models.ScaleFile]

:raises storage.exceptions.ArchivedWorkspace: If one of the files has a workspace that is archived :raises storage.exceptions.MissingRemoteMount: If a required mount location is missing

class storage.models.Workspace(*args, **kwargs)

Bases: django.db.models.base.Model

Represents a storage location where files can be stored and retrieved for processing

Parameters:
  • name (django.db.models.CharField) – The identifying name of the workspace used by clients for queries
  • title (django.db.models.CharField) – The human-readable name of the workspace
  • description (django.db.models.CharField) – An optional description of the workspace
  • base_url (str) – The base URL used to retrieve files from this workspace if supported.
  • is_active (django.db.models.BooleanField) – Whether the workspace is active (can be used and displayed)
  • json_config (django.contrib.postgres.fields.JSONField) – JSON configuration describing how to store/retrieve files for this workspace
  • is_move_enabled (django.db.models.BooleanField) – Whether the workspace allows files to be moved within it
  • created (django.db.models.DateTimeField) – When the workspace was created
  • deprecated (django.db.models.DateTimeField) – When the workspace was archived (no longer active)
  • last_modified (django.db.models.DateTimeField) – When the workspace was last modified
ALPHABETIZE_FIELDS = [u'name', u'title', u'description']
exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

base_url

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

created

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

delete_files(files, update_model=True)

Deletes the given files using the workspace’s broker and saves the ScaleFile model changes in the database. If this workspace’s broker uses a container volume, the workspace expects this volume file system to already be mounted at workspace_volume_path or an exception will be raised.

Parameters:files ([storage.models.ScaleFile]) – List of files to delete

:raises storage.exceptions.MissingVolumeMount: If the required volume mount is missing

deprecated

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

description

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

download_files(file_downloads)

Downloads the given files to the given local file system paths using the workspace’s broker. If this workspace’s broker uses a container volume, the workspace expects this volume file system to already be mounted at workspace_volume_path or an exception will be raised.

Parameters:file_downloads ([storage.brokers.broker.FileDownload]) – List of files to download

:raises storage.exceptions.MissingVolumeMount: If the required volume mount is missing

get_broker()

Returns the configured broker for this workspace

Returns:The configured broker
Return type:storage.brokers.broker.Broker
get_configuration()

Returns the workspace configuration object

Returns:The configuration in v2 of the JSON schema
Return type:dict
get_file_system_paths(files)

Returns the local file system paths for the given files, if supported by the workspace’s broker. If this workspace’s broker uses a container volume, the workspace expects this volume file system to already be mounted at workspace_volume_path or an exception will be raised.

Parameters:files ([storage.models.ScaleFile]) – List of files
Returns:The list of local file system paths if supported, None otherwise
Return type:[string]
get_next_by_created(*moreargs, **morekwargs)
get_next_by_last_modified(*moreargs, **morekwargs)
get_previous_by_created(*moreargs, **morekwargs)
get_previous_by_last_modified(*moreargs, **morekwargs)
get_v6_configuration_json()

Returns the workspace configuration in v6 of the JSON schema

Returns:The workspace configuration in v6 of the JSON schema
Return type:dict
id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

is_active

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

is_move_enabled

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

json_config

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

last_modified

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

list_files(recursive)

Lists files within a workspace, with optional full tree recursion.

Parameters:recursive (boolean) – Flag to indicate whether file searching should be done recursively
Returns:Generator of files matching given expression
Return type:Generator[storage.brokers.broker.FileDetails]
move_files(file_moves)

Moves the given files to the new file system paths and saves the ScaleFile model changes in the database. If this workspace’s broker uses a container volume, the workspace expects this volume file system to already be mounted at workspace_volume_path or an exception will be raised.

Parameters:file_moves ([storage.brokers.broker.FileMove]) – List of files to move

:raises storage.exceptions.MissingVolumeMount: If the required volume mount is missing

name

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <storage.models.WorkspaceManager object>
scalefile_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

title

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

upload_files(file_uploads)

Uploads the given files from the given local file system paths and saves the ScaleFile models in the database. If this workspace’s broker uses a container volume, the workspace expects this volume file system to already be mounted at workspace_volume_path or an exception will be raised.

Parameters:file_uploads ([storage.brokers.broker.FileUpload]) – List of files to upload

:raises storage.exceptions.MissingVolumeMount: If the required volume mount is missing

volume

If this workspace’s broker uses a container volume, this property returns the information needed to set up a volume that can be mounted into the task container. If this workspace’s broker does not use a container volume, this property should be None.

Returns:The container volume information needed for this workspace’s broker, possibly None
Return type:storage.brokers.broker.BrokerVolume
workspace_volume_path

Returns the absolute local path within the container onto which the broker’s container volume is mounted

Returns:The absolute local path within the container for the broker’s volume
Return type:string
class storage.models.WorkspaceManager

Bases: django.db.models.manager.Manager

Provides additional methods for handling workspaces.

create_workspace(*args, **kwargs)

Creates a new Workspace with the given configuration and returns the new Workspace model. The Workspace model will be saved in the database and all changes to the database will occur in an atomic transaction.

Parameters:
  • name (string) – The identifying name of this Workspace
  • title (string) – The human-readable name of this Workspace
  • description (string) – A description of this Workspace
  • configuration (storage.configuration.workspace_configuration.WorkspaceConfiguration) – The Workspace configuration
  • base_url (string) – The URL prefix used to download files stored in the Workspace.
  • is_active (bool) – Whether or not the Workspace is available for use.
Returns:

The new Workspace
Return type:

storage.models.Workspace

:raises storage.configuration.exceptions.InvalidWorkspaceConfiguration: If the configuration is invalid

edit_workspace(*args, **kwargs)

Edits the given Workspace and saves the changes in the database. All database changes occur in an atomic transaction. An argument of None for a field indicates that the field should not change.

Parameters:
  • workspace_id (int) – The unique identifier of the Workspace to edit
  • title (string) – The human-readable name of this Workspace
  • description (string) – A description of this Workspace
  • configuration (storage.configuration.workspace_configuration.WorkspaceConfiguration) – The Workspace configuration
  • base_url (string) – The URL prefix used to download files stored in the Workspace.
  • is_active (bool) – Whether or not the Workspace is available for use.

:raises storage.configuration.exceptions.InvalidWorkspaceConfiguration: If the configuration is invalid

get_details(workspace_id, is_staff=False)

Returns the workspace for the given ID with all detail fields included.

There are currently no additional fields included.

Parameters:
  • workspace_id (int) – The unique identifier of the workspace.
  • is_staff (bool) – Whether the requesting user is a staff member
Returns:

The workspace with all detail fields included.
Return type:

storage.models.Workspace
get_workspaces(started=None, ended=None, names=None, order=None)

Returns a list of workspaces within the given time range.

Parameters:
  • started (datetime.datetime) – Query workspaces updated after this amount of time.
  • ended (datetime.datetime) – Query workspaces updated before this amount of time.
  • names ([string]) – Query workspaces with the given name.
  • order ([string]) – A list of fields to control the sort order.
Returns:

The list of workspaces that match the time range.
Return type:

[storage.models.Workspace]
validate_workspace_v6(name, configuration)

Validates a new workspace prior to attempting a save

Parameters:
  • name (string) – The identifying name of a Workspace to validate
  • configuration (dict) – The Workspace configuration
Returns:

The workspace validation.
Return type:

storage.models.WorkspaceValidation
class storage.models.WorkspaceValidation(is_valid, errors, warnings)

Bases: tuple

errors

Alias for field number 1

is_valid

Alias for field number 0

warnings

Alias for field number 2

storage.move_files_job module

Defines the functions necessary to move a file to a different workspace/uri

storage.move_files_job.move_files(file_ids, new_workspace=None, new_file_path=None)

Moves the given files to a different workspace/uri

Parameters:
  • file_ids ([int]) – List of ids of ScaleFile objects to move; should all be from the same workspace
  • new_workspace (storage.models.Workspace) – New workspace to move files to
  • new_file_path (string) – New path for files

storage.serializers module

Defines the serializers for Scale files and workspaces

class storage.serializers.GeoJsonField(**kwargs)

Bases: rest_framework.fields.CharField

Field for displaying geometry objects as Well Known Text

to_representation(value)

Converts the model field to GeoJson

Parameters:value (GEOSGeometry) – the associated geometry info
Return type:string
Returns:the GeoJson representation
type_label = u'geojson'
type_name = u'GeoJsonField'
class storage.serializers.ScaleFileBasePlusCountriesSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: util.rest.ModelIdSerializer

Converts Scale file model fields to REST output

class storage.serializers.ScaleFileBaseSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: util.rest.ModelIdSerializer

Converts Scale file model fields to REST output

class storage.serializers.ScaleFileDetailsSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: storage.serializers.ScaleFileSerializerV6

Converts file model fields to REST output

class storage.serializers.ScaleFileSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: storage.serializers.ScaleFileBaseSerializerV6

Converts Scale file model fields to REST output

class BatchBaseSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: util.rest.ModelIdSerializer

Base serializer for batches

class JobTypeBaseSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: util.rest.ModelIdSerializer

Converts job type model fields to REST output

class RecipeTypeBaseSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: util.rest.ModelIdSerializer

Base serializer for recipe types

class storage.serializers.WktField(**kwargs)

Bases: rest_framework.fields.CharField

Field for displaying geometry objects as Well Known Text

to_representation(value)

Converts the model field to WKT

Parameters:value (GEOSGeometry) – the associated geometry info
Return type:string
Returns:the WKT representation
type_label = u'wtk'
type_name = u'WktField'
class storage.serializers.WorkspaceBaseSerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: util.rest.ModelIdSerializer

Converts workspace model fields to REST output

class storage.serializers.WorkspaceDetailsSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: storage.serializers.WorkspaceSerializerV6

Converts workspace model fields to REST output

class storage.serializers.WorkspaceSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: storage.serializers.WorkspaceBaseSerializer

Converts workspace model fields to REST output

storage.settings module

Defines all the custom settings used by this application.

storage.urls module

Defines the URLs for the RESTful storage services

storage.views module

Defines the views for the RESTful storage services

class storage.views.FileDetailsView(**kwargs)

Bases: rest_framework.generics.RetrieveAPIView

This view is the endpoint for retrieving details of a scale file.

queryset
retrieve(request, file_id)

Determine api version and call specific method

Parameters:
  • request (rest_framework.request.Request) – the HTTP POST request
  • file_id (int encoded as a string) – The id of the file
Return type:

rest_framework.response.Response
Returns:

the HTTP response to send back to the user
retrieve_impl(request, file_id)

Retrieves the details for a file and return them in JSON form

Parameters:
  • request (rest_framework.request.Request) – the HTTP GET request
  • file_id (int encoded as a string) – The id of the file
Return type:

rest_framework.response.Response
Returns:

the HTTP response to send back to the user
serializer_class

alias of ScaleFileDetailsSerializerV6

class storage.views.FilesView(**kwargs)

Bases: rest_framework.generics.ListAPIView

This view is the endpoint for retrieving source/product files

get_serializer_class()

Returns the appropriate serializer based off the requests version of the REST API

list(request)

Retrieves the batches and returns them in JSON form

Parameters:request (rest_framework.request.Request) – the HTTP GET request
Return type:rest_framework.response.Response
Returns:the HTTP response to send back to the user
queryset
class storage.views.PurgeSourceFileView(**kwargs)

Bases: rest_framework.views.APIView

This view is the endpoint for submitting a source file ID to be purged

post(request)

Kicks off the process of purging a given source file from Scale

Parameters:request (rest_framework.request.Request) – the HTTP POST request
Return type:rest_framework.response.Response
Returns:the HTTP response to send back to the user
class storage.views.WorkspaceDetailsView(**kwargs)

Bases: rest_framework.generics.GenericAPIView

This view is the endpoint for retrieving/updating details of a workspace.

get(request, workspace_id)

Retrieves the details for a workspace and return them in JSON form

Parameters:
  • request (rest_framework.request.Request) – the HTTP GET request
  • workspace_id (int encoded as a str) – The id of the workspace
Return type:

rest_framework.response.Response
Returns:

the HTTP response to send back to the user
get_serializer_class()

Returns the appropriate serializer based off the requests version of the REST API

patch(request, workspace_id)

Edits an existing workspace and returns the updated details

Parameters:
  • request (rest_framework.request.Request) – the HTTP GET request
  • workspace_id (int encoded as a str) – The id of the workspace
Return type:

rest_framework.response.Response
Returns:

the HTTP response to send back to the user
queryset
class storage.views.WorkspacesValidationView(**kwargs)

Bases: rest_framework.views.APIView

This view is the endpoint for validating a new workspace before attempting to actually create it

post(request)

Validates a new workspace and returns any warnings discovered

Parameters:request (rest_framework.request.Request) – the HTTP POST request
Return type:rest_framework.response.Response
Returns:the HTTP response to send back to the user
queryset
class storage.views.WorkspacesView(**kwargs)

Bases: rest_framework.generics.ListCreateAPIView

This view is the endpoint for retrieving the list of all workspaces.

create(request)

Creates a new Workspace and returns it in JSON form

Parameters:request (rest_framework.request.Request) – the HTTP POST request
Return type:rest_framework.response.Response
Returns:the HTTP response to send back to the user
get_serializer_class()

Returns the appropriate serializer based off the requests version of the REST API

list(request)

Retrieves the list of all workspaces and returns it in JSON form

Parameters:request (rest_framework.request.Request) – the HTTP GET request
Return type:rest_framework.response.Response
Returns:the HTTP response to send back to the user
queryset

Module contents