data package

Subpackages

Submodules

data.apps module

Defines the application configuration for the data application

class data.apps.DataConfig(app_name, app_module)

Bases: django.apps.config.AppConfig

Configuration for the data app

label = u'data'
name = u'data'
verbose_name = u'Data'

data.exceptions module

Defines exceptions that can occur when interacting with datasets, datasetmembers, and datasetfiles

exception data.exceptions.InvalidDataSet(name, description)

Bases: util.exceptions.ValidationException

Exception indicating the dataset is invalid

exception data.exceptions.InvalidDataSetDefinition(name, description)

Bases: util.exceptions.ValidationException

Exception indicating that a dataset definition was given an invalid value

exception data.exceptions.InvalidDataSetMember(name, description)

Bases: util.exceptions.ValidationException

Exception indicating the dataset is invalid

data.models module

Defines the database models for datasets

class data.models.DataSet(*args, **kwargs)

Bases: django.db.models.base.Model

Represents a DataSet object

Parameters:
  • name (django.db.models.CharField) – The identifying name of the dataset used by clients for queries
  • version (django.db.models.CharField) – The version of the dataset
  • version_array (list()) – The version of the dataset split into SemVer integer components (major,minor,patch,prerelease)
  • title (django.db.models.CharField) – The human-readable title of this dataset (optional)
  • description (django.db.models.CharField) – The description of the dataset (optional)
  • created (django.db.models.DateTimeField) – Defines the created time of the dataset
  • definition (class:django.contrib.postgres.fields.JSONField) – Defines the dataset
ALPHABETIZE_FIELDS = [u'title', u'description']
exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

created

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

datasetfile_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

datasetmember_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

definition

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

description

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

get_dataset_definition()

Returns the dataset definition

Returns:The dataset definition json
Return type:dict
get_dataset_files_json()

Returns the JSON for the associated dataset files

Returns:Returns the outgoing primitive representation.
Return type:dict?
get_dataset_members_json()

Returns the JSON for the associated dataset members

Returns:Returns the outgoing primitive representation.
Return type:dict?
get_definition()

Returns the dataset definition

Returns:The DataSet definition
Return type:data.dataset.dataset.DataSetDefinition
get_next_by_created(*moreargs, **morekwargs)
get_previous_by_created(*moreargs, **morekwargs)
get_v6_definition_json()

Returns the dataset definition in v6 of the JSON schema

Returns:The dataset definition in v6 of the JSON schema
Return type:dict
id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <data.models.DataSetManager object>
title

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

class data.models.DataSetFile(*args, **kwargs)

Bases: django.db.models.base.Model

The actual file in a dataset member

Parameters:
  • dataset (django.db.models.ForeignKey) – Refers to the dataset the file is a member of
  • scale_file (django.db.models.ForeignKey) – Refers to the ScaleFile
  • parameter_name (django.db.models.CharField) – Refers to the File parameter name
exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

dataset

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

dataset_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <data.models.DataSetFileManager object>
parameter_name

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

scale_file

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

scale_file_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

class data.models.DataSetFileManager

Bases: django.db.models.manager.Manager

Manages the datasetfile model

create_dataset_files(dataset, data, existing_scale_ids)

Creates dataset files for the given dataset and data

get_dataset_files(dataset_id)

Returns the dataset files associated with the given dataset_id

Parameters:dataset_id (integer) – The id of the associated dataset
Returns:The DataSetFiles associated with that dataset_id
Return type:[data.models.DataSetFile]
get_dataset_ids(file_ids, all_files=False)

Returns a list of the dataset IDs that contain the given files

Parameters:
  • file_ids – The ids of the files to look for
  • all_files (bool) – Whether or not a dataset must contain all files or just some of the files in the list
Returns:

The list of dataset IDs
Return type:

list()
get_datasets(file_ids, all_files=False)

Returns the datasets associated with the given file_id

Parameters:
  • file_id (integer) – The id of the associated file
  • all_files (bool) – Whether or not a dataset must contain all files or just some of the files in the list
Returns:

The DataSets associated with that dataset_id
Return type:

[data.models.DataSet]
get_file_ids(dataset_ids, parameter_names=None)

Returns a list of the file IDs for the given datasets, optionally filtered by parameter_name.

Parameters:
  • dataset_ids (integer) – The ids of the associated datasets
  • parameter_names (string) – The parameter names to search for in the given datasets
Returns:

The list of scale file IDs
Return type:

list()
get_files(dataset_ids, parameter_names=None)

Returns the dataset files associated with the given dataset_ids

Parameters:
  • dataset_ids (integer) – The ids of the associated datasets
  • parameter_names (string) – The parameter names to search for in the given datasets
Returns:

The DataSetFiles associated with that dataset_id
Return type:

[data.models.DataSetFile]
class data.models.DataSetManager

Bases: django.db.models.manager.Manager

Provides additional methods for handling datasets

create_dataset_v6(definition, title=None, description=None)

Creates and returns a new dataset for the given name/title/description/definition/version??

Parameters:
  • definition (data.dataset.dataset.DataSetDefinition) – Parameter definition of the dataset
  • title (string) – Optional title of the dataset
  • description (string) – Optional description of the dataset
Returns:

The new dataset
Return type:

data.models.DataSet

:raises data.exceptions.InvalidDataSet: If a give dataset has an invalid value

filter_datasets(started=None, ended=None, dataset_ids=None, keywords=None, order=None)

Returns a query for dataset models that filters on the given fields

Parameters:
  • started (datetime.datetime) – Query datasets created after this amount of time.
  • ended (datetime.datetime) – Query datasets created before this amount of time.
  • dataset_ids (list()) – Query datasets assciated with the given id(s)
  • keywords (list()) – Query datasets with title or description matching one of the specified keywords
  • order (list()) – A list of fields to control the sort order.
Returns:

The dataset query
Return type:

django.db.models.QuerySet
get_dataset_files(dataset_id)

Returns the files associated with the given dataset

Returns:The list of DataSetFiles matching the file_id
Return type:[data.models.DataSetFile]
get_dataset_members(dataset_id)

Returns the members associated with the given dataset_id :returns: The list of DataSetMembers :rtype: [data.models.DataSetMember]

get_datasets_v6(started=None, ended=None, dataset_ids=None, keywords=None, order=None)

Handles retrieving datasets - possibly filtered and ordered

Returns:The list of datasets that match the given filters
Return type:[data.models.DataSet]
get_details_v6(dataset_id)

Gets additional details for the given dataset id

Returns:The full dataset for the given id
Return type:data.models.DataSet
validate_dataset_v6(definition, title=None, description=None)

Validates the given dataset definiton

Parameters:definition (dict) – The dataset definition
Returns:The dataset validation
Return type:datset.models.DataSetValidation
class data.models.DataSetMember(*args, **kwargs)

Bases: django.db.models.base.Model

Defines the data of a dataset? contains list/descriptors of DataFiles

Parameters:
  • dataset (django.db.models.ForeignKey) – Refers to dataset member belongs to
  • data – JSON description of the data in this DataSetMember.
  • created (datetime) – Created Time
exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

created

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

data

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

dataset

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

dataset_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

file_ids

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

get_data()

Returns the data for this datasetmember

Returns:The data for this datasetmember
Return type:data.data.data.Data
get_dataset_definition()

Returns the dataset definition

Returns:The dataset definition
Return type:data.dataset.dataset.DataSetDefinition
get_next_by_created(*moreargs, **morekwargs)
get_previous_by_created(*moreargs, **morekwargs)
get_v6_data_json()

Returns the data for this datasetmember as v6 json with the version stripped

Returns:The v6 JSON output data dict for this datasetmember
Return type:dict
id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <data.models.DataSetMemberManager object>
class data.models.DataSetMemberManager

Bases: django.db.models.manager.Manager

Provides additional methods for handling dataset members

build_data_list(template, data_started=None, data_ended=None, created_started=None, created_ended=None, source_started=None, source_ended=None, source_sensor_classes=None, source_sensors=None, source_collections=None, source_tasks=None, mod_started=None, mod_ended=None, job_type_ids=None, job_type_names=None, job_ids=None, is_published=None, is_superseded=None, file_names=None, job_outputs=None, recipe_ids=None, recipe_type_ids=None, recipe_nodes=None, batch_ids=None, order=None)

Builds a list of data dictionaries from a template and file filters

Parameters:
  • template (dict) – The template to fill with files found through filters
  • data_started (datetime.datetime) – Query files where data started after this time.
  • data_ended (datetime.datetime) – Query files where data ended before this time.
  • created_started (datetime.datetime) – Query files created after this time.
  • created_ended (datetime.datetime) – Query files created before this time.
  • source_started (datetime.datetime) – Query files where source collection started after this time.
  • source_ended (datetime.datetime) – Query files where source collection ended before this time.
  • source_sensor_classes (list()) – Query files with the given source sensor class.
  • source_sensor (list()) – Query files with the given source sensor.
  • source_collection (list()) – Query files with the given source class.
  • source_tasks (list()) – Query files with the given source tasks.
  • mod_started (datetime.datetime) – Query files where the last modified date is after this time.
  • mod_ended (datetime.datetime) – Query files where the last modified date is before this time.
  • job_type_ids (list()) – Query files with jobs with the given type identifier.
  • job_type_names (list()) – Query files with jobs with the given type name.
  • job_ids (list()) – Query files with a given job id
  • is_published (bool) – Query files flagged as currently exposed for publication.
  • is_superseded (bool) – Query files that have/have not been superseded.
  • file_names (list()) – Query files with the given file names.
  • job_outputs (list()) – Query files with the given job outputs
  • recipe_ids (list()) – Query files with a given recipe id
  • recipe_nodes (list()) – Query files with a given recipe nodes
  • recipe_type_ids (list()) – Query files with the given recipe types
  • batch_ids (list()) – Query files with batches with the given identifiers.
  • order (list()) – A list of fields to control the sort order.
create_dataset_members(dataset, data_list)

Creates a dataset member

Parameters:
get_dataset_members(dataset)

Returns dataset members for the given dataset

Returns:members for a given dataset
Return type:QuerySet<DataSetMember>
get_details_v6(dsm_id)

Gets additional details for the given dataset member id

Returns:The full dataset member for the given id
Return type:data.models.DataSetMember
validate_data_list(dataset_def, data_list)

Validates a list of data objects against a dataset

Parameters:
  • dataset_def – The dataset definition the member is a part of
  • data_list ([data.data.data.Data]) – Data definitions of the dataset members
class data.models.DataSetValidation(is_valid, errors, warnings)

Bases: tuple

errors

Alias for field number 1

is_valid

Alias for field number 0

warnings

Alias for field number 2

data.serializers module

Defines the serializers for jobs and job types

class data.serializers.DataSetBaseSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: util.rest.ModelIdSerializer

Converts dataset model fields to REST output

class data.serializers.DataSetDetailsSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: data.serializers.DataSetBaseSerializerV6

Converts dataset model fields to REST output

class data.serializers.DataSetFileBaseSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: util.rest.ModelIdSerializer

Converts the datasetfile model fields to REST output

class data.serializers.DataSetFileSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: data.serializers.DataSetFileBaseSerializerV6

Converts the datasetfile model fields to REST output

class data.serializers.DataSetListSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: data.serializers.DataSetBaseSerializerV6

Converts dataset model fields to REST output

class data.serializers.DataSetMemberDetailsSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: data.serializers.DataSetMemberSerializerV6

class data.serializers.DataSetMemberSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: util.rest.ModelIdSerializer

Converts datasetmember model fields to REST output

data.urls module

Defines the URLs for the RESTful dataset services

data.views module

class data.views.DataSetDetailsView(**kwargs)

Bases: rest_framework.generics.GenericAPIView

This view is the endpoint for retrieving details of a specific dataset

get(request, dataset_id)

Retrieves the details for a data set and return them in JSON form

Parameters:
  • request (rest_framework.request.Request) – the HTTP GET request
  • dataset_id (int encoded as a str) – The dataset id
Return type:

rest_framework.response.Response
Returns:

the HTTP response to send back to the user
get_v6(request, dataset_id)

Retrieves the details for a dataset version and return them in JSON form

Parameters:
  • request (rest_framework.request.Request) – the HTTP GET request
  • dataset_id (int encoded as a str) – The dataset id
Return type:

rest_framework.response.Response
Returns:

the HTTP response to send back to the user
post(request, dataset_id)

Adds a datsetmember to the dataset

Parameters:
  • request (rest_framework.request.Request) – the HTTP request
  • dataset_id (int encoded as a str) – The id of the dataset
Return type:

rest_framework.response.Response
Returns:

the HTTP response to send back to the user
post_v6(request, dataset_id)

Adds a datsetmember to the dataset

Parameters:
  • request (rest_framework.request.Request) – the HTTP request
  • dataset_id (int encoded as a str) – The id of the dataset
Return type:

rest_framework.response.Response
Returns:

the HTTP response to send back to the user
queryset
serializer_class

alias of DataSetDetailsSerializerV6

class data.views.DataSetMemberDetailsView(**kwargs)

Bases: rest_framework.generics.GenericAPIView

This view is the endpoint for retrieving details of a specific dataset member

get(request, dsm_id)

Retrieves the details for a data set and return them in JSON form

Parameters:
  • request (rest_framework.request.Request) – the HTTP GET request
  • dsm_id (int encoded as a str) – The dataset member id
Return type:

rest_framework.response.Response
Returns:

the HTTP response to send back to the user
get_v6(request, dsm_id)

Retrieves the details for a dataset member and return them in JSON form

Parameters:
  • request (rest_framework.request.Request) – the HTTP GET request
  • dsm_id (int encoded as a str) – The dataset member id
Return type:

rest_framework.response.Response
Returns:

the HTTP response to send back to the user
queryset
serializer_class

alias of DataSetMemberDetailsSerializerV6

class data.views.DataSetMembersView(**kwargs)

Bases: rest_framework.generics.ListAPIView

This view is the endpoint for retrieving members of a specific dataset

list(request, dataset_id)

Retrieves the details for a data set and return them in JSON form

Parameters:
  • request (rest_framework.request.Request) – the HTTP GET request
  • dataset_id (int encoded as a str) – The dataset id
Return type:

rest_framework.response.Response
Returns:

the HTTP response to send back to the user
list_v6(request, dataset_id)

Retrieves the members for a dataset version and return them in JSON form

Parameters:
  • request (rest_framework.request.Request) – the HTTP GET request
  • dataset_id (int encoded as a str) – The dataset id
Return type:

rest_framework.response.Response
Returns:

the HTTP response to send back to the user
queryset
serializer_class

alias of DataSetMemberSerializerV6

class data.views.DataSetValidationView(**kwargs)

Bases: rest_framework.views.APIView

This view is the endpoint for validating a new dataset before attempting to create it

post(request)

Validates a new dataset and returns any errors/warnings discovered

Parameters:request (rest_framework.request.Request) – the HTTP POST request
Return type:rest_framework.response.Response
Returns:the HTTP response to send back to the user
post_v6(request)

Validates a new dataset and returns any warnings discovered

Parameters:request (rest_framework.request.Request) – the HTTP POST request
Return type:rest_framework.response.Response
Returns:the HTTP response to send back to the user
queryset
class data.views.DataSetView(**kwargs)

Bases: rest_framework.generics.ListCreateAPIView

This view is the endpoint for retrieving the list of all datasets.

create(request)

Creates a new dataset and returns a link to the detail URL

Parameters:request (rest_framework.request.Request) – the HTTP POST request
Return type:rest_framework.response.Response
Returns:the HTTP response to send back to the user
create_all_v6(request)

Creates or edits a dataset - including the dataset members - and returns a link to the detail URL

create_v6(request)

Creates or edits a dataset and returns a link to the detail URL

Parameters:request (rest_framework.request.Request) – the HTTP POST request
Return type:rest_framework.response.Response
Returns:the HTTP response to send back to the user
list(request)

Retrieves the list of all datasets and returns it in JSON form

Parameters:request (rest_framework.request.Request) – the HTTP GET request
Return type:rest_framework.response.Response
Returns:the HTTP response to send back to the user
list_v6(request)

Retrieves the list of all datasets and returns it in JSON form

Parameters:request (rest_framework.request.Request) – the HTTP GET request
Return type:rest_framework.response.Response
Returns:the HTTP response to send back to the user
queryset
serializer_class

alias of DataSetListSerializerV6

Module contents