storage.brokers package

Submodules

storage.brokers.broker module

Defines the base broker class

class storage.brokers.broker.Broker(broker_type)

Bases: object

Abstract class for a broker that can download and upload files for a given storage backend

broker_type

The type of this broker

Returns:The broker type Return type:string

delete_files(volume_path, files, update_model=True)

Deletes the given files.

If this broker uses a container volume, volume_path will contain the absolute local container location where that volume file system is mounted. This means that the path to where a ScaleFile currently exists is the result of os.path.join(volume_path, files[i].file_path). If this broker does not use a container volume, None will be given for volume_path.

The files list contains the ScaleFile models representing the files to be deleted. The broker should only delete each file itself and not any parent directories. If the update model flag is set each file model should be updated and saved when a delete is successful, including fields such as is_deleted and deleted.

Parameters:

• volume_path (string) – Absolute path to the local container location onto which the volume file system was mounted, None if this broker does not use a container volume
• files ([storage.models.ScaleFile]) – List of files to delete
• update_model (bool) – Flag to determine if ScaleFile model should be updated after delete

download_files(volume_path, file_downloads)

Downloads the given files to the given local file system paths.

If this broker uses a container volume, volume_path will contain the absolute local container location where that volume file system is mounted. This means that the path to a ScaleFile that is accessible to the container is the result of os.path.join(volume_path, file_downloads[i].file.file_path). If this broker does not use a container volume, None will be given for volume_path.

The file_downloads list contains named tuples that each contain a ScaleFile model to be downloaded and the absolute local container path where the file should be downloaded. Typically, no changes are needed to file models during a download, but any changes should be saved by the broker. Any directories in the absolute local container paths should already exist.

If a file does not exist in its expected location, raise a MissingFile exception.

Parameters:

• volume_path (string) – Absolute path to the local container location onto which the volume file system was mounted, None if this broker does not use a container volume
• file_downloads ([storage.brokers.broker.FileDownload]) – List of files to download

:raises storage.exceptions.MissingFile: If a file to download does not exist at the expected path

get_file_system_paths(volume_path, files)

Returns the local file system paths for the given files, if supported by the broker.

If this broker uses a container volume, volume_path will contain the absolute local container location where that volume file system is mounted. This means that the path to a ScaleFile that is accessible to the container is the result of os.path.join(volume_path, scale_files[i].file_path). If this broker does not use a container volume, None will be given for volume_path. If this method is not supported by the broker, None will be returned.

Parameters:

• volume_path (string) – Absolute path to the local container location onto which the volume file system was mounted, None if this broker does not use a container volume
• files ([storage.models.ScaleFile]) – List of files

Returns:

The list of local file system paths if supported, None otherwise

Return type:

[string]

list_files(volume_path, recursive)

List the files under the given file system paths.

If this broker uses a container volume, volume_path will contain the absolute local container location where that volume file system is mounted. If this broker does not use a container volume, None will be given for volume_path.

Retrieval of objects is provided by the boto3 paginator over list_objects. This allows for simple paging support with unbounded object counts. As a result of the time that may be required for the full result set to be returned, the results are returned via a generator. This generator will contain objects of type storage.brokers.broker.FileDetails.

Parameters:

• volume_path (string) – Absolute path to the local container location onto which the volume file system was mounted, None if this broker does not use a container volume
• recursive (boolean) – Flag to indicate whether file searching should be done recursively

Returns:

Generator of files matching given expression

Return type:

Generator[storage.brokers.broker.FileDetails]

load_configuration(config)

Loads the given configuration

Parameters:config (dict) – The configuration as a dictionary

move_files(volume_path, file_moves)

Moves the given files to the new file system paths.

If this broker uses a container volume, volume_path will contain the absolute local container location where that volume file system is mounted. This means that the path to where a ScaleFile currently exists is the result of os.path.join(volume_path, files_moves[i].file.file_path) and the new path is given by os.path.join(volume_path, files_moves[i].file.new_path). If this broker does not use a container volume, None will be given for volume_path.

The file_moves list contains named tuples that each contain a ScaleFile model to be moved and the new relative file_path field for the new location of the file. The broker is expected to set the file_path field of each ScaleFile model to its new location (which the broker may alter) and is free to alter any additional fields as necessary. The broker is responsible for saving any changes to models when a move is successful The directories in the new file_path may not exist, so it is the responsibility of the broker to create them if necessary.

If a file does not exist in its expected location, raise a MissingFile exception.

Parameters:

• volume_path (string) – Absolute path to the local container location onto which the volume file system was mounted, None if this broker does not use a container volume
• file_moves ([storage.brokers.broker.FileMove]) – List of files to move

:raises storage.exceptions.MissingFile: If a file to move does not exist at the expected path

upload_files(volume_path, file_uploads)

Uploads the given files from the given local file system paths.

If this broker uses a container volume, volume_path will contain the absolute local container location where that volume file system is mounted. This means that the path to where a ScaleFile should be uploaded is the result of os.path.join(volume_path, file_uploads[i].file.file_path). If this broker does not use a container volume, None will be given for volume_path.

The file_uploads list contains named tuples that each contain a ScaleFile model to be uploaded and the absolute local container path where the file currently exists. The broker is free to alter the ScaleFile fields of the uploaded files, including the final file_path (the given file_path is a recommendation by Scale that guarantees path uniqueness). The ScaleFile models may not have been saved to the database yet and so may not have their id field populated. The broker should perform a model save/update to the database for any files that are successfully uploaded. The directories in the remote file_path may not exist, so it is the responsibility of the broker to create them if necessary.

Parameters:

• volume_path (string) – Absolute path to the local container location onto which the volume file system was mounted, None if this broker does not use a container volume
• file_uploads ([storage.brokers.broker.FileUpload]) – List of files to upload

validate_configuration(config)

Validates the given configuration

Parameters:config (dict) – The configuration as a dictionary Returns:A list of warnings discovered during validation. Return type:[storage.configuration.workspace_configuration.ValidationWarning]

:raises storage.brokers.exceptions.InvalidBrokerConfiguration: If the given configuration is invalid

volume

If this broker uses a container volume, this property returns the information needed to set up a volume that can be mounted into the task container. If this broker does not use a container volume, this property should be None.

Returns:The container volume information needed for this broker, possibly None Return type:storage.brokers.broker.BrokerVolume

class storage.brokers.broker.BrokerVolume(driver, remote_path)

Bases: object

Represents the properties of a container volume that must be mounted into the container for a broker to work

class storage.brokers.broker.FileDetails(file, size)

Bases: tuple

file

Alias for field number 0

size

Alias for field number 1

class storage.brokers.broker.FileDownload(file, local_path, partial)

Bases: tuple

file

Alias for field number 0

local_path

Alias for field number 1

partial

Alias for field number 2

class storage.brokers.broker.FileMove(file, new_path)

Bases: tuple

file

Alias for field number 0

new_path

Alias for field number 1

class storage.brokers.broker.FileUpload(file, local_path)

Bases: tuple

file

Alias for field number 0

local_path

Alias for field number 1

storage.brokers.exceptions module

Defines the exceptions related to workspace brokers

exception storage.brokers.exceptions.InvalidBrokerConfiguration(name, description)

Bases: storage.configuration.exceptions.InvalidWorkspaceConfiguration

Exception indicating that a broker configuration was invalid

storage.brokers.factory module

Defines the factory for creating brokers

storage.brokers.factory.add_broker_type(broker_class)

Registers a broker class so it can be used for storage operations.

Parameters:broker_class (class:storage.brokers.broker.Broker) – The class definition for a broker.

storage.brokers.factory.get_broker(broker_type)

Returns a broker of the given type.

Parameters:broker_type (string) – The unique identifier of a registered broker. Returns:A broker for storing and retrieving files. Return type:storage.brokers.broker.Broker

storage.brokers.factory.get_broker_types()

Returns a list of type identifiers for all registered brokers.

Returns:A list of broker type identifiers. Return type:[string]

storage.brokers.host_broker module

Defines a broker that mounts a local host directory into the task container as its backend storage

class storage.brokers.host_broker.HostBroker

Bases: storage.brokers.broker.Broker

Broker that utilizes a local host path mounted into the task container

delete_files(volume_path, files, update_model=True)

See storage.brokers.broker.Broker.delete_files()

download_files(volume_path, file_downloads)

See storage.brokers.broker.Broker.download_files()

get_file_system_paths(volume_path, files)

See storage.brokers.broker.Broker.get_file_system_paths()

list_files(volume_path, recursive)

See storage.brokers.broker.Broker.list_files()

load_configuration(config)

See storage.brokers.broker.Broker.load_configuration()

move_files(volume_path, file_moves)

See storage.brokers.broker.Broker.move_files()

upload_files(volume_path, file_uploads)

See storage.brokers.broker.Broker.upload_files()

validate_configuration(config)

See storage.brokers.broker.Broker.validate_configuration()

storage.brokers.nfs_broker module

Defines an NFS broker that utilizes a network file system as its backend storage

class storage.brokers.nfs_broker.NfsBroker

Bases: storage.brokers.broker.Broker

Broker that utilizes the docker-volume-netshare plugin (https://github.com/gondor/docker-volume-netshare) to mount an NFS volume into the task container

delete_files(volume_path, files, update_model=True)

See storage.brokers.broker.Broker.delete_files()

download_files(volume_path, file_downloads)

See storage.brokers.broker.Broker.download_files()

get_file_system_paths(volume_path, files)

See storage.brokers.broker.Broker.get_file_system_paths()

load_configuration(config)

See storage.brokers.broker.Broker.load_configuration()

move_files(volume_path, file_moves)

See storage.brokers.broker.Broker.move_files()

upload_files(volume_path, file_uploads)

See storage.brokers.broker.Broker.upload_files()

validate_configuration(config)

See storage.brokers.broker.Broker.validate_configuration()

storage.brokers.s3_broker module

Defines an S3 broker that utilizes the Amazon Web Services Simple Storage Service (S3) as its storage system

class storage.brokers.s3_broker.S3Broker

Bases: storage.brokers.broker.Broker

Broker that utilizes the AWS Boto library to read/write files to S3 cloud storage.

delete_files(volume_path, files, update_model=True)

See storage.brokers.broker.Broker.delete_files()

download_files(volume_path, file_downloads)

See storage.brokers.broker.Broker.download_files()

list_files(volume_path, recursive)

See storage.brokers.broker.Broker.list_files()

load_configuration(config)

See storage.brokers.broker.Broker.load_configuration()

move_files(volume_path, file_moves)

See storage.brokers.broker.Broker.move_files()

upload_files(volume_path, file_uploads)

See storage.brokers.broker.Broker.upload_files()

validate_configuration(config)

See storage.brokers.broker.Broker.validate_configuration()

Module contents