ingest.strike.monitors package¶

Submodules¶

ingest.strike.monitors.dir_monitor module¶

Defines a monitor that watches a file system directory for incoming files

class ingest.strike.monitors.dir_monitor.DirWatcherMonitor¶

Bases: ingest.strike.monitors.monitor.Monitor

A monitor that watches a file system directory for incoming files

load_configuration(configuration)¶: See ingest.strike.monitors.monitor.Monitor.load_configuration()

run()¶: See ingest.strike.monitors.monitor.Monitor.run()

stop()¶: See ingest.strike.monitors.monitor.Monitor.stop()

validate_configuration(configuration)¶: See ingest.strike.monitors.monitor.Monitor.validate_configuration()

ingest.strike.monitors.exceptions module¶

Defines the exceptions related to Strike monitors

exception ingest.strike.monitors.exceptions.InvalidMonitorConfiguration(description)¶

Bases: util.exceptions.ValidationException

Exception indicating that the provided strike configuration was invalid

exception ingest.strike.monitors.exceptions.S3NoDataNotificationError¶

Bases: exceptions.Exception

Error class used when there is no data associated with an S3 notification

exception ingest.strike.monitors.exceptions.SQSNotificationError¶

Bases: exceptions.Exception

Error class used when there is a problem processing an SQS S3 notification

ingest.strike.monitors.factory module¶

Defines the factory for creating monitors

ingest.strike.monitors.factory.add_monitor_type(monitor_class)¶

Registers a monitor class so it can be used for Strike

Parameters:	monitor_class (class:ingest.strike.monitors.monitor.Monitor) – The class definition for a monitor

ingest.strike.monitors.factory.get_monitor(monitor_type)¶

Returns a monitor of the given type that is set to monitor the given workspace

Parameters:	monitor_type (string) – The unique identifier of a registered monitor
Returns:	A monitor for storing and retrieving files.
Return type:	`ingest.strike.monitors.monitor.Monitor`

ingest.strike.monitors.factory.get_monitor_types()¶

Returns a list of type identifiers for all registered monitors

Returns:	A list of monitor types
Return type:	[string]

ingest.strike.monitors.monitor module¶

Defines the base monitor class

class ingest.strike.monitors.monitor.Monitor(monitor_type, supported_broker_types)¶

Bases: object

Abstract class for a monitor that processes incoming files to ingest. Sub-classes must have a no-argument constructor that passes in the correct monitor type and supported broker types and should override the load_configuration(), run(), stop(), and validate_configuration() methods.

load_configuration(configuration)¶

Loads the given configuration

Parameters:	configuration (dict) – The configuration as a dictionary

monitor_type¶

The type of this monitor

Returns:	The monitor type
Return type:	string

reload_configuration()¶: Reloads the configuration for this monitor from the database

run()¶

Runs the monitor until signaled to stop by the stop() method. Sub-classes that override this method should make it block until the stop() method is called and should call reload_configuration() on a regular basis to get updated configuration from the database.

Sub-classes should call Ingest.objects.create_ingest() when they detect a new file in the monitored workspace. If the sub-class is tracking transfer time (the amount of time it takes for the file to be copied into the monitored workspace), it should call _start_transfer(), _update_transfer() as updates occur, and finally _complete_transfer() and _process_ingest() when the transfer is complete. If the sub-class is not tracking transfer time, it should just call _process_ingest().

setup_workspaces(monitored_workspace, file_handler)¶

Sets up the workspaces that will be used by this monitor

Parameters:	monitored_workspace (string) – The name of the workspace that is being monitored file_handler (`ingest.strike.handlers.file_handler.FileHandler`) – The file handler configured for this monitor

stop()¶: Signals the monitor to stop running. Sub-classes that override this method should make it stop the run() call as gracefully and quickly as possible.

supported_broker_types¶

The broker types supported by this monitor

Returns:	The broker types supported by this monitor
Return type:	[string]

validate_configuration(configuration)¶

Validates the given configuration

Parameters:	configuration (dict) – The configuration as a dictionary
Returns:	A list of warnings discovered during validation
Return type:	[`ingest.strike.configuration.strike_configuration.ValidationWarning`]