queue package

Submodules

queue.apps module

Defines the application configuration for the queue application

class queue.apps.QueueConfig(app_name, app_module)

Bases: django.apps.config.AppConfig

Configuration for the queue app

label = u'queue'
name = u'queue'
ready()

Registers the job load metrics processor with the clock system.

verbose_name = u'Queue'

queue.job_exe module

Defines the class that represents queued job executions being considered for scheduling

class queue.job_exe.QueuedJobExecution(queue)

Bases: object

This class represents a queued job execution that is being considered for scheduling

create_job_exe_model(framework_id, when)

Creates and returns a scheduled job execution model

Parameters:
  • framework_id (string) – The scheduling framework ID
  • when (datetime.datetime) – The start time
Returns:

The job execution model
Return type:

job.models.JobExecution
scheduled(agent_id, node_id, resources)

Indicates that this job execution has been scheduled on a node and passes the agent, node, and resource information

Parameters:

queue.job_load module

Defines the clock event processor for tracking historical job load.

class queue.job_load.JobLoadProcessor

Bases: job.clock.ClockEventProcessor

This class queries and stores job load statistics for tracking and trending.

process_event(event, last_event=None)

See job.clock.ClockEventProcessor.process_event().

Calculates metrics for the job load over time.

queue.models module

Defines the database model for a queue entry

class queue.models.JobLoad(*args, **kwargs)

Bases: django.db.models.base.Model

Represents the load counts for each job type at various points in time.

Parameters:
  • job_type (django.db.models.ForeignKey) – The type of job being measured.
  • measured (django.db.models.DateTimeField) – When the counts were actually measured.
  • pending_count (django.db.models.IntegerField) – The number of jobs in pending status for the type.
  • queued_count (django.db.models.IntegerField) – The number of jobs in queued status for the type.
  • running_count (django.db.models.IntegerField) – The number of jobs in running status for the type.
  • total_count (django.db.models.IntegerField) – The number of jobs in pending, queued, or running status for the type.
exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

get_next_by_measured(*moreargs, **morekwargs)
get_previous_by_measured(*moreargs, **morekwargs)
id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

job_type

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

job_type_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

measured

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <queue.models.JobLoadManager object>
pending_count

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

queued_count

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

running_count

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

total_count

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

class queue.models.JobLoadGroup(time, pending_count=0, queued_count=0, running_count=0)

Bases: object

Represents a group of job load models.

Parameters:
  • time (datetime.datetime) – When the counts were actually measured.
  • pending_count (int) – The number of jobs in pending status for the type.
  • queued_count (int) – The number of jobs in queued status for the type.
  • running_count (int) – The number of jobs in running status for the type.
class queue.models.JobLoadManager

Bases: django.db.models.manager.Manager

This class manages the JobLoad model.

calculate(*args, **kwargs)

Calculates and saves new job load models grouped by job type based on a current jobs snapshot.

get_job_loads(started=None, ended=None, job_type_ids=None, job_type_names=None, order=None)

Returns a list of job loads within the given time range.

Parameters:
  • started (datetime.datetime) – Query jobs updated after this amount of time.
  • ended (datetime.datetime) – Query jobs updated before this amount of time.
  • job_type_ids ([int]) – Query jobs of the type associated with the identifier.
  • job_type_names ([string]) – Query jobs of the type associated with the name.
  • order ([string]) – A list of fields to control the sort order.
Returns:

The list of job loads that match the time range.
Return type:

list[queue.models.JobLoad]
group_by_time(job_loads)

Groups the given job loads by job type.

Parameters:job_loads (list[queue.models.JobLoad]) – Query jobs updated after this amount of time.
Returns:A list of job loads grouped by job type.
Return type:list[queue.models.JobLoadGroup]
class queue.models.Queue(*args, **kwargs)

Bases: django.db.models.base.Model

Represents a job execution that is queued and ready to be run on a node

Parameters:
  • job_type (django.db.models.ForeignKey) – The type of this job
  • job (django.db.models.ForeignKey) – The job that has been queued
  • recipe (django.db.models.ForeignKey) – The original recipe that created this job
  • batch (django.db.models.ForeignKey) – The batch that contains this job
  • exe_num (django.db.models.IntegerField) – The number for this job execution
  • input_file_size (django.db.models.FloatField) – The amount of disk space in MiB required for input files for this job
  • is_canceled (django.db.models.BooleanField) – Whether this queued job execution has been canceled
  • priority (django.db.models.IntegerField) – The priority of the job (lower number is higher priority)
  • timeout (django.db.models.IntegerField) – The maximum amount of time to allow this execution to run before being killed (in seconds)
  • interface (django.contrib.postgres.fields.JSONField) – JSON description describing the job’s interface
  • configuration (django.contrib.postgres.fields.JSONField) – JSON description describing the execution configuration for how the job should be run
  • resources (django.contrib.postgres.fields.JSONField) – JSON description describing the resources required for this job
  • created (django.db.models.DateTimeField) – When the queue model was created
  • queued (django.db.models.DateTimeField) – When the job was placed onto the queue
  • docker_image (str) – The docker image to be retrieved for job that is retrieved from job_type_rev.docker_image
exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

batch

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

batch_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

configuration

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

created

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

docker_image

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

exe_num

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

get_execution_configuration()

Returns the execution configuration for this queued job

Returns:The execution configuration for this queued job
Return type:job.execution.configuration.json.exe_config.ExecutionConfiguration
get_job_interface()

Returns the interface for this queued job

Returns:The job interface
Return type:job.configuration.interface.job_interface.JobInterface
get_next_by_created(*moreargs, **morekwargs)
get_next_by_queued(*moreargs, **morekwargs)
get_previous_by_created(*moreargs, **morekwargs)
get_previous_by_queued(*moreargs, **morekwargs)
get_resources()

Returns the resources required by this queued job

Returns:The required resources
Return type:node.resources.node_resources.NodeResources
id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

input_file_size

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

interface

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

is_canceled

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

job

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

job_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

job_type

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

job_type_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <queue.models.QueueManager object>
priority

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

queued

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

recipe

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

recipe_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

resources

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

timeout

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

class queue.models.QueueManager

Bases: django.db.models.manager.Manager

Provides additional methods for managing the queue

cancel_queued_jobs(job_ids)

Marks the queued job executions for the given jobs as canceled

Parameters:job_ids (list()) – The list of job IDs being canceled
get_queue(order_mode, ignore_job_type_ids=None)

Returns the list of queue models sorted according to their priority first, and then according to the provided mode

Parameters:
  • order_mode (string) – The mode determining how to order the queue (FIFO or LIFO)
  • ignore_job_type_ids (list()) – The list of job type IDs to ignore
Returns:

The list of queue models
Return type:

list[queue.models.Queue]
get_queue_status()

Returns the current status of the queue with statistics broken down by job type.

Returns:A list of each job type with calculated statistics.
Return type:list[queue.models.QueueStatus]
queue_jobs(jobs, requeue=False, priority=None)

Queues the given jobs. The caller must have obtained model locks on the job models in an atomic transaction. Any jobs that are not in a valid status for being queued, are without job input, or are superseded will be ignored.

Parameters:
  • jobs (list()) – The job models to put on the queue
  • requeue (bool) – Whether this is a re-queue (True) or a first queue (False)
  • priority (int) – An optional argument to reset the jobs’ priority when they are queued
Returns:

The list of job IDs that were successfully QUEUED
Return type:

list()
queue_new_job_for_user_v6(*args, **kwargs)

Creates a new job for the given type and data at the request of a user. The new job is immediately placed on the queue. The given job_type model must have already been saved in the database (it must have an ID). The new job, event, job_exe, and queue models are saved in the database in an atomic transaction. If the data is invalid, a job.configuration.data.exceptions.InvalidData will be thrown.

Parameters:
Returns:

The ID of the new job
Return type:

int
queue_new_job_v6(job_type, data, event, job_configuration=None)

Creates a new job for the given type and data. The new job is immediately placed on the queue. The new job, job_exe, and queue models are saved in the database in an atomic transaction.

Parameters:
Returns:

The new queued job
Return type:

job.models.Job
Raises:

job.configuration.data.exceptions.InvalidData – If the job data is invalid
queue_new_recipe_for_user_v6(*args, **kwargs)

Creates a new recipe for the given type and data at the request of a user.

The new jobs in the recipe with no dependencies on other jobs are immediately placed on the queue. The given event model must have already been saved in the database (it must have an ID). All database changes occur in an atomic transaction.

Parameters:
  • recipe_type (recipe.models.RecipeType) – The type of the new recipe to create
  • recipe_input (data.data.data.data) – The recipe data to run on, should be None if superseded_recipe is provided
Returns:

New recipe type
Return type:

recipe.models.Recipe

:raises recipe.configuration.data.exceptions.InvalidRecipeData: If the recipe data is invalid

queue_new_recipe_v6(recipe_type, recipe_input, event, ingest_event=None, recipe_config=None, batch_id=None, superseded_recipe=None)

Creates a new recipe for the given type and data. and queues any of its jobs that are ready to run. If the new recipe is superseding an old recipe, superseded_recipe, delta, and superseded_jobs must be provided and the caller must have obtained a model lock on all job models in superseded_jobs and on the superseded_recipe model. All database changes occur in an atomic transaction.

Parameters:
Returns:

New recipe type
Return type:

recipe.models.Recipe

:raises recipe.configuration.data.exceptions.InvalidRecipeData: If the recipe data is invalid :raises recipe.exceptions.InactiveRecipeType: If the recipe type is inactive

class queue.models.QueueStatus(job_type, count=0, longest_queued=None, highest_priority=100)

Bases: object

Represents queue status statistics.

Parameters:
  • job_type (job.models.JobType) – The job type being counted.
  • count (int) – The number of job executions running for the associated job type.
  • longest_queued (datetime.datetime) – The date/time of the last queued job execution for the associated job type.
  • highest_priority (int) – The priority of the most important job execution for the associated job type.

queue.serializers module

Defines the serializers for queue models

class queue.serializers.JobLoadGroupSerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: rest_framework.serializers.Serializer

Converts job load model fields to REST output

class queue.serializers.QueueStatusSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: rest_framework.serializers.Serializer

Converts queue status model fields to REST output

class JobTypeBaseSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: util.rest.ModelIdSerializer

Converts job type model fields to REST output

queue.urls module

Defines the URLs for the RESTful queue services

queue.views module

Defines the views for the RESTful queue services

class queue.views.JobLoadView(**kwargs)

Bases: rest_framework.generics.ListAPIView

This view is the endpoint for retrieving the job load for a given time range.

list(request)

Retrieves the job load for a given time range and returns it in JSON form

Parameters:request (rest_framework.request.Request) – the HTTP GET request
Return type:rest_framework.response.Response
Returns:the HTTP response to send back to the user
queryset
serializer_class

alias of JobLoadGroupSerializer

class queue.views.QueueStatusView(**kwargs)

Bases: rest_framework.generics.ListAPIView

This view is the endpoint for retrieving the queue status.

get_serializer_class()

Returns the appropriate serializer based off the requests version of the REST API.

list(request)

Retrieves the job load for a given time range and returns it in JSON form

Parameters:request (rest_framework.request.Request) – the HTTP GET request
Return type:rest_framework.response.Response
Returns:the HTTP response to send back to the user
queryset

Module contents