scheduler package

Subpackages

• scheduler.cleanup package
  • Submodules
  • scheduler.cleanup.manager module
  • scheduler.cleanup.node module
  • Module contents
• scheduler.database package
  • Submodules
  • scheduler.database.updater module
  • Module contents
• scheduler.dependencies package
  • Submodules
  • scheduler.dependencies.manager module
  • Module contents
• scheduler.management package
  • Subpackages
    • scheduler.management.commands package
      • Submodules
      • scheduler.management.commands.load_all_data module
      • scheduler.management.commands.scale_db_update module
      • scheduler.management.commands.scale_scheduler module
      • Module contents
  • Module contents
• scheduler.messages package
  • Submodules
  • scheduler.messages.restart_scheduler module
  • Module contents
• scheduler.migrations package
  • Submodules
  • scheduler.migrations.0001_initial module
  • scheduler.migrations.0002_auto_20151007_1352 module
  • scheduler.migrations.0003_auto_20160201_0846 module
  • scheduler.migrations.0004_auto_20170209_1616 module
  • scheduler.migrations.0005_scheduler_queue_mode module
  • scheduler.migrations.0006_scheduler_status module
  • scheduler.migrations.0007_scheduler_num_message_handlers module
  • scheduler.migrations.0008_auto_20171103_1334 module
  • scheduler.migrations.0009_scheduler_debugging_level module
  • scheduler.migrations.0010_scheduler_resource_level module
  • scheduler.migrations.0011_auto_20180517_1559 module
  • scheduler.migrations.0012_remove_scheduler_resource_level module
  • scheduler.migrations.0013_auto_20181220_2014 module
  • scheduler.migrations.0014_clusterresources module
  • scheduler.migrations.0015_auto_20200812_1929 module
  • Module contents
• scheduler.node package
  • Submodules
  • scheduler.node.agent module
  • scheduler.node.conditions module
  • scheduler.node.manager module
  • scheduler.node.node_class module
  • Module contents
• scheduler.recon package
  • Submodules
  • scheduler.recon.manager module
  • Module contents
• scheduler.resources package
  • Submodules
  • scheduler.resources.agent module
  • scheduler.resources.manager module
  • scheduler.resources.offer module
  • Module contents
• scheduler.scheduling package
  • Submodules
  • scheduler.scheduling.manager module
  • scheduler.scheduling.scheduling_node module
  • Module contents
• scheduler.sync package
  • Submodules
  • scheduler.sync.job_type_manager module
  • scheduler.sync.workspace_manager module
  • Module contents
• scheduler.task package
  • Submodules
  • scheduler.task.manager module
  • Module contents
• scheduler.tasks package
  • Subpackages
    • scheduler.tasks.services package
      • Subpackages
        • scheduler.tasks.services.messaging package
          • Submodules
          • scheduler.tasks.services.messaging.message_handler_task module
          • scheduler.tasks.services.messaging.messaging_service module
          • Module contents
      • Submodules
      • scheduler.tasks.services.service module
      • Module contents
  • Submodules
  • scheduler.tasks.db_update_task module
  • scheduler.tasks.manager module
  • scheduler.tasks.system_task module
  • Module contents
• scheduler.test package
  • Subpackages
    • scheduler.test.database package
      • Submodules
      • scheduler.test.database.test_updater module
      • Module contents
    • scheduler.test.dependencies package
      • Submodules
      • scheduler.test.dependencies.test_manager module
      • Module contents
    • scheduler.test.messages package
      • Submodules
      • scheduler.test.messages.test_restart_scheduler module
      • Module contents
    • scheduler.test.node package
      • Submodules
      • scheduler.test.node.test_conditions module
      • scheduler.test.node.test_manager module
      • scheduler.test.node.test_node_class module
      • Module contents
    • scheduler.test.resources package
      • Submodules
      • scheduler.test.resources.test_manager module
      • Module contents
    • scheduler.test.scheduling package
      • Submodules
      • scheduler.test.scheduling.test_manager module
      • scheduler.test.scheduling.test_node module
      • Module contents
    • scheduler.test.sync package
      • Submodules
      • scheduler.test.sync.test_job_type_manager module
      • scheduler.test.sync.test_scheduler_manager module
      • scheduler.test.sync.test_workspace_manager module
      • Module contents
    • scheduler.test.tasks package
      • Subpackages
        • scheduler.test.tasks.services package
          • Subpackages
            • scheduler.test.tasks.services.messaging package
              • Submodules
              • scheduler.test.tasks.services.messaging.test_messaging_service module
              • Module contents
          • Module contents
      • Submodules
      • scheduler.test.tasks.test_manager module
      • Module contents
  • Submodules
  • scheduler.test.test_initialize module
  • scheduler.test.test_scheduler module
  • scheduler.test.test_views module
  • Module contents
• scheduler.threads package
  • Submodules
  • scheduler.threads.base_thread module
  • scheduler.threads.messaging module
  • scheduler.threads.recon module
  • scheduler.threads.schedule module
  • scheduler.threads.scheduler_status module
  • scheduler.threads.sync module
  • scheduler.threads.task_handling module
  • scheduler.threads.task_update module
  • Module contents
• scheduler.vault package
  • Submodules
  • scheduler.vault.manager module
  • Module contents

Submodules

scheduler.admin module

class scheduler.admin.SchedulerAdmin(model, admin_site)

Bases: django.contrib.admin.options.ModelAdmin

list_display = (u'is_paused', u'id')

media

scheduler.apps module

class scheduler.apps.SchedulerConfig(app_name, app_module)

Bases: django.apps.config.AppConfig

Configuration for the scheduler app

label = u'scheduler'

name = u'scheduler'

ready()

Registers components related to the scheduler

verbose_name = u'Scheduler'

scheduler.cluster_utils module

scheduler.cluster_utils.wait_for_leader(zk_url, my_id, func, *args, **kargs)

scheduler.configuration module

Defines the class represents the scheduler configuration

class scheduler.configuration.SchedulerConfiguration(scheduler=None)

Bases: object

This class represents the scheduler configuration

scheduler.initialize module

Provides initialization functionality for the Scale system

scheduler.initialize.initialize_system()

Performs any necessary functions needed for initializing Scale

scheduler.manager module

Defines the class that manages high-level scheduler configuration and metrics

class scheduler.manager.SchedulerManager

Bases: object

This class manages the high-level scheduler configuration and metrics. It is a combination of data from both the Scale database and the Mesos master. This class is thread-safe.

PAUSED = SchedulerState(state=u'PAUSED', title=u'Paused', description=u'Scheduler is paused, so no new jobs will be scheduled. Existing jobs will continue to run.')

READY = SchedulerState(state=u'READY', title=u'Ready', description=u'Scheduler is ready to run new jobs.')

add_new_offer_count(new_offer_count)

Add count from a group of newly received offers

Parameters:new_offer_count (int) – The number of new offers received

add_scheduling_counts(job_launch_count, task_launch_count, offer_launch_count)

Add metric counts from a round of scheduling

Parameters:

• job_launch_count (int) – The number of new job executions launched
• task_launch_count (int) – The number of tasks launched
• offer_launch_count (int) – The number of offers accepted in the launch

add_task_update_counts(was_task_finished, was_job_finished)

Add metric counts from a new task update

Parameters:

• was_task_finished (bool) – Whether the task was finished (terminal task update)
• was_job_finished (bool) – Whether a job execution was finished (terminal state)

generate_status_json(status_dict)

Generates the portion of the status JSON that describes the scheduler settings and metrics

Parameters:status_dict (dict) – The status JSON dict

is_warning_active(warning, description=None)

Indicates that the given warning is now active

Parameters:

• warning (scheduler.node.conditions.NodeWarning) – The node warning
• description (string) – An optional specific description for the warning

paused_desc = u'Scheduler is paused, so no new jobs will be scheduled. Existing jobs will continue to run.'

sync_with_database()

Syncs with the database to retrieve an updated scheduler model

update_from_mesos(framework_id=None, mesos_address=None)

Updates the scheduler information from Mesos

Parameters:

• framework_id (string) – The framework ID of the scheduler
• mesos_address (util.host.HostAddress) – The address of the Mesos master

warning_active(warning, description=None)

Indicates that the given warning is now active

Parameters:

• warning (scheduler.node.conditions.NodeWarning) – The node warning
• description (string) – An optional specific description for the warning

warning_inactive(warning)

Indicates that the given warning is now inactive

Parameters:warning (scheduler.node.conditions.NodeWarning) – The node warning

class scheduler.manager.SchedulerState(state, title, description)

Bases: tuple

description

Alias for field number 2

state

Alias for field number 0

title

Alias for field number 1

class scheduler.manager.SchedulerWarning(name, title, description)

Bases: tuple

description

Alias for field number 2

name

Alias for field number 0

title

Alias for field number 1

scheduler.models module

class scheduler.models.ClusterResources(*args, **kwargs)

Bases: django.db.models.base.Model

Represents the total cluster resources.

Parameters:

• mem (django.db.models.FloatField) – The total amount of memory
• gpus – The total number of gpus
• disk – The total amount of disk space
• cpus – The total number of cpus

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

cpus

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

disk

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

gpus

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

mem

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <django.db.models.manager.Manager object>

class scheduler.models.Scheduler(*args, **kwargs)

Bases: django.db.models.base.Model

Represents a scheduler instance. There should only be a single instance of this and it’s used for storing cluster-wide state related to scheduling in Mesos.

Parameters:

• is_paused (django.db.models.BooleanField()) – True if the entire cluster is currently paused and should not accept new jobs
• num_message_handlers (django.db.models.IntegerField) – The number of message handlers to have scheduled
• system_logging_level (django.db.models.CharField) – The logging level for all scale system components

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

QUEUE_MODES = ((u'FIFO', u'FIFO'), (u'LIFO', u'LIFO'))

get_queue_mode_display(*moreargs, **morekwargs)

id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

is_paused

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

num_message_handlers

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <scheduler.models.SchedulerManager object>

queue_mode

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

status

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

system_logging_level

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

class scheduler.models.SchedulerManager

Bases: django.db.models.manager.Manager

Provides additional methods for handling scheduler db entry

get_master()

Gets the current master scheduler instance for the cluster.

Returns:The master scheduler. Return type:scheduler.models.Scheduler

initialize_scheduler()

Initializes the scheduler table by creating a model if one does not already exist

is_master_active()

Checks whether the current master scheduler is ready to schedule.

Returns:True if the master scheduler is not registered or not paused. Return type:bool

update_scheduler(new_data)

Update the data for the scheduler.

Parameters:new_data (dict) – Updated data for the node

scheduler.scale_scheduler module

The Scale Mesos scheduler

class scheduler.scale_scheduler.ScaleScheduler

Bases: object

Mesos scheduler for the Scale framework

NORMAL_WARN_THRESHOLD = datetime.timedelta(0, 0, 5000)

disconnected(message)

Invoked when the scheduler becomes disconnected from the master, e.g. the master fails and another is taking over.

error(message)

Invoked when there is an unrecoverable error in the scheduler or scheduler driver. The driver will be aborted BEFORE invoking this callback.

initialize()

Initializes the scheduler and gets it ready to connect to Mesos. This method should only ever be called once.

offers(offers)

Invoked when resources have been offered to this framework. A single offer will only contain resources from a single agent. Resources associated with an offer will not be re-offered to _this_ framework until either (a) this framework has rejected those resources or (b) those resources have been rescinded. Note that resources may be concurrently offered to more than one framework at a time (depending on the allocator being used). In that case, the first framework to launch tasks using those resources will be able to use them while the other frameworks will have those resources rescinded (or if a framework has already launched tasks with those resources then those tasks will fail with a TASK_LOST status and a message saying as much).

reconnected(message)

Invoked when the scheduler re-registers with a newly elected Mesos master. This is only called when the scheduler has previously been registered.

rescind(offer)

Invoked when an offer is no longer valid (e.g., the slave was lost or another framework used resources in the offer.) If for whatever reason an offer is never rescinded (e.g., dropped message, failing over framework, etc.), a framework that attempts to launch tasks using an invalid offer will receive TASK_LOST status updates for those tasks.

run(client)

Launch scheduler with callbacks for Mesos events.

Parameters:client (mesoshttp.client.MesosClient) – MesosClient object with

shutdown()

Performs any clean up required by this scheduler implementation.

Currently this method just notifies any background threads to break out of their work loops.

subscribed(driver)

Invoked when the scheduler successfully registers with a Mesos master. It is called with a SchedulerDriver which has attributes that include the frameworkId, and the mesos_url, which indicates the master currently leading.

update(status)

Invoked when the status of a task has changed (e.g., a slave is lost and so the task is lost, a task finishes and an executor sends a status update saying so, etc.) Note that returning from this callback acknowledges receipt of this status update. If for whatever reason the scheduler aborts during this callback (or the process exits) another status update will be delivered. Note, however, that this is currently not true if the slave sending the status update is lost or fails during that time.

scheduler.serializers module

Defines the serializers for schedulers

class scheduler.serializers.SchedulerSerializerV6(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: rest_framework.serializers.ModelSerializer

V6 Serializer for the scheduler

class Meta

Bases: object

Meta class used to define what is serialized and how

fields = (u'is_paused', u'num_message_handlers', u'system_logging_level', u'queue_mode')

model

alias of Scheduler

scheduler.urls module

Defines the URLs for the RESTful node services

scheduler.views module

Scheduler Views

class scheduler.views.QueuedResourcesView(**kwargs)

Bases: rest_framework.generics.GenericAPIView

This view is the endpoint for viewing information about resources in queued and running.

get(request)

Gets various information about resources queued.

Parameters:request (rest_framework.request.Request) – the HTTP GET request Return type:rest_framework.response.Response Returns:the HTTP response to send back to the user

get_v6(request)

Gets various information about resources queued for a v6 request

Parameters:request (rest_framework.request.Request) – the HTTP GET request Return type:rest_framework.response.Response Returns:the HTTP response to send back to the user

class scheduler.views.SchedulerView(**kwargs)

Bases: rest_framework.generics.GenericAPIView

This view is the endpoint for viewing and modifying the scheduler

get(request)

Gets scheduler information

Parameters:request (rest_framework.request.Request) – the HTTP GET request Return type:rest_framework.response.Response Returns:the HTTP response to send back to the user

get_serializer_class()

Returns the appropriate serializer based off the requests version of the REST API

get_v6(request)

Gets v6 scheduler info

Parameters:request (rest_framework.request.Request) – the HTTP GET request Return type:rest_framework.response.Response Returns:the HTTP response to send back to the user

patch(request)

Modify scheduler info with a subset of fields

Parameters:request (rest_framework.request.Request) – the HTTP PATCH request Return type:rest_framework.response.Response Returns:the HTTP response to send back to the user

patch_v6(request)

Modify v6 scheduler info with a subset of fields

Parameters:request (rest_framework.request.Request) – the HTTP PATCH request Return type:rest_framework.response.Response Returns:the HTTP response to send back to the user

queryset

update_fields = (u'is_paused', u'num_message_handlers', u'system_logging_level', u'queue_mode')

class scheduler.views.StatusView(**kwargs)

Bases: rest_framework.generics.GenericAPIView

This view is the endpoint for viewing overall system information

STATUS_FRESHNESS_THRESHOLD = 12.0

get(request)

Gets high level status information

Parameters:request (rest_framework.request.Request) – the HTTP GET request Return type:rest_framework.response.Response Returns:the HTTP response to send back to the user

get_v6(request)

The v6 version to get high level status information

Parameters:request (rest_framework.request.Request) – the HTTP GET request Return type:rest_framework.response.Response Returns:the HTTP response to send back to the user

class scheduler.views.VersionView(**kwargs)

Bases: rest_framework.generics.GenericAPIView

This view is the endpoint for viewing version/build information

get(request)

Gets various version/build information

Parameters:request (rest_framework.request.Request) – the HTTP GET request Return type:rest_framework.response.Response Returns:the HTTP response to send back to the user

get_v6(request)

Gets various version/build information for a v6 request

Parameters:request (rest_framework.request.Request) – the HTTP GET request Return type:rest_framework.response.Response Returns:the HTTP response to send back to the user

permission_classes = (<class 'rest_framework.permissions.AllowAny'>,)

Module contents