scheduler.scheduling package

Submodules

scheduler.scheduling.manager module

Defines the class that manages all scheduling

class scheduler.scheduling.manager.SchedulingManager

Bases: object

This class manages all scheduling. This class is NOT thread-safe and should only be used within the scheduling thread.

perform_scheduling(client, when)

Organizes and analyzes the cluster resources, schedules new job executions, and launches tasks

Parameters:
  • client (mesoshttp.client.MesosClient) – The Mesos scheduler client
  • when (datetime.datetime) – The current time
Returns:

The number of tasks that were scheduled
Return type:

int

scheduler.scheduling.scheduling_node module

Defines the class that manages scheduling for a node

class scheduler.scheduling.scheduling_node.SchedulingNode(agent_id, node, tasks, running_job_exes, resource_set)

Bases: object

This class manages scheduling for a node.

accept_job_exe_next_task(job_exe, waiting_tasks)

Asks the node if it can accept the next task for the given job execution. If the next task is waiting on resources, the task is added to the given waiting list. This should be used for job executions that have already been scheduled on this node, not new job executions.

Parameters:
Returns:

True if waiting tasks were added to the list, False otherwise
Return type:

bool
accept_new_job_exe(job_exe)

Asks the node if it can accept the given new job execution

Parameters:job_exe (queue.job_exe.QueuedJobExecution) – The new job execution
Returns:True if the new job execution was accepted, False otherwise
Return type:bool
accept_node_tasks(when, waiting_tasks)

Asks the node to accept any node tasks that need to be scheduled. If any node tasks are waiting on resources, those tasks are added to the given waiting list.

Parameters:
  • when (datetime.datetime) – The current time
  • waiting_tasks ([job.tasks.base_task.Task]) – List of tasks that are waiting for resources
Returns:

True if waiting tasks were added to the list, False otherwise
Return type:

bool
accept_system_task(system_task)

Asks the node if it can accept the given system task

Parameters:system_task (job.tasks.base_task.Task) – The system task
Returns:True if the system task was accepted, False otherwise
Return type:bool
add_allocated_offers(offers)

Adds the resource offers that have been allocated to run this node’s tasks. If the offer resources are not enough to cover the current allocation, job executions and tasks are removed as necessary.

Parameters:offers (list()) – The resource offers to add
add_scheduled_job_exes(job_exes)

Hands the node its queued job executions that have now been scheduled in the database and are now running

Parameters:job_exes (list()) – The running job executions that have now been scheduled in the database
reset_new_job_exes()

Resets the allocated new job executions and deallocates any resources associated with them

score_job_exe_for_reservation(job_exe, job_type_resources)

Returns an integer score (lower is better) indicating how well this node is a fit for reserving (temporarily blocking additional job executions of lower priority) for the given job execution. If the job execution cannot reserve this node, None is returned.

Parameters:
  • job_exe (queue.job_exe.QueuedJobExecution) – The job execution to score
  • job_type_resources (list()) – The list of all of the job type resource requirements
Returns:

The integer score indicating how good of a fit reserving this node is for this job execution, possibly None
Return type:

int
score_job_exe_for_scheduling(job_exe, job_type_resources)

Returns an integer score (lower is better) indicating how well the given job execution fits on this node for scheduling. If the job execution cannot be scheduled on this node, None is returned.

Parameters:
  • job_exe (queue.job_exe.QueuedJobExecution) – The job execution to score
  • job_type_resources (list()) – The list of all of the job type resource requirements
Returns:

The integer score indicating how good of a fit this job execution is for this node, possibly None
Return type:

int
score_system_task_for_scheduling(system_task, job_type_resources)

Returns an integer score (lower is better) indicating how well the given system task fits on this node for scheduling. If the system task cannot be scheduled on this node, None is returned.

Parameters:
  • system_task (job.tasks.base_task.Task) – The system task to score
  • job_type_resources (list()) – The list of all of the job type resource requirements
Returns:

The integer score indicating how good of a fit this system task is for this node, possibly None
Return type:

int
start_job_exe_tasks()

Tells the node to start the next task on all scheduled job executions

Module contents