scheduler.resources package

Submodules

scheduler.resources.agent module

Defines the class that represents an agent’s set of resource offers

class scheduler.resources.agent.AgentResources(agent_id)

Bases: object

This class represents an agent’s set of resource offers.

allocate_offers(resources, when)

Directs the agent to allocate offers sufficient to match the given resources. Any offers that have been held too long will automatically be included. It’s possible that the offer resources returned are less than requested.

Parameters:
Returns:

The list of allocated offers
Return type:

[scheduler.resources.offer.ResourceOffer]
decline_offers()

Removes offers that haven’t been allocated and returns them for the scheduler to decline them.

Returns:list of declined offers
Return type:list()
generate_status_json(node_dict, total_running=None, total_offered=None, total_watermark=None, total=None)

Generates the portion of the status JSON that describes the resources for this agent

Parameters:
Returns:

The total number of offers this agent has
Return type:

int
get_max_resources()

Gets the maximum resources for the agent

Returns:The maximum resources for an agent
Return type:node.resources.node_resources.NodeResources
has_total_resources()

Indicates whether this agent knows its total resources or not

Returns:True if agent knows its total resources, False otherwise
Return type:bool
refresh_resources(offers, tasks)

Refreshes the agent’s resources by setting the current running tasks and adding new resource offers. Returns a copy of the set of resources for the agent.

Parameters:
Returns:

A copy of the set of agent resources
Return type:

scheduler.resources.agent.ResourceSet
rescind_offers(offer_ids)

Rescinds the offers with the given IDs

Parameters:offer_ids ([str]) – The list of IDs of the offers to remove
reset_watermark()

Resets the agent’s watermark to the highest recent value

set_shortage(shortage_resources=None)

Sets the resource shortage for the agent, if any

Parameters:shortage_resources (node.resources.node_resources.NodeResources) – The resource shortage
set_total(total_resources)

Sets the total resources for the agent

Parameters:total_resources (node.resources.node_resources.NodeResources) – The total resources
class scheduler.resources.agent.ResourceSet(offered_resources=None, task_resources=None, watermark_resources=None)

Bases: object

This class represents a set of resources on an agent

scheduler.resources.manager module

Defines the class that manages all scheduler resources

class scheduler.resources.manager.ResourceManager

Bases: object

This class manages all resources from the cluster nodes. This class is thread-safe.

add_new_offers(offers)

Adds new resource offers to the manager

Parameters:offers ([scheduler.resources.offer.ResourceOffer]) – The list of new offers to add
allocate_offers(resources, when)

Directs all agents to allocate offers sufficient to match the given resources. Any offers that have been held too long will automatically be included (including agents that resources were not requested from). It’s possible that the offer resources returned for an agent are less than requested or that an agent is not included in the results.

Parameters:
  • resources (dict) – Dict where agent ID maps to the requested resources
  • when (datetime.datetime) – The current time
Returns:

Dict where agent ID maps to a list of the allocated offers
Return type:

dict
clear()

Clears all offer data from the manager. This method is intended for testing only.

decline_offers()

Directs all agents to remove offers that have not been allocated and return them to be declined by the scheduler.

Returns:list of offers to decline
Return type:list()
generate_status_json(status_dict)

Generates the portion of the status JSON that describes the resources

Parameters:status_dict (dict) – The status JSON dict
get_all_available_resources()

Gets the total of all of the available resources across all agents

Returns:A copy of these resources
Return type:dict
get_max_available_resources()

Gets the maximum available resources across all agents

Returns:A copy of these resources
Return type:node.resources.node_resources.NodeResources
get_queued_resources(statuses=[u'PENDING', u'QUEUED', u'RUNNING'])

Gets the total cluster resources, how many jobs are queued, and how much total resources the queued jobs require. Includes information for pending, queued, and running.

Parameters:statuses – The list of possible statuses a job could have that should be included.
Returns:A dictionary of information about jobs queued.
Return type:dict
lost_agent(agent_id)

Informs the manager that the agent with the given ID was lost and has gone offline

Parameters:agent_id (str) – The ID of the lost agent
refresh_agent_resources(tasks, when)

Refreshes the agents with the current tasks that are running on them and with the new resource offers that have been added to the manager since the last time this method was called. Returns a dict containing all of the current offered resources and watermark resources for each agent.

Parameters:
Returns:

Dict where agent ID maps to a copy of the set of resources for the agent
Return type:

dict
rescind_offers(offer_ids)

Rescinds the offers with the given IDs from the manager

Parameters:offer_ids (list()) – The list of IDs of the offers to rescind
set_agent_shortages(agent_shortages)

Sets any resource shortages on the appropriate agents

Parameters:agent_shortages (dict) – Dict where resource shortage is stored by agent ID
sync_with_mesos(host_address)

Syncs with Mesos to retrieve the resource totals needed by any agents

Parameters:host_address (util.host.HostAddress) – The address for the Mesos master
update_all_cluster_resources()

Updates the values for the currently available resources that are stored in the database.

scheduler.resources.offer module

Defines the class that represents resource offers

class scheduler.resources.offer.ResourceOffer(offer_id, agent_id, framework_id, node_resources, when, mesos_offer)

Bases: object

This class represents an offer of resources from a node.

Module contents