pipelines.runners package#

class pipelines.runners.PipelineRunner#

Bases: object

The base class to use when implementing a custom pipeline runner.

classmethod get_flat_task_list(pipeline_result: PipelineResult) → List[TaskExecution]#

Returns a list of all the tasks in an order they can be ran in order to conform to all runner ordering guarantees. :param pipeline_result: The pipeline result object to build the graph from

classmethod get_task_graph(pipeline_result: PipelineResult) → TopologicalSorter#

Returns a TopologicalSorter (see graphlib.TopologicalSorter) object representing the ordering of tasks in the pipeline.

Parameters:

pipeline_result – The pipeline result object to build the graph from

run(pipeline_execution: PipelineExecution, reporter: PipelineReporter)#

Runs the pipeline. This should be reimplemented by each runner to schedule the pipeline as required.

Returns True if the pipeline was scheduled successfully, False otherwise. This has no guarantee on the completion of the pipeline, just that it has been successfully scheduled.

Parameters:

  • pipeline_execution – The pipeline execution object representing the pipeline to run.

  • reporter – The reporter object to write messages to.

start(pipeline_execution: PipelineExecution, reporter: PipelineReporter) → bool#

Starts the pipeline running by the runner. This will pass the parameters off to the run method so the runner can handle the pipeline as it wishes.

Returns True if the pipeline was scheduled successfully, False otherwise. This has no guarantee on the completion of the pipeline, just that it has been successfully scheduled.

Parameters:

  • pipeline_execution – The pipeline execution object representing the pipeline to run.

  • reporter – The reporter object to write messages to.

Subpackages#

Submodules#