pipelines.runners.celery.runner module

class pipelines.runners.celery.runner.Runner

Bases: PipelineRunner

classmethod build_celery_task(task: TaskResult)

Converts the task result into a celery signature. A signature is added to catch errors and record the status change on the results object. If the task config has a celery_queue property the task will be assigned to that queue.

Parameters:

task – The task result that represents the task to convert.

classmethod expand_celery_tasks(task: TaskExecution) → signature

Builds a celery chord or canvas based on the task execution. After all tasks instances are ran a task will be ran to update the task execution status.

If the task has an iterable a chord will be generated to run each task instance in parallel. If there is no iterator a chain will be built.

Parameters:

task – The task execution object to expand.

expand_pipeline_execution(pipeline_execution: PipelineExecution)

Expands the pipeline execution into a celery canvas. After all pipeline instances are ran a task will be ran to update the pipeline execution status.

If the pipeline has an iterable a chord will be generated to run each pipeline instance in parallel. If there is no iterator a chain will be built.

Parameters:

pipeline_execution – The pipeline execution object to expand.

expand_pipeline_result(pipeline_result: PipelineResult)

Expands the pipeline into a celery canvas. Currently this builds a chain running each expanded task in series but will be updated to schedule tasks in parallel where possible.

Parameters:

pipeline_result – The pipeline instance to expand

run(pipeline_execution: PipelineExecution, reporter: PipelineReporter) → bool

Creates a celery canvas representing the pipeline and schedules it with the celery broker.

Returns True if the pipeline was scheduled successfully, False otherwise. This has no guarantee on the completion of the pipeline, just that it has been successfully scheduled.

Parameters:

• pipeline_execution – The pipeline execution object representing the pipeline to run.

• reporter – The reporter object to write messages to.