pipelines.models module#

class pipelines.models.OrmPipelineExecution(*args, **kwargs)#

Bases: PipelineExecution, Model

Model to store pipeline execution status in the django ORM

exception DoesNotExist#: Bases: ObjectDoesNotExist

exception MultipleObjectsReturned#: Bases: MultipleObjectsReturned

completed#: The date and time when the run finishes (whether successful or not)

get_pipeline() → Pipeline#: Returns the registered pipeline class

get_pipeline_results() → Sequence[PipelineResult]#: Returns all pipeline results associated with this execution from the database

get_status()#: Returns the current status of the pipeline

get_status_display(*, field=<django.db.models.fields.CharField: status>)#

id#: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

input_data#: The data passed into the pipeline when ran

objects = <django.db.models.manager.ManagerFromOrmPipelineExecutionQuerySet object>#

pipeline_id#: The id of the registered pipeline class

results#

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

run_id#: The id of the run to link the execution to

started#: The date and time when the run was started

status#: The current status of the execution

class pipelines.models.OrmPipelineExecutionQuerySet(model=None, query=None, using=None, hints=None)#

Bases: QuerySet

Custom query set for managing pipeline execution objects.

with_extra_stats()#: Attaches extra properties to the pipeline execution queryset so that each of the objects conform to the results storage interface.

class pipelines.models.OrmPipelineResult(*args, **kwargs)#

Bases: PipelineResult, Model

The status of a given pipeline result

exception DoesNotExist#: Bases: ObjectDoesNotExist

exception MultipleObjectsReturned#: Bases: MultipleObjectsReturned

completed#: The date and time when the run finishes (whether successful or not)

execution#: The pipeline execution relating to the current pipeline run

execution_id#

property failed#: True if the result is in a failed state, otherwise false

get_pipeline() → Pipeline#: Returns the registered pipeline class

get_pipeline_execution() → PipelineExecution#: Returns the parent pipeline execution object

get_status() → PipelineTaskStatus#: Returns the current status of the pipeline

get_status_display(*, field=<django.db.models.fields.CharField: status>)#

get_task_executions() → Sequence[TaskExecution]#: Returns all task execution objects for this pipeline instance

id#: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

property input_data#: The data the pipeline was started with

objects = <django.db.models.manager.ManagerFromOrmPipelineResultQuerySet object>#

property pipeline_id#: The id of the registered pipeline class

pipeline_object#: The deserialized pipeline object for this pipeline instance

reporter#: The python path to the reporter the pipeline was started with

property run_id#: The id of the run for this pipeline

runner#: The python path to the runner the pipeline was started with

serializable_pipeline_object#: The object related to this results instance

started#: The time the pipeline was started

status#: The current status of this result

task_executions#

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

class pipelines.models.OrmPipelineResultQuerySet(model=None, query=None, using=None, hints=None)#

Bases: QuerySet

for_run_id(run_id)#

Fetches all pipeline results for a given run id

Parameters:: run_id – The run id to fetch all objects for.

with_extra_stats()#: Attaches extra properties to the pipeline result queryset so that each of the objects conform to the results storage interface.

class pipelines.models.OrmTaskExecution(id, pipeline_result, task_id, pipeline_task, config, status, started, completed)#

Bases: TaskExecution, Model

exception DoesNotExist#: Bases: ObjectDoesNotExist

exception MultipleObjectsReturned#: Bases: MultipleObjectsReturned

completed#: The date and time when the run finishes (whether successful or not)

config#: The configuration used to instantiate the task

get_status()#: Returns the status of the task

get_status_display(*, field=<django.db.models.fields.CharField: status>)#

get_task() → Task#: Returns the registered task class

get_task_results() → Sequence[TaskResult]#: Returns all the task results objects for this execution

id#: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

property input_data#: Returns the input data the pipeline was started with

objects = <django.db.models.manager.ManagerFromTaskExecutionQuerySet object>#

property pipeline_id#: The id of the registered pipeline class

pipeline_object#: An instance of the registered pipeline class

pipeline_result#: The pipeline result this task is linked to

pipeline_result_id#

pipeline_task#: The name of the task property on the piepline

results#

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

property run_id#: The id of the current pipeline run

property serializable_pipeline_object#: The serialized version of the object associated with the related pipeline instance

started#: The time the pipeline was started

status#: The string representation of the task status

task_id#: The id of the registered task class

class pipelines.models.OrmTaskResult(id, execution, serializable_task_object, status, started, completed)#

Bases: TaskResult, Model

exception DoesNotExist#: Bases: ObjectDoesNotExist

exception MultipleObjectsReturned#: Bases: MultipleObjectsReturned

completed#: The date and time when the run finishes (whether successful or not)

property config#: The configuration used to instantiate the task

property duration#: The time taken for the task to finish

execution#: The task execution object the result is linked to

execution_id#

get_status()#: Returns the status of the task

get_status_display(*, field=<django.db.models.fields.CharField: status>)#

get_task()#: Returns an instance of the registered task

get_task_execution() → TaskExecution#: Returns the linked task execution

id#: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

property input_data#: Returns the input data the pipeline was started with

objects = <django.db.models.manager.ManagerFromOrmTaskResultQuerySet object>#

property pipeline_id#: The id of the registered pipeline class

pipeline_object#: The deserialized version of the object linked to the pipeline instance

property pipeline_task#: The name of the task property on the piepline

property run_id#: The id of the current pipeline run

property serializable_pipeline_object#: The serialized version of the object associated with the related pipeline instance

serializable_task_object#: The serialized version of the object associated with this task instance

started#: The time the pipeline was started

status#: The string representation of the task status

property task_id#: The id of the registered task class

task_object#: The deserialized version of the object linked to the task instance

class pipelines.models.OrmTaskResultQuerySet(model=None, query=None, using=None, hints=None)#

Bases: QuerySet

for_run_id(run_id)#

Returns a queryset containing all task results for the given run id

Parameters:: run_id – The id of the run to filter by

not_completed()#: Returns all uncompleted task executions

with_duration()#: Annotates the queryset with the time it took for all tasks to complete

class pipelines.models.PipelineLog(*args, **kwargs)#

Bases: TimeStampedModel

Model to store pipeline logs in the database

exception DoesNotExist#: Bases: ObjectDoesNotExist

exception MultipleObjectsReturned#: Bases: MultipleObjectsReturned

context_id#: The id of the object to log a message against

context_type#: The type of object to log a message against (one of PipelineExecution, PipelineResult, TaskExecution and TaskResult)

created#: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

get_context_type_display(*, field=<django.db.models.fields.CharField: context_type>)#

get_next_by_created(*, field=<django_extensions.db.fields.CreationDateTimeField: created>, is_next=True, **kwargs)#

get_next_by_modified(*, field=<django_extensions.db.fields.ModificationDateTimeField: modified>, is_next=True, **kwargs)#

get_previous_by_created(*, field=<django_extensions.db.fields.CreationDateTimeField: created>, is_next=False, **kwargs)#

get_previous_by_modified(*, field=<django_extensions.db.fields.ModificationDateTimeField: modified>, is_next=False, **kwargs)#

get_status_display(*, field=<django.db.models.fields.CharField: status>)#

id#: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

property log_message#

message#: The message to record

modified#: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <django.db.models.manager.Manager object>#

pipeline_id#: Id of the pipeline being ran

run_id#: The run_id of the object

status#: The status of the object

class pipelines.models.TaskExecutionQuerySet(model=None, query=None, using=None, hints=None)#

Bases: QuerySet

for_run_id(run_id)#

Returns a queryset containing all task executions for the given run id

Parameters:: run_id – The id of the run to filter by

not_completed()#: Returns all uncompleted task executions

with_duration()#: Annotates the queryset with the time it took for all tasks to complete