Results#
When a task changes state it’s new state and any status message should be recorded in the results storage.
There are 4 types of object in the result storage:
Pipeline Executions: This records the overall state of a pipeline being ran. There will only be one of these per run.
Pipeline Results: This records the state of a single pipeline instance within a pipeline execution along with any pipeline object it has been started with. There will be one per run, per object in the pipeline iterator.
Task Execution: This records the overall state of a task inside a pipeline result. Per run there will be on per task, per pipeline result.
Task Result: This records the state of a task instance within a task execution along with any task object it was started with. Per run there will be one per object in a task iterator for each task.
Storage Utilities#
Rather than interacting with the storage directly you should use the set
of utility methods defined in pipelines.results.helpers
.
This module provides a set of methods for creating and fetching result
objects from the storage and will work for all storage classes provided
they follow the defined interface.
Custom Storage Class#
To define you own custom storage class you must extend the base storage
class (pipelines.results.base.PipelineStorage
) and each
object returned will need to extend the relevant storage object (
pipelines.results.base.PipelineExecution
,
pipelines.results.base.PipelineResult
,
pipelines.results.base.TaskExecution
and
pipelines.results.base.TaskResult
).
When accessing properties from storage objects, a get_
method
should be used. Similarly for setting values a set_
method
should be used, this allows for the maximum flexibility in how the data
is stored and retrieved.
For convenience, is a get_
method is not implemented for a given
property, it will fallback to retrieving the property by name on the
object. For example, if we try to use get_foo
to retrieve the
foo
property the object will:
First try to find a method called
get_foo
and use that.If it doesnt exist but the object has a property called
foo
it will return a method that returnsfoo
.If neither exist an
AttributeError
will be raised as normal.
Similarly, for set_foo
methods will set foo
on the object
unless it’s otherwise defined.
For an example of how to build a custom storage class see
pipelines.results.orm.OrmPipelineResultsStorage
.
To override the storage class used, change the value of
PIPELINES_PIPELINES_RESULTS_STORAGE
in your settings file
("pipelines.results.orm.OrmPipelineResultsStorage"
by default).
This can either be a string where the value matches the python path to the
class to use or a 2-tuple where the first element matches the python path to
the class to use and the second element is a dictionary of keyword arguments
to pass to the storage class when initialised.