Stay organized with collections
Save and categorize content based on your preferences.
When Vertex AI Pipelines runs a pipeline, it checks to see whether or not
an execution exists in Vertex ML Metadata with the interface
(cache key) of each pipeline step.
The step's interface is defined as the combination of the following:
The pipeline step's inputs. These inputs include the input
parameters' value (if any) and the input artifact ID (if any).
The pipeline step's output definition. This output definition
includes output parameter definition (name, if any) and output artifact
definition (name, if any).
The component's specification. This specification includes the
image, commands, arguments and environment variables being used, as well
as the order of the
commands and arguments.
Additionally, only the pipelines with the same pipeline name will share the
cache.
If there is a matching execution in Vertex ML Metadata, the outputs of
that execution are used and the step is skipped. This helps to reduce costs by
skipping computations that were completed in a previous pipeline run.
You can turn off execution caching at task level by setting the following:
eval_task.set_caching_options(False)
You can turn off execution caching for an entire pipeline job. When you run
a pipeline using PipelineJob(), you can use the enable_caching argument to
specify that this pipeline run does not use caching. All steps within the
pipeline job won't use caching.
Learn more about creating pipeline runs.
Use the following sample to turn off caching:
pl=PipelineJob(display_name="My first pipeline",# Whether or not to enable caching# True = enable the current run to use caching results from previous runs# False = disable the current run's use of caching results from previous runs# None = defer to cache option for each pipeline component in the pipeline definitionenable_caching=False,# Local or Cloud Storage path to a compiled pipeline definitiontemplate_path="pipeline.yaml",# Dictionary containing input parameters for your pipelineparameter_values=parameter_values,# Cloud Storage path to act as the pipeline rootpipeline_root=pipeline_root,)
The following limitations apply to this feature:
The cached result doesn't have a time to live (TTL), and can be reused
as long as the entry is not deleted from the
Vertex ML Metadata. If the entry is deleted from
Vertex ML Metadata, the task will rerun to regenerate the
result again.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-25 UTC."],[],[],null,["# Configure execution caching\n\nWhen Vertex AI Pipelines runs a pipeline, it checks to see whether or not\nan *execution* exists in Vertex ML Metadata with the interface\n(cache key) of each pipeline step.\n\nThe step's interface is defined as the combination of the following:\n\n1. The **pipeline step's inputs**. These inputs include the input\n parameters' value (if any) and the input artifact ID (if any).\n\n2. The **pipeline step's output definition**. This output definition\n includes output parameter definition (name, if any) and output artifact\n definition (name, if any).\n\n3. The **component's specification**. This specification includes the\n image, commands, arguments and environment variables being used, as well\n as the order of the\n commands and arguments.\n\nAdditionally, only the pipelines with the same pipeline name will share the\ncache.\n\nIf there is a matching execution in Vertex ML Metadata, the outputs of\nthat execution are used and the step is skipped. This helps to reduce costs by\nskipping computations that were completed in a previous pipeline run.\n\nYou can turn off execution caching at task level by setting the following: \n\n eval_task.set_caching_options(False)\n\nYou can turn off execution caching for an entire pipeline job. When you run\na pipeline using `PipelineJob()`, you can use the `enable_caching` argument to\nspecify that this pipeline run does not use caching. All steps within the\npipeline job won't use caching.\n[Learn more about creating pipeline runs](/vertex-ai/docs/pipelines/run-pipeline).\n\nUse the following sample to turn off caching: \n\n pl = PipelineJob(\n display_name=\"My first pipeline\",\n\n # Whether or not to enable caching\n # True = enable the current run to use caching results from previous runs\n # False = disable the current run's use of caching results from previous runs\n # None = defer to cache option for each pipeline component in the pipeline definition\n enable_caching=False,\n\n # Local or Cloud Storage path to a compiled pipeline definition\n template_path=\"pipeline.yaml\",\n\n # Dictionary containing input parameters for your pipeline\n parameter_values=parameter_values,\n\n # Cloud Storage path to act as the pipeline root\n pipeline_root=pipeline_root,\n )\n\n| **Important:** Pipeline components should be built to be deterministic. A given set of inputs should always produce the same output. Depending on their interface, non-deterministic pipeline components can be unexpectedly skipped due to execution caching.\n\nThe following limitations apply to this feature:\n\n- The cached result doesn't have a time to live (TTL), and can be reused as long as the entry is not deleted from the Vertex ML Metadata. If the entry is deleted from Vertex ML Metadata, the task will rerun to regenerate the result again."]]
