Storage batch operations

This document describes storage batch operations, a Cloud Storage capability that lets you perform operations on billions of objects in a serverless manner. Using storage batch operations, you can automate large-scale API operations on billions of objects, reducing the development time required to write and maintain scripts for each request.

To learn how to create storage batch operations jobs, see Create and manage storage batch operations jobs.

Overview

Storage batch operations let you run one of four transformations on multiple objects at once: placing an object hold, deleting an object, updating object metadata, and rewriting objects. To use storage batch operations, you create a job configuration that defines what transformations should be applied to which objects.

Creating a batch operation returns a long-running operation (LRO) that indicates the status of your request: whether the transformation has been applied to all specified objects in your request.

Benefits

  • Scalability: Perform transformations on millions of objects with a single storage batch operations job.
  • Serverless execution: Run batch jobs in a serverless environment, eliminating the need to manage infrastructure.
  • Automation: Automate complex and repetitive tasks, improving operational efficiency.
  • Reduced development time: Avoid writing and maintaining complex custom scripts.
  • Performance: Complete time-sensitive operations within the required time. With multiple batch jobs running concurrently on a bucket, you can process up to one billion objects within three hours.
  • Automatic retries: Automatic retries for failed operations.
  • Job monitoring: Detailed progress tracking to monitor the status and completion of all jobs.

Use cases

When used with Storage Insights datasets, storage batch operations allow you to accomplish the following tasks:

  • Security management:

    • Set encryption keys on multiple objects using the rewrite object method.
    • Apply or remove object holds to control object immutability.

  • Compliance:

    • Use object holds to meet data retention requirements for regulatory compliance.
    • Delete data between specific timeframes, to meet wipeout compliance requirements.

  • Data transformation: Perform bulk updates to object metadata.

  • Cost optimization: Bulk delete objects in Cloud Storage buckets to reduce storage costs.

Job configurations

To create a storage batch operations job, you'll need to set the following job configurations. Job configurations are parameters that control how the job is defined for different processing requirements.

  • Job name: A unique name to identify the storage batch operations job. This is used for tracking, monitoring, and referencing the job. Job names are alphanumeric, for example, job-01.

  • Job Description (Optional): A brief description of the job's purpose. This helps with understanding and documenting the job details. For example, Deletes all objects in a bucket.

  • Bucket name: The name of the storage bucket containing the objects to be processed. This is essential for locating the input data. For example, my-bucket. You can specify only one bucket name for a job.

  • Object selection: The selection criteria that defines which objects to process. You can specify the criteria using any one of the following options:

    • Manifest: Create a manifest and specify its location when you create the storage batch operations job. The manifest is a CSV file, uploaded to Google Cloud, that contains one object or a list of objects that you want to process. Each row in the manifest must include the bucket and name of the object. You can optionally specify the generation of the object. If you don't specify the generation, the current version of the object is used.

      The file must include a header row of the following format:

      bucket,name,generation

      The following is an example of the manifest:

      bucket,name,generation
bucket_1,object_1,generation_1
bucket_1,object_2,generation_2
bucket_1,object_3,generation_3

      You can also create a manifest using Storage Insights datasets. For details, see Create a manifest using Storage Insights datasets.

    • Object prefixes: Specify a list of prefixes to filter objects within the bucket. Only objects with these prefixes are processed. If empty, all objects in the bucket are processed.

  • Job type: Storage batch operations supports the following job types, running a single job per batch operation.

    • Object deletion: You can delete objects within a bucket. This is crucial for cost optimization, data lifecycle management, and compliance with data deletion policies.

    • Metadata updates: You can modify the object metadata. This includes updating custom metadata, storage class, and other object properties.

    • Object hold updates: You can enable or disable object holds. Object holds prevent objects from being deleted or modified, which is essential for compliance and data retention purposes.

    • Object encryption key updates: You can manage the customer-managed encryption keys for one or more objects. This includes applying or changing encryption keys using the rewrite object method.

Limitations

Storage batch operations has the following limitations:

  • Storage batch operations jobs have a maximum lifetime of 14 days. Any ongoing job that doesn't complete within 14 days of its creation is automatically cancelled.

  • We don't recommend running more than 20 concurrent batch operations jobs on the same bucket.

  • Storage batch operations is not supported on the following buckets:

    • Buckets that have Requestor Pays enabled.

    • Buckets located in the us-west8 region.

What's next