Improve transfer speeds

Transfer speeds are affected by factors including source location and provider, file sizes, and number of files.

If your transfer is progressing more slowly than expected, refer to the information on this page for possible reasons and resolutions.

Agentless transfers

For transfers to Cloud Storage from Amazon S3, Microsoft Azure, URL lists, or Cloud Storage, Storage Transfer Service manages the transfer without the need for hosted transfer agents.

Create multiple parallel transfers

Storage Transfer Service has a maximum number of allowed queries per second (QPS) per transfer job. If your job involves a large number of relatively small files, its transfer speed is limited by this QPS cap. Transferring an object can trigger list, read, and write operations, each of which count against the maximum QPS.

To work around the QPS limit, split your large transfer into multiple transfer jobs. Use include and exclude prefixes to create transfer jobs containing fewer files. You can create:

• up to 5000 jobs per day
• with 200 jobs running at any point in time

For example, to transfer only files whose filename or path begin with the letters a through e:

gcloud transfer jobs create SOURCE DESTINATION \
  --include-prefixes="a,b,c,d,e"

{
    "description": "YOUR DESCRIPTION",
    "status": "ENABLED",
    "projectId": "PROJECT_ID",
    "schedule": {
        "scheduleStartDate": {
            "day": 1,
            "month": 1,
            "year": 2015
        },
        "startTimeOfDay": {
            "hours": 1,
            "minutes": 1
        }
    },
    "transferSpec": {
        "gcsDataSource": {
            "bucketName": "GCS_SOURCE_NAME"
        },
        "gcsDataSink": {
            "bucketName": "GCS_SINK_NAME"
        },
        "transferOptions": {
            "deleteObjectsFromSourceAfterTransfer": true
        },
        "objectConditions": {
            "includePrefixes": [
                "a","b","c","d","e"
            ],
            "excludePrefixes": [
                "path_1/subpath_2/object_5"
            ]
        }
    }
}

Agent-based transfers

For tips on speeding up agent-based transfers, refer to Best practices for file system transfers.