ECR Rate Limit Error response from daemon: toomanyrequests: Rate exceeded

[maxkostow]

Bug report

Describe the bug

Sometimes when running supabase start it will return the error Error response from daemon: toomanyrequests: Rate exceeded

To Reproduce

Steps to reproduce the behavior, please provide code snippets or a repository:

1. Delete docker image cache
2. supabase start
3. Sometimes you will see the error

Expected behavior

• There should be documentation around ECR/Docker rate limits and how to mitigate them.
• I have tried using the docker-login action as well as aws-ecr-login which did not seem to affect the docker client in use here.

Perhaps ImagePullOptions.RegistryAuth can be exposed somehow?

https://github.com/moby/moby/blob/5f698d112134d27ffe9850bfd46708bd9fa35860/api/types/client.go#L278-L283

cli/internal/utils/docker.go

Line 174 in 5a3fc3b

out, err := Docker.ImagePull(ctx, imageUrl, types.ImagePullOptions{})

• Perhaps the service quota needs to be increased? https://docs.aws.amazon.com/AmazonECR/latest/public/public-service-quotas.html
• AWS ECR Auth: https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html#registry_auth_http

[sweatybridge]

Hi @maxkostow , thanks for detailed bug report.

Supporting RegistryAuth should be the right way to go as unauthenticated rate limit on ECR cannot be adjusted. While we are working on this fix and related docs, you may set the following env var on GitHub Action to use docker hub images instead.

SUPABASE_INTERNAL_IMAGE_REGISTRY=docker.io supabase start

Let me know if it works for you.

[CharlesFJWilliamson]

I just got the same error as above, but the temporary fix worked for me @sweatybridge thanks

[satyavh]

Tried out Supabase for the very first time, got stuck already with supabase start and this error --- not impressed so far, feels like a fragile framework

[satyavh]

Supabase CLI 1.4.5

supabase start
Error: Error response from daemon: toomanyrequests: Rate exceeded

SUPABASE_INTERNAL_IMAGE_REGISTRY=docker.io supabase start
Error: Error response from daemon: No such image: supabase/postgres:14.1.0.66

supabase start
Error: Error response from daemon: No such image: public.ecr.aws/t3w2s2c9/postgres:14.1.0.66

😐 I give up...

[satyavh]

Talked to support, pulling the image directly docker pull public.ecr.aws/t3w2s2c9/postgres:14.1.0.66 showed the correct error: Docker running out of disk space 😬 Would be great if the Supabase CLI would forward that info 🙏

[StefanBRas]

For anyone else experiencing this, running the command three times in a row worked for me (local setup on Ubuntu 22.04).

[sweatybridge]

Thanks for the feedback on this issue. We have released v1.5.4 that adds retries with exponential backoff when pulling images. We ran 97 tests yesterday on CI and all of the supabase start commands succeeded in one try. You should be able to reliably pull from ECR now even as an anonymous user.

Please feel free to reopen if you encounter further problems.