Docker Hub Took Down Our Production AKS Cluster. Here's How We Fixed It.

Before the outage, our application Helm charts all looked something like this:

# values.yaml — before
image:
  repository: nginx
  tag: "latest"
  pullPolicy: IfNotPresent

Docker Hub was pulling images for nginx, node, redis, postgres, and more. It was simple. It worked. Until it didn't.

What happened

One day Azure did a routine AKS node image update. Old nodes were replaced, pods rescheduled onto fresh nodes, and every single new node tried to pull all those images from Docker Hub at the same time. In a few minutes we blew through the anonymous pull rate limit. Pods stuck in ContainerCreating.

I checked the events and saw this everywhere:

Failed to pull image "nginx:latest": toomanyrequests: You have reached your
pull rate limit.

After manually patching things to get the cluster stable again, the real fix was obvious: stop pulling from Docker Hub in production. Mirror the images into Azure Container Registry (ACR) and pull from there instead.

The fix: mirror once, pull from our own registry

I used SyncerD, a small open-source Go tool built by our team. It watches Docker Hub and copies new tags into a registry you control, on a schedule, so your cluster never touches the public internet for an image it already has.

Here's exactly what I did

1. Wrote a config file (syncerd.yaml) telling SyncerD which images to watch and where to copy them:

source:
  type: dockerhub
  registry: docker.io

destinations:
  - name: rg-shared-prod-eus
    type: acr
    registry: acrsharedprodeus.azurecr.io

images:
  - name: library/nginx
    watch_tags: true
  - name: library/node
    watch_tags: true
  - name: library/redis
    watch_tags: true
  - name: library/postgres
    tags: ["15", "15-alpine"]

2. Logged in and ran a one-time sync to backfill everything we needed:

$ export SYNCERD_SOURCE_USERNAME=<dockerhub-user>
$ export SYNCERD_SOURCE_PASSWORD=<dockerhub-token>
$ az acr login --name acrsharedprodeus
$ syncerd sync --once

3. Switched the Helm chart to pull from ACR instead of Docker Hub. One line changed:

# values.yaml — after
image:
  repository: acrsharedprodeus.azurecr.io/nginx
  tag: "latest"
  pullPolicy: IfNotPresent

4. Deployed SyncerD itself as a Kubernetes CronJob, so the mirror stays current without anyone remembering to run it:

# prod.yaml — Helm values for the SyncerD chart
cronjob:
  schedule: "0 */6 * * *"   # every 6 hours
  concurrencyPolicy: Forbid
  restartPolicy: OnFailure

config:
  source: { type: dockerhub, registry: docker.io }
  destinations:
    - { name: rg-shared-prod-eus, type: acr, registry: acrsharedprodeus.azurecr.io }

The result

That was it. The next time AKS rotated nodes, every image pull came from ACR — fast, unlimited, and entirely inside the Azure network. No rate limits. No outage. No panic.

If your production cluster still pulls public images straight from Docker Hub, fix it before a routine node update reminds you the hard way. It took under 30 minutes to set up and it's been quietly running ever since.

We wrote up a second, deeper walkthrough of SyncerD here: Beating Docker Hub rate limits with SyncerD.

SyncerD on GitHub →