We Were Pulling the Same Docker Image 30 Times a Day. Here's the Fix.

We run GitHub self-hosted runners inside an Azure VNet. Every CI job builds an application image, and the Dockerfile starts with something ordinary:

FROM node:18-alpine
FROM nginx:latest

Just pulling base images from Docker Hub. For one application that's maybe 500MB. But we run 20+ microservices, they build on every push across every feature branch, and the runners don't cache layers between jobs, every build is a fresh checkout.

The worst part only became obvious once we looked: we were downloading the exact same node:18-alpine layer roughly 30 times a day. Every single time, from Docker Hub, over the internet.

What that was actually costing us

The runners sit in a private subnet, so all that traffic goes out through a NAT gateway. NAT gateways charge per GB processed, and at 10 to 15GB of outbound data a day just for public image pulls, a rate that sounds tiny turned into a real line item.

westus3, numbers rounded. And this was only the steady-state cost, during release weeks or when we scaled up runners, it doubled.

$49 a month for one environment isn't the headline. Multiply it across dev, staging, and production, add the build time lost to pulling over a NAT instead of inside Azure's own network, and add the risk of a Docker Hub rate limit failing a build mid-release, and the real cost is bigger than the invoice line suggests.

The fix: stop pulling from the internet

If those base images lived in our own Azure Container Registry, every pull would happen inside the Azure backbone instead. No NAT processing charge, no public bandwidth, no Docker Hub rate limit. Just fast pulls from a registry we control.

The only missing piece was keeping ACR in sync with upstream. That's what SyncerD does, a small open-source Go tool that watches Docker Hub and copies new tags into a registry you control.

Here's exactly what we did

1. Wrote a syncerd.yaml listing every image our CI actually uses:

# syncerd.yaml
source:
  type: dockerhub
  registry: docker.io

destinations:
  - name: rg-core-dev-wus3
    type: acr
    registry: acrcoredevwus3.azurecr.io

images:
  - name: library/node
    tags: ["18-alpine", "20-alpine"]
  - name: library/nginx
    tags: ["latest", "1.25"]
  - name: library/redis
    tags: ["7-alpine"]
  - name: library/python
    tags: ["3.12-slim"]

2. Ran the first sync by hand to backfill everything:

$ export SYNCERD_SOURCE_USERNAME=<dockerhub-user>
$ export SYNCERD_SOURCE_PASSWORD=<dockerhub-token>
$ az acr login --name acrcoredevwus3
$ syncerd sync --once

3. Added a scheduled GitHub Actions workflow so the mirror keeps itself current, no separate infrastructure to run:

# .github/workflows/sync-base-images.yml
name: Sync base images to ACR

on:
  schedule:
    - cron: "0 */6 * * *"   # every 6 hours
  workflow_dispatch:

jobs:
  sync:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4
      - name: Azure login
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}
      - name: Log in to ACR
        run: az acr login --name acrcoredevwus3
      - name: Run SyncerD
        uses: clouddrove/SyncerD@v1
        with:
          config: syncerd.yaml
          once: "true"
        env:
          SYNCERD_SOURCE_USERNAME: ${{ secrets.DOCKERHUB_USER }}
          SYNCERD_SOURCE_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}

4. Changed the Dockerfiles. One line each:

# before
FROM node:18-alpine

# after
FROM acrcoredevwus3.azurecr.io/node:18-alpine

That's it. The CI environment now keeps its own registry current, and the runners pull from inside the Azure network. No extra infrastructure to babysit.

The result

~$16.20 a month back, with peaks up to $40 during heavy weeks, is the number on the invoice. It's not the real win. The real win is three things at once: builds got faster because ACR pulls beat Docker Hub over a NAT every time, we stopped worrying about Docker Hub rate limits mid-release, and the same fix scales across dev, staging, and production without anyone touching it again.

What to do next

If your CI runners pull base images through a NAT gateway, a proxy, or any metered connection, mirror those images into a private registry. This took under 30 minutes to set up and it's been running itself since. We wrote up a related story on what happens when you don't do this before a routine node rotation: Docker Hub Took Down Our Production AKS Cluster.

SyncerD on GitHub →