How And Why I Moved From Docker Hub To GitHub Docker Registry.

10 Min Read

The Story

On August 2020, Docker announced that they are introducing rate-limiting for Docker container pulls for free or anonymous users, which meant if you did not login to your DockerHub registry via command-line you would be limited to 100 pulls per 6 hours. At first, this did not affect me as I rarely pulled 10 images per day, but recently I have been tinkering with Kubernetes, Prometheus, Jaeger (you can check this post on how to install Prometheus & Grafana on K3s cluster) and other tools which usually pulls multiple images per run. You can check

This meant that I would be pulling images more frequently than I used to in the past. That’s when I got the dreaded error message, “429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit”.

This meant that I either had to configure Kubernetes secrets for me to pull from an authenticated DockerHub registry or find an alternative registry does will not issue rate limits every 100th pull. Coincidentally, roughly at the same time GitHub introduced their container registry (offering private container registry that integrates easily with the existing CI/CD tooling), and we were saved. Since it’s still in its beta stage usage is free.

In this post, I will detail how I migrated from using DockerHub to GitHub as my Docker container registry.

TL;DR

Setup GitHub Container registry and configure Kubernetes pod container to pull from the private registry (Optional)

But wait, What is a Container registry?

According to RedHat,

A container registry is a repository, or collection of repositories, used to store container images for Kubernetes, DevOps, and container-based application development.

GitHub is the home for free and open-source software and it’s in a great spot to offer a container registry, which integrates well with their existing services and operates as an extension of GitHub Packages. Thus making it a good competitor to DockerHub.

The How

After deploying my application on my Kubernetes cluster. I noticed a few errors and after troubleshooting, I found that the docker pull rate limit was hit and this drove me insane.

$ kubectl describe pod frontend-app-6b885c795d-9vbfx | tail

Events:
  Type     Reason          Age                From                Message
  ----     ------          ----               ----                -------
  Warning  FailedMount     72s                kubelet, dashboard  MountVolume.SetUp failed for volume "default-token-6zmpp" : failed to sync secret cache: timed out waiting for the condition
  Normal   SandboxChanged  71s                kubelet, dashboard  Pod sandbox changed, it will be killed and re-created.
  Warning  Failed          44s                kubelet, dashboard  Failed to pull image "mmphego/frontend:v7": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/mmphego/frontend:v7": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/mmphego/frontend/manifests/sha256:2994ce56c38abe2947935d7bc9d6a743dfc30186659aae80d5f2b51a0b8f37d1: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Warning  Failed          44s                kubelet, dashboard  Error: ErrImagePull
  Normal   BackOff         43s                kubelet, dashboard  Back-off pulling image "mmphego/frontend:v7"
  Warning  Failed          43s                kubelet, dashboard  Error: ImagePullBackOff
  Normal   Pulling         31s (x2 over 70s)  kubelet, dashboard  Pulling image "mmphego/frontend:v7"

The Walk-through

Setting up your container registry is straight forward.

Steps

1. Create a GitHub Personal Token on https://github.com/settings/apps (See images below)

   • Select Personal Access Tokens

     

   • Click Generate a new token

     

   • Add a Note, check write: packages and hit generate.

     

   When done, you will be provided with a token that you need to backup.

2. GitHub recommends placing the token into a file.

   Either add the token into your ~/.bashrc or ~/.bash_profile and risk exposing them as environmental variables or place them in a file under a secret directory with reading/writing privileges (I prefer the latter).

    vim ~/.secrets/github_docker_token
   

   Paste the token into the github_docker_token file.

3. Login to GitHub Container Registry

   Setup your username as an environmental variable:

    $ cat ~/.bashrc
   
    export GH_EMAIL=$(git config user.email)
    export GH_USERNAME=$(git config user.username) # or hardcode your username (Not Recommended!)
   

   Log in to your container registry with your username and personal token.

    cat ~/.secrets/github_docker_token | docker login ghcr.io -u ${GH_USERNAME} --password-stdin
   

   

   If successful, you should see an image similar to the one above.

   Note: Typing secrets on the command line may store them in your shell history unprotected, and those secrets might also be visible to other users on your PC.

4. Confirm that you successfully logged in.

   To confirm that you logged in we need to build, tag and push our image to ghcr (GitHub Container Registry).

    export USERNAME="add information"
    export REPOSITORY="add information"
    export IMAGE="add information"
    export VERSION="add information"
    docker build . -t ghcr.io/${USERNAME}/${REPOSITORY}/${IMAGE}:${VERSION}
    docker push ghcr.io/${USERNAME}/${REPOSITORY}/${IMAGE}:${VERSION}
   

   or in my case, I have a docker-compose yaml file to make my life easier (I suppose).

    cat docker-compose-file.yaml
   

    version: '3'
    services:
    hello_world_app:
        build: ../../app
        image: ghcr.io/mmphego/jaeger-tracing-example/jaeger-tracing-example:v2
   

   Then build the tagged image.

    docker-compose -f deployment/docker/docker-compose-file.yaml build
   

   

   continues…

   

   After a successful build, we push the image to the registry.

    docker-compose -f deployment/docker/docker-compose-file.yaml push
   

   

   Note: The manual build and push steps will be used on GitHub Actions (so at this point ensure everything works 100%)

5. After pushing the image you should see new package(s) on your profile under Packages.

   

6. Setup GitHub Action workflow for auto-build and publish (Optional)

   Optionally, set up a workflow environment in your repository/project for the CI/CD to build and publish your container to the GitHub container registry.

    mkdir -p .github/workflows
    vim .github/workflows/docker-image-publisher.yaml
   

   Paste the following code snippet into your docker-image-publisher.yaml. This workflow will build and push images on pull requests and master branches.

    ---
    name: Docker Image CI
   
    on:
    workflow_dispatch: # Run workflow manually (without waiting for the cron to be called), through the Github Actions Workflow page directly
    push:
        branches:
        - master
    pull_request:
        branches:
        - '*'
   
    jobs:
    build:
        runs-on: ubuntu-latest
        steps:
        - uses: actions/checkout@v2
            with:
            fetch-depth: 0
   
        - name: Build the Docker image
            run: |
            export USERNAME=${{ github.repository_owner }}
            docker-compose -f deployment/docker/docker-compose-file.yaml build
   
        - name: Login and Push Docker images to GitHub container registry
            run: |
            export USERNAME=${{ github.repository_owner }}
            echo "${{ secrets.DOCKER_PASSWORD }}" | docker login ghcr.io -u ${{ secrets.DOCKER_USERNAME }} --password-stdin
            docker-compose -f deployment/docker/docker-compose-file.yaml push
   

   Run the GitHub Action build manually…

   

   Alternatively, checkout the Docker Build & Push Action or Build and push Docker images

7. Configure Kubernetes to use your new container registry (Optional)

   Kubernetes supports a special type of secret that you can create which will be used to fetch images for your pods from any container registry that requires authentication.

   Create a Kubernetes Secret, naming it my-secret-docker-reg and providing credentials:

    kubectl create secret docker-registry my-secret-docker-reg \
        --docker-server=https://ghcr.io \
        --docker-username=${GH_USERNAME} \
        --docker-password=$(cat  ~/.secrets/github_docker_token) \
        --docker-email=${GH_EMAIL} -o yaml > docker-secret.yaml
        # or  kubectl apply the output of an imperative command in one line
        # --docker-email=${GH_EMAIL} -o yaml | kubectl apply -f -
   
    # You can then apply the file like any other Kubernetes 'yaml':
    kubectl apply -f docker-secret.yaml
   

   Inspect the Secret: my-secret-docker-reg

    kubectl get secrets
   

   

Final Result

Successful pull from (then private) GitHub container registry!

Me: After setting up my GitHub container registry and Kubernetes docker-secrets.

Conclusion

Hopefully, you have learned something new in this post (and enjoyed Denzel Washington gifs) and will consider using GitHub Container Registry to house your images and GitHub Actions to build and push them to your GitHub Container Registry! Finally, to configure your Kubernetes cluster to use GitHub Container Registry for fetching images for your pods.

Reference

• Migrating to GitHub Container Registry for Docker images
• Pull an Image from a Private Registry