GitHub Self-Hosted Private Runners on AWS

A self-hosted runner is a system that you deploy and manage to execute jobs from GitHub Actions on GitHub.com. It give you more control over your CI/CD (Continuous Integration/Continuous Deployment) pipelines.

Benefits of Self-Hosted Runners

• Customization: Tailor environments to specific needs, such as using specialized software or hardware.

• Performance: Choose instance types that optimize your workflow’s performance.

• Cost-Effectiveness: For extensive workflows, self-hosting may reduce expenses.

• Security: Maintain control over the environment, meeting stringent compliance requirements.

You can add self-hosted runners at various levels in the management hierarchy:

• Repository-level runners are dedicated to a single repository.
• Organization-level runners can process jobs for multiple repositories in an organization.
• Enterprise-level runners can be assigned to multiple organizations in an enterprise account.

Key Steps:

Download the GitHub Runner:

• Navigate to the GitHub Actions runner page and copy the download URL for the latest version of the runner. On your server instance, run:

mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-x64-2.319.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.319.0/actions-runner-linux-x64-2.319.0.tar.gz
tar xzf ./actions-runner-linux-x64-2.319.0.tar.gz

Configure the GitHub Runner:

• Obtain the GitHub runner token:
  Go to your repository on GitHub.
  Navigate to Settings > Actions > Runners > New self-hosted runner.
  Follow the instructions to generate a token.
  https://github.com/YourGithubOrg/YourGithubRepo/settings/actions/runners/new?arch=x64&os=linux
  

• On the server instance, configure the runner using the token:

./config.sh --unattended --url https://github.com/${GH_ORGNAME}/${GH_REPONAME} --token ${GH_TOKEN} --name aws-hosted-runner --labels aws-hosted-runner --disableupdate

Run the GitHub Runner

• Set Up the Runner as a Service:
  To ensure the runner starts automatically on boot, configure it as a service.

./svc.sh install
./svc.sh start

• Verify the Runner in GitHub:
  Go back to your repository on GitHub and check the Settings > Actions > Runners section. You should see your runner listed as Idle
  

In this demo we will showcase the Build of a sample application code as Docker image and push it to Amazon ECR using Github Actions running on self-hosted runners. We will create an EC2 instance, ECR Repository & IAM instance profile role using Terraform configuration.

Create EC2 Instance on AWS

Use Terraform to launch an EC2 instance in a private subnet. Configure the instance with an IAM role for accessing the ECR repository and set up the necessary security groups.

resource "aws_instance" "gh_runner_server" {
  ami                         = data.aws_ami.ubuntu.id
  instance_type               = var.gh_runner_server_instance_type
  associate_public_ip_address = false
  subnet_id                   = aws_subnet.private_subnet[0].id
  vpc_security_group_ids      = [aws_security_group.gh_runner_server_sg.id]
  source_dest_check           = false

  user_data = data.template_file.init.rendered

  root_block_device {
    volume_size           = 30
    delete_on_termination = true
  }

  iam_instance_profile = aws_iam_instance_profile.ecr_instance_profile.name

  tags = {
    Name = "${var.prefix}-gh-runner-server"
  }

  depends_on = [aws_nat_gateway.natgw]
}

# User Data Init
data "template_file" "init" {
  template = file("${path.module}/gh_sf_runner.sh")

  vars = {
    GH_TOKEN    = var.gh_token
    GH_ORGNAME  = var.gh_orgname
    GH_REPONAME = var.gh_reponame
  }
}

Install GitHub Runner

Retrieve the runner registration token from your GitHub repository and set up the runner on the EC2 instance. This includes downloading and configuring the GitHub runner, starting the services, and installing Docker.

We will install the Github runner and Docker Engine in the EC2 instance using user data initialization script as part of Terraform Configuration.

#!/bin/bash
# Owner: Saurav Mitra
# Description: Configure Gihub Self-hosted Runner

# Install GH Runner
mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-x64-2.319.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.319.0/actions-runner-linux-x64-2.319.0.tar.gz
tar xzf ./actions-runner-linux-x64-2.319.0.tar.gz
export RUNNER_ALLOW_RUNASROOT="1"
./config.sh --unattended --url https://github.com/${GH_ORGNAME}/${GH_REPONAME} --token ${GH_TOKEN} --name aws-hosted-runner --labels aws-hosted-runner --disableupdate
# ./run.sh
./svc.sh install
./svc.sh start
./svc.sh status

# Install Docker Engine for Demo
apt update
apt install apt-transport-https ca-certificates curl software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt update
apt install docker-ce -y
systemctl status docker

Configure GitHub Actions

Update your GitHub repository’s workflow file to specify the self-hosted runner. This setup will build and push Docker images to Amazon ECR.

Setup Github Repository secrets:

• AWS_REGION: eu-central-1
• AWS_ACCOUNT_ID: 123456789012
• ECR_REPO_NAME: backend-api

Workflow YAML File:

• In your GitHub repository, edit the .github/workflows/your-workflow.yml file to specify the self-hosted runner:

name: CI
on:
  push:
    branches:
      - main
jobs:
  build:
    name: Build and Push Docker image to ECR
    runs-on: ["aws-hosted-runner"]
    steps:
      - name: Checkout Code Repository
        uses: actions/checkout@v4

      - name: Set AWS Region      
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-region: $\{\{ secrets.AWS_REGION }}

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2
        with:
          registries: $\{\{ secrets.AWS_ACCOUNT_ID }}

      - name: Build and Push Docker image to Amazon ECR
        env:
          ECR_REGISTRY: $\{\{ steps.login-ecr.outputs.registry }}
          ECR_REPOSITORY: $\{\{ secrets.ECR_REPO_NAME }}
          IMAGE_TAG: $\{\{ github.sha }}
        run: |
          docker build -t "$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" .
          docker push "$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG"
          echo "IMAGE $IMAGE_TAG is pushed to $ECR_REGISTRY/$ECR_REPOSITORY"
          echo "Image Tag: $IMAGE_TAG" 
          echo "Image Name: $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG"

Commit and Push:

• Commit the changes to your repository and push them. The workflow should now run using your self-hosted runner.

You can refer this Github Repository for the full codebase.

Conclusion

Setting up self-hosted GitHub runners on AWS provides flexibility, performance, and potentially cost savings for your CI/CD pipelines. While the initial setup requires some effort, the control and customization it offers are well worth it, especially for complex workflows or high-security environments. By following this guide, you’ll be well on your way to optimizing your GitHub Actions workflows using AWS.