DEV Community

Purushotam Adhikari
Purushotam Adhikari

Posted on

Setting Up a Self-Hosted GitLab Runner for CI/CD

#cicd #webdev #git #gitlab

GitLab's CI/CD capabilities are incredibly powerful, but sometimes you need more control over your build environment than what shared runners can provide. Whether you're dealing with specific hardware requirements, security constraints, or simply want to reduce build times, setting up your own GitLab Runner can be a game-changer.

In this comprehensive guide, we'll walk through the entire process of setting up a self-hosted GitLab Runner, from initial installation to advanced configuration options.

Why Self-Host Your GitLab Runner?

Before diving into the setup, let's understand why you might want to self-host:

Performance Benefits: Your own hardware means no queuing behind other users, and you can optimize for your specific workloads.

Security & Compliance: Keep sensitive builds within your infrastructure, meeting enterprise security requirements.

Custom Environment: Install specific tools, databases, or services that your CI/CD pipeline needs.

Cost Control: For high-volume projects, self-hosting can be more economical than using shared runners extensively.

Prerequisites

Before we begin, ensure you have:

  • A GitLab account (GitLab.com or self-hosted GitLab instance)
  • A server or virtual machine (Linux, Windows, or macOS)
  • Administrator access to your GitLab project
  • Basic command line knowledge

Step 1: Preparing Your Environment

First, let's set up our server environment. I'll use Ubuntu 22.04 for this example, but the process is similar for other distributions.

Update your system packages:

sudo apt update && sudo apt upgrade -y
Enter fullscreen mode Exit fullscreen mode

Install essential tools:

sudo apt install -y curl git wget software-properties-common
Enter fullscreen mode Exit fullscreen mode

Step 2: Installing GitLab Runner

GitLab provides official repositories for easy installation. Let's add the repository and install the runner:

# Add the GitLab Runner repository
curl -L "http://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

# Install GitLab Runner
sudo apt install gitlab-runner
Enter fullscreen mode Exit fullscreen mode

Verify the installation:

gitlab-runner --version
Enter fullscreen mode Exit fullscreen mode

Step 3: Registering Your Runner

Now comes the crucial step - registering your runner with GitLab. You'll need a registration token from your GitLab project.

Getting Your Registration Token

  1. Navigate to your GitLab project
  2. Go to Settings → CI/CD
  3. Expand the Runners section
  4. Copy the registration token

Registering the Runner

Run the registration command:

sudo gitlab-runner register
Enter fullscreen mode Exit fullscreen mode

You'll be prompted for several configuration options:

Enter the GitLab instance URL: http://gitlab.com/
Enter the registration token: [your-token-here]
Enter a description for the runner: My Self-Hosted Runner
Enter tags for the runner: linux,docker,self-hosted
Enter optional maintenance note: 
Enter an executor: docker
Enter the default Docker image: alpine:latest
Enter fullscreen mode Exit fullscreen mode

Step 4: Configuring Docker Executor

The Docker executor is the most versatile option, allowing you to run jobs in isolated containers. Let's configure it properly.

Edit the runner configuration:

sudo nano /etc/gitlab-runner/config.toml
Enter fullscreen mode Exit fullscreen mode

Here's an optimized configuration example:

concurrent = 4
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "My Self-Hosted Runner"
  url = "http://gitlab.com/"
  token = "your-runner-token"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock:rw"]
    shm_size = 0
    pull_policy = "if-not-present"
Enter fullscreen mode Exit fullscreen mode

Step 5: Setting Up Docker

Since we're using the Docker executor, we need Docker installed on our runner machine:

# Add Docker's official GPG key
sudo apt install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL http://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add the repository
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] http://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Add gitlab-runner user to docker group
sudo usermod -aG docker gitlab-runner
Enter fullscreen mode Exit fullscreen mode

Step 6: Optimizing Runner Performance

Concurrent Jobs

Adjust the concurrent setting in your config.toml based on your server's resources:

concurrent = 4  # Number of jobs that can run simultaneously
Enter fullscreen mode Exit fullscreen mode

Caching Strategy

Implement effective caching to speed up builds:

[runners.cache]
  Type = "local"
  Path = "/tmp/gitlab-runner-cache"
  Shared = true
Enter fullscreen mode Exit fullscreen mode

Resource Limits

Set appropriate resource limits for Docker containers:

[runners.docker]
  memory = "2g"
  cpus = "1.0"
  cpu_shares = 1024
Enter fullscreen mode Exit fullscreen mode

Step 7: Creating Your First Pipeline

Let's create a simple .gitlab-ci.yml file to test our runner:

stages:
  - test
  - build
  - deploy

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

test:
  stage: test
  tags:
    - self-hosted
  script:
    - echo "Running tests on self-hosted runner"
    - whoami
    - pwd
    - docker --version
  only:
    - main

build:
  stage: build
  tags:
    - self-hosted
  script:
    - echo "Building application"
    - docker build -t my-app .
  only:
    - main

deploy:
  stage: deploy
  tags:
    - self-hosted
  script:
    - echo "Deploying application"
  only:
    - main
  when: manual
Enter fullscreen mode Exit fullscreen mode

Step 8: Security Considerations

Runner Isolation

Always run your GitLab Runner with appropriate user permissions:

# Create a dedicated user for the runner
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

# Install and register as the gitlab-runner user
sudo -u gitlab-runner gitlab-runner register
Enter fullscreen mode Exit fullscreen mode

Network Security

Configure firewall rules to restrict access:

# Allow SSH (adjust port as needed)
sudo ufw allow 22

# Allow outbound HTTPS for GitLab communication
sudo ufw allow out 443

# Enable firewall
sudo ufw enable
Enter fullscreen mode Exit fullscreen mode

Secret Management

Never hardcode secrets in your CI/CD configuration. Use GitLab's built-in secret management:

  1. Go to Settings → CI/CD → Variables
  2. Add your secrets as protected variables
  3. Reference them in your pipeline: $SECRET_VARIABLE

Step 9: Monitoring and Maintenance

Health Checks

Regularly verify your runner's health:

# Check runner status
sudo gitlab-runner verify

# View runner logs
sudo journalctl -u gitlab-runner -f
Enter fullscreen mode Exit fullscreen mode

Automated Updates

Set up automatic updates for your runner:

# Create a cron job for weekly updates
echo "0 2 * * 0 root apt update && apt upgrade -y gitlab-runner" | sudo tee -a /etc/crontab
Enter fullscreen mode Exit fullscreen mode

Resource Monitoring

Monitor system resources to ensure optimal performance:

# Install monitoring tools
sudo apt install htop iotop nethogs

# Monitor in real-time
htop
Enter fullscreen mode Exit fullscreen mode

Step 10: Advanced Configuration Options

Multiple Executors

You can configure multiple runners with different executors:

# Register additional runners
sudo gitlab-runner register --executor shell --tag-list "shell,deployment"
sudo gitlab-runner register --executor kubernetes --tag-list "kubernetes,scalable"
Enter fullscreen mode Exit fullscreen mode

Custom Docker Images

Create optimized Docker images for your specific needs:

FROM node:18-alpine
RUN apk add --no-cache git python3 make g++
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
Enter fullscreen mode Exit fullscreen mode

Artifacts and Caching

Configure advanced caching strategies:

build:
  stage: build
  cache:
    key: "$CI_COMMIT_REF_SLUG"
    paths:
      - node_modules/
      - .npm/
  artifacts:
    paths:
      - dist/
    expire_in: 1 week
Enter fullscreen mode Exit fullscreen mode

Troubleshooting Common Issues

Runner Not Appearing

If your runner doesn't appear in GitLab:

  1. Check the registration token
  2. Verify network connectivity
  3. Review runner logs: sudo journalctl -u gitlab-runner

Build Failures

Common build failure causes:

  • Insufficient permissions
  • Missing dependencies
  • Resource constraints
  • Network connectivity issues

Performance Issues

To improve performance:

  • Increase concurrent job limits
  • Optimize Docker image sizes
  • Use local caching
  • Monitor resource usage

Best Practices Summary

Security First: Always follow the principle of least privilege and keep your runner updated.

Resource Management: Monitor and optimize resource usage to prevent bottlenecks.

Tagging Strategy: Use meaningful tags to route jobs to appropriate runners.

Maintenance Schedule: Regularly update and maintain your runner infrastructure.

Monitoring: Implement comprehensive monitoring and alerting.

Conclusion

Setting up a self-hosted GitLab Runner gives you unprecedented control over your CI/CD environment. While it requires initial setup and ongoing maintenance, the benefits in terms of performance, security, and customization make it worthwhile for many projects.

Remember to start simple and gradually add complexity as your needs grow. The configuration we've covered provides a solid foundation that you can build upon.

Have you set up your own GitLab Runner? What challenges did you face, and how did you solve them? Share your experiences in the comments below!

Want to learn more about GitLab CI/CD? Check out my other articles on advanced pipeline patterns and deployment strategies.

Follow me for more DevOps and CI/CD content! 🚀

Top comments (0)