Setting up external monitoring with OpenTelemetry

External monitoring approaches

External monitoring with OpenTelemetry allows you to integrate your GitHub Enterprise Server instance with existing monitoring infrastructure and tools. GitHub Enterprise Server provides two primary approaches for external monitoring:

1. Prometheus endpoint: Expose metrics in Prometheus format for external tools to scrape
2. Custom OpenTelemetry pipelines: Configure the OpenTelemetry collector to push metrics to external monitoring systems

Prerequisites

• GitHub Enterprise Server 3.18 or later with OpenTelemetry metrics enabled
• Network connectivity between your GitHub Enterprise Server instance and external monitoring systems
• Administrative access to both your GitHub Enterprise Server instance and monitoring systems

Choosing your monitoring approach

Choose the monitoring approach that best fits your infrastructure and requirements.

Prometheus endpoint (recommended for most users)

Use the Prometheus endpoint when:

• You have existing Prometheus-based monitoring infrastructure
• You prefer a pull-based monitoring model
• You want to use external tools to scrape GitHub Enterprise Server metrics
• You need simple integration with minimal configuration

Custom OpenTelemetry pipelines

Use custom OpenTelemetry pipelines when:

• You need to push metrics to multiple monitoring systems simultaneously
• You want to transform, filter, or aggregate metrics before sending them externally
• You're using cloud-native monitoring solutions that prefer OTLP
• You need advanced metric processing capabilities

Option 1: Using the Prometheus endpoint

The OpenTelemetry monitoring stack includes a Prometheus-compatible endpoint that exposes all collected metrics in Prometheus format, enabling integration with external monitoring tools. See Text-based format in the documentation for Prometheus.

Enabling the Prometheus endpoint

You can enable the Prometheus endpoint using either the Console de Gerenciamento or the command line interface.

Using the Management Console

1. Em uma conta administrativa no GitHub Enterprise Server, no canto superior direito de qualquer página, clique em .

2. Se você ainda não estiver na página "Administração do site", no canto superior esquerdo, clique em Administração do site.

3. Na barra lateral " Site admin", clique em Console de Gerenciamento.

4. In the monitoring settings, select Enable OpenTelemetry.

5. Optionally, adjust the Metrics retention and Metrics scrape interval fields to optimize performance.

6. Select Enable Prometheus-compatible (/metrics) endpoint.

7. In the Endpoint password field, enter a strong password to control access to the endpoint.

8. Optionally, in the Trusted IPv4/IPv6 addresses or CIDR blocks in a comma-separated list field, enter the IP addresses or CIDR blocks that are allowed to access the endpoint.

9. Na barra lateral "Configurações", clique em Salvar configurações.

   Observação

   Se você salvar as configurações no Console de Gerenciamento, isso reiniciará os serviços do sistema, o que poderá resultar em tempo de inatividade visível pelo usuário.

10. Aguarde a conclusão da execução de suas configurações.

Using the command line interface

1. SSH into your GitHub Enterprise Server instance.

2. Enable the Prometheus endpoint:

   ghe-config observability.metrics.prometheus-endpoint-enabled true
   ghe-config-apply
   

3. Set up authentication for the endpoint (strongly recommended):

   ghe-config observability.metrics.prometheus-endpoint-password your-secure-password
   ghe-config-apply
   

Accessing the Prometheus endpoint

The Prometheus endpoint will be available at:

http://[hostname]:8010/metrics

Authentication uses:

• Username: ghes-metrics
• Password: The value set in your configuration

Configuring external tools to scrape metrics

You can configure various external monitoring tools to scrape metrics from the Prometheus endpoint.

Prometheus

Add the following job configuration to your Prometheus configuration file:

scrape_configs:
  - job_name: 'github-enterprise-server'
    static_configs:
      - targets: ['your-ghes-hostname:8010']
    basic_auth:
      username: 'ghes-metrics'
      password: 'your-secure-password'
    scrape_interval: 30s
    metrics_path: '/metrics'
    scheme: http
    tls_config:
      # Set `true` only when testing with self-signed certificates
      insecure_skip_verify: false  

Other monitoring tools

The Prometheus endpoint is compatible with any monitoring tool that can scrape Prometheus-format metrics, including:

• Grafana (using Prometheus data source)
• Datadog (using Prometheus check)
• New Relic (using Prometheus integration)
• Victoria Metrics
• InfluxDB (using Prometheus input plugin)

Security considerations for the Prometheus endpoint

Additional security measures can be implemented to protect your metrics endpoint.

Password protection

The Prometheus endpoint supports using a password to limit access. You can configure this through either the Console de Gerenciamento or the command line interface.

• Using the Management Console:

  In the monitoring settings, use the Endpoint password field to enter a strong password to control access to the endpoint.

• Using the command line interface:

  ghe-config observability.metrics.prometheus-endpoint-password your-secure-password
  ghe-config-apply
  

IP allowlisting

The Prometheus endpoint supports IP allowlisting for enhanced security. You can configure this through either the Console de Gerenciamento or the command line interface.

• Using the Management Console:

  In the monitoring settings, use the Trusted IPv4/IPv6 addresses or CIDR blocks in a comma-separated list field to enter IP addresses or CIDR blocks that should have access to the endpoint.

• Using the command line interface:

  ghe-config observability.metrics.prometheus-endpoint-trusted-ips "10.0.0.0/8,192.168.0.0/16"
  ghe-config-apply
  

Testing the endpoint

Check that metrics are being exported:

# Test the Prometheus endpoint
curl -u ghes-metrics:your-password http://your-ghes-hostname:8010/metrics

Option 2: Using custom OpenTelemetry pipelines

GitHub Enterprise Server supports custom OpenTelemetry collector configurations, allowing you to extend the monitoring capabilities by adding custom pipelines, exporters, and processors. This enable you to ship metrics to external monitoring systems while maintaining the default GitHub Enterprise Server observability stack.

Important considerations for custom configurations

• Custom configurations are additive to the default GitHub Enterprise Server OpenTelemetry configuration
• Reserved paths: The paths /ghes and /internal are reserved for GitHub Enterprise Server internal use and must not be used in custom configuration files
• otlp/ghes and prometheus/ghes receivers are the built-in sources of GitHub Enterprise Server metrics that you can use in your custom pipelines
• Test your configuration thoroughly in a non-production environment before applying to production
• Monitor resource usage after enabling custom configurations, as additional pipelines may increase CPU and memory consumption

Enabling custom OpenTelemetry configuration

You can enable custom OpenTelemetry configuration using either the Console de Gerenciamento or the command line interface.

Using the Management Console

1. Em uma conta administrativa no GitHub Enterprise Server, no canto superior direito de qualquer página, clique em .

2. Se você ainda não estiver na página "Administração do site", no canto superior esquerdo, clique em Administração do site.

3. Na barra lateral " Site admin", clique em Console de Gerenciamento.

4. In the monitoring settings, first select Enable OpenTelemetry.

5. Select Enable custom OpenTelemetry Collector pipelines.

6. Use the Browse... button to upload your custom OpenTelemetry collector config file (*.yml), see Example configurations for popular monitoring systems.

7. Na barra lateral "Configurações", clique em Salvar configurações.

   Observação

   Se você salvar as configurações no Console de Gerenciamento, isso reiniciará os serviços do sistema, o que poderá resultar em tempo de inatividade visível pelo usuário.

8. Aguarde a conclusão da execução de suas configurações.

Using the command line

1. SSH into your GitHub Enterprise Server instance and run the following command.

   ghe-config observability.metrics.custom-config-enabled true
   

2. Create your custom OpenTelemetry configuration file at /data/user/common/otelcol.yaml:

   sudo nano /data/user/common/otelcol.yaml
   

3. Add your custom configuration (see Example configurations for popular monitoring systems).

4. Apply the configuration:

   ghe-config-apply
   

Example configurations for popular monitoring systems

The following examples show how to configure custom OpenTelemetry pipelines for popular monitoring platforms. For information on other exporters or to create your own, see the repository of OpenTelemetry Collectors.

Grafana Cloud

---
exporters:
  prometheusremotewrite:
    endpoint: "http://prometheus-us-central1.grafana.net/api/prom/push"
    headers:
      Authorization: "Bearer your-grafana-cloud-token"

service:
  pipelines:
    metrics:
      receivers: [otlp/ghes, prometheus/ghes]
      processors: [batch/ghes]
      exporters: [prometheusremotewrite]

Datadog

---
exporters:
  datadog:
    api:
      site: datadoghq.com
      key: your-datadog-api-key
    host_metadata:
      enabled: true

service:
  pipelines:
    metrics:
      receivers: [otlp/ghes, prometheus/ghes]
      processors: [batch/ghes]
      exporters: [datadog]

New Relic

---
exporters:
  otlp:
    endpoint: "http://otlp.nr-data.net:4317"
    headers:
      api-key: "your-new-relic-license-key"

service:
  pipelines:
    metrics:
      receivers: [otlp/ghes, prometheus/ghes]
      processors: [batch/ghes]
      exporters: [otlp]

Generic OTLP endpoint

---
exporters:
  otlp:
    endpoint: "http://your-otel-collector:4317"
    headers:
      api-key: "your-api-key"
    tls:
      insecure: false

service:
  pipelines:
    metrics:
      receivers: [otlp/ghes, prometheus/ghes]
      processors: [batch/ghes]
      exporters: [otlp]

Network and security considerations

Configure your network and security settings based on your chosen monitoring approach.

TLS and certificates

For secure connections to external monitoring systems:

• Use valid TLS certificates when possible
• Configure appropriate tls_config settings in your monitoring system
• Consider using mutual TLS (mTLS) for enhanced security

Next steps

• To configure OpenTelemetry settings, see Configuring OpenTelemetry for your instance.
• To learn more about available metrics, see About OpenTelemetry metrics.