How to debug a backend plugin using Docker

Grafana Plugin Development Share & showcase
1

Docker containers have become a big part in any developer’s life and help in many scenarios from development to production. When I decided to containerize Grafana, my main purpose was to isolate the developer environment from the OS, but it ended up bringing other benefits as well. For example, an easy way of changing Grafana versions in scenarios where I needed to test plugin functionality on different Grafana iterations.

In this post, I’ll describe how to set up a Docker developer environment that lets you attach a debugger to your backend plugin. You’ll extend the official Grafana Docker image with delve, a popular debugger for Go applications, and connect to it remotely from VSCode.

An enjoying debugging experience when developing backend plugins using Docker demands that you address two main challenges:

  • Attaching to a process inside a Docker container from an IDE on the host.
  • Reattaching to the new process whenever the plugin restarts.

This post mostly focuses on the former, with the latter still requiring some manual work.

You might also be interested in other posts on the topic, such as How to use Docker for plugin development and How to add live reload for panel plugins.

Set up the environment

To debug a Go application in a Docker container, you need to install the debugger (delve) inside the container. In this step, you’ll create a Grafana Dockerfile with delve installed, and a Docker Compose configuration that allows you to debug remotely from the host.

This post assumes the following folder structure:

docker-grafana
│   Dockerfile
│   docker-compose.yaml
│
└───plugins
│   │   grafana-k6-app
│   │   github-datasource
│
└───config
    │
    │ config.ini
  • The Dockerfile and docker-compose.yaml in the top-level folder.
  • The plugins folder contains the Grafana plugins.
  • The config folder contains the config.ini and any custom configuration.

The Dockerfile extends the Grafana image with the following:

  • mage, for building the backend plugin.
  • delve, for debugging.
  • Go, for building both mage and delve.
  • A README with instructions on how to start delve.

Dockerfile

FROM grafana/grafana:8.4.5-ubuntu

USER root
WORKDIR /root

RUN apt-get -y update
RUN apt-get -y install git build-essential

RUN curl -L https://golang.org/dl/go1.18.linux-amd64.tar.gz > go1.18.linux-amd64.tar.gz

RUN rm -rf /usr/local/go && \
    tar -C /usr/local -xzf go1.18.linux-amd64.tar.gz

RUN touch README; printf "~~~~~~ START THE DLV SERVER WITH THIS COMMAND BEFORE RUNNING IDE DEBUGGER ~~~~~~ \r\ndlv attach --headless --listen=:2345 PID\r\n\r\n" >> README

RUN echo "export PATH=$PATH:/usr/local/go/bin:~/go/bin" >> ~/.bashrc
RUN echo "cat ~/README" >> ~/.bashrc

RUN /usr/local/go/bin/go install github.com/go-delve/delve/cmd/dlv@latest
RUN git clone https://github.com/magefile/mage; \
    cd mage; \
    export PATH=$PATH:/usr/local/go/bin; \
    go run bootstrap.go

The docker-compose.yaml configures the environment, such as Docker network and the folders to mount inside the container.

docker-compose.yaml

version: "3"

networks:
  grafana:

services:
  grafana-oss:
    build: .
    cap_add:
      - SYS_PTRACE
    security_opt:
      - seccomp:unconfined
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - 3000:3000
      - 2345:2345
    networks:
      - grafana
    container_name: grafana-oss
    command: --config /var/lib/grafana/config.ini
    volumes:
      - ./data/grafana-oss:/var/lib/grafana
      - ./config/config.ini:/var/lib/grafana/config.ini
      - ./plugins/grafana-k6-app:/var/lib/grafana/plugins/grafana-k6-app
      - ./plugins/github-datasource:/var/lib/grafana/plugins/github-datasource

  • The following allows debugging within the container:

      cap_add:
    - SYS_PTRACE
  security_opt:
    - seccomp:unconfined

  • extra_hosts allows the container to connect to services on the host machine. This allows Grafana to connect to, for example, an Oracle SQL Server located on the host machine when setting up the Oracle data source.

  • ports exposes 3000 for the Grafana instance and 2345 for delve.

  • volumes mounts the plugins found in the plugins folder, the configuration file, and the license file. Keeping - ./data/grafana-oss:/var/lib/grafana as a volume saves any progress in the Grafana instance so that everything will still be in place when you restart the container.

In the config.ini file, you can add any configuration you need for your Grafana instance.

config.ini

app_mode = development
instance_name = grafana-oss

[plugins]
enable_alpha = true
app_tls_skip_verify_insecure = false
allow_loading_unsigned_plugins = github-datasource,grafana-k6-app

[auth]
login_cookie_name = grafana_oss_session

[panels]
disable_sanitize_html = false

The environment is ready, and you can now start developing your plugins! Go ahead and fire up the environment by running the following:

docker-compose up -d

You can now access Grafana on community.grafana.com.

Build your plugin with debug information

Now that you have a developer environment running, the next step is to build the plugin with debug information. By including debug information in your build, you’ll drastically improve your debugging experience.

To build your plugin with debug information:

  1. Connect to bash inside the running container:

    docker exec -ti grafana-oss bash

  2. Navigate to the plugin folder:

    cd /var/lib/grafana/plugins/github-datasource

  3. Build the backend plugin with debug information:

    mage build:debug

  4. Reload the backend plugin:

    mage reloadPlugin

Your backend plugin is now running with debug information!

Attach the debugger to the plugin

Before you can start debugging, you need to attach delve to the plugin you want to debug.

To attach delve to a process, you need to know its process ID (PID).

  1. In your terminal, run top.
  2. In the list of processes, look for your plugin, e.g. gpx_github-datasource.
  3. Copy the PID, and then exit top.

To attach the debugger to the plugin process, run the following command:

dlv attach --headless --listen=:2345 <PID>
  • The headless flag tells delve to attach to the backend plugin process in headless mode, which lets VS Code remotely connect to delve through a container port (:2345).

You’re now ready to return to your IDE and connect to the delve debugger!

Connect to the debugger from VSCode

The following instructions are for VSCode, but many other IDEs support debugging Go applications using delve.

  1. In the root folder of your plugin, create a file called .vscode/launch.json with the following content:

    {
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Debug in Container",
      "type": "go",
      "request": "attach",
      "mode": "remote",
      "remotePath": "/var/lib/grafana/plugins/github-datasource/",
      "port": 2345,
      "host": "127.0.0.1",
      "apiVersion": 1,
      "trace": "verbose"
    }
  ]
}

  2. Replace github-datasource in the remotePath to the name of your plugin folder. This needs to be a path within the container.

  3. In VSCode, click Run and Debug → Debug in Container to start debugging. Note that the plugin will hang whenever it arrives at a breakpoint until you resume execution.

For more information on how to debug an application using VSCode, refer to Debugging.

Improving the debugging experience for Grafana plugins

In this post, you’ve learned how to debug a running backend plugin using VSCode, using Docker and the delve debugger.

The main limitation of this workflow is the lack of support for hot reloading. I imagine that you could build some sort of watchdog that detects changes in your code and runs a script that rebuilds and reloads the plugin and then automatically reattaches delve to it. Or even better: a way for delve to manage the process lifecycle itself based on code changes. I’ve tried the latter approach in the past, without success.

If you have an idea of how to enable hot reloading, please let me know!

Happy debugging!

5 Likes