# How to debug a backend plugin using Docker

**URL:** <https://community.grafana.com/t/how-to-debug-a-backend-plugin-using-docker/63503>\
**Category:** Share & showcase\
**Created:** [April 5, 2022, 12:19pm UTC](https://community.grafana.com/t/how-to-debug-a-backend-plugin-using-docker/63503 "2022-04-05T12:19:01Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![victormarin](https://avatars.discourse-cdn.com/v4/letter/v/258eb7/32.png) [@victormarin](https://community.grafana.com/u/victormarin)\
**Post date:** [April 5, 2022, 12:19pm UTC](https://community.grafana.com/t/how-to-debug-a-backend-plugin-using-docker/63503/1 "2022-04-05T12:19:01Z")

</div>

Docker containers have become a big part in any developer’s life and help in many scenarios from development to production. When I decided to containerize Grafana, my main purpose was to isolate the developer environment from the OS, but it ended up bringing other benefits as well. For example, an easy way of changing Grafana versions in scenarios where I needed to test plugin functionality on different Grafana iterations.

In this post, I’ll describe how to set up a Docker developer environment that lets you attach a debugger to your backend plugin. You’ll extend the official Grafana Docker image with [delve](https://github.com/go-delve/delve), a popular debugger for Go applications, and connect to it remotely from VSCode.

An enjoying debugging experience when developing backend plugins using Docker demands that you address two main challenges:

- Attaching to a process inside a Docker container from an IDE on the host.
- Reattaching to the new process whenever the plugin restarts.

This post mostly focuses on the former, with the latter still requiring some manual work.

You might also be interested in other posts on the topic, such as [How to use Docker for plugin development](https://community.grafana.com/t/how-i-use-docker-for-plugin-development/58533) and [How to add live reload for panel plugins](https://community.grafana.com/t/docker-for-plugin-development-with-live-reload/60239).

## Set up the environment

To debug a Go application in a Docker container, you need to install the debugger (delve) inside the container. In this step, you’ll create a Grafana Dockerfile with delve installed, and a Docker Compose configuration that allows you to debug remotely from the host.

This post assumes the following folder structure:

```auto
docker-grafana
│ Dockerfile
│ docker-compose.yaml
│
└───plugins
│ │ grafana-k6-app
│ │ github-datasource
│
└───config
    │
    │ config.ini

```

- The `Dockerfile` and `docker-compose.yaml` in the top-level folder.
- The `plugins` folder contains the Grafana plugins.
- The `config` folder contains the `config.ini` and any custom configuration.

The `Dockerfile` extends the Grafana image with the following:

- mage, for building the backend plugin.
- delve, for debugging.
- Go, for building both mage and delve.
- A `README` with instructions on how to start `delve`.

**Dockerfile**

```dockerfile
FROM grafana/grafana:8.4.5-ubuntu

USER root
WORKDIR /root

RUN apt-get -y update
RUN apt-get -y install git build-essential

RUN curl -L https://golang.org/dl/go1.18.linux-amd64.tar.gz > go1.18.linux-amd64.tar.gz

RUN rm -rf /usr/local/go && \
    tar -C /usr/local -xzf go1.18.linux-amd64.tar.gz

RUN touch README; printf " ~~~~~~START THE DLV SERVER WITH THIS COMMAND BEFORE RUNNING IDE DEBUGGER~~~~~~ \r\ndlv attach --headless --listen=:2345 PID\r\n\r\n" >> README

RUN echo "export PATH=$PATH:/usr/local/go/bin:~/go/bin" >> ~/.bashrc
RUN echo "cat ~/README" >> ~/.bashrc

RUN /usr/local/go/bin/go install github.com/go-delve/delve/cmd/dlv@latest
RUN git clone https://github.com/magefile/mage; \
    cd mage; \
    export PATH=$PATH:/usr/local/go/bin; \
    go run bootstrap.go

```

The `docker-compose.yaml` configures the environment, such as Docker network and the folders to mount inside the container.

**docker-compose.yaml**

```yaml
version: "3"

networks:
  grafana:

services:
  grafana-oss:
    build: .
    cap_add:
      - SYS_PTRACE
    security_opt:
      - seccomp:unconfined
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - 3000:3000
      - 2345:2345
    networks:
      - grafana
    container_name: grafana-oss
    command: --config /var/lib/grafana/config.ini
    volumes:
      - ./data/grafana-oss:/var/lib/grafana
      - ./config/config.ini:/var/lib/grafana/config.ini
      - ./plugins/grafana-k6-app:/var/lib/grafana/plugins/grafana-k6-app
      - ./plugins/github-datasource:/var/lib/grafana/plugins/github-datasource

```

- The following allows debugging within the container:

- `extra_hosts` allows the container to connect to services on the host machine. This allows Grafana to connect to, for example, an Oracle SQL Server located on the host machine when setting up the Oracle data source.

- `ports` exposes `3000` for the Grafana instance and `2345` for `delve`.

- `volumes` mounts the plugins found in the `plugins` folder, the configuration file, and the license file. Keeping `- ./data/grafana-oss:/var/lib/grafana` as a volume saves any progress in the Grafana instance so that everything will still be in place when you restart the container.

In the config.ini file, you can add any configuration you need for your Grafana instance.

**config.ini**

```ini
app_mode = development
instance_name = grafana-oss

[plugins]
enable_alpha = true
app_tls_skip_verify_insecure = false
allow_loading_unsigned_plugins = github-datasource,grafana-k6-app

[auth]
login_cookie_name = grafana_oss_session

[panels]
disable_sanitize_html = false

```

The environment is ready, and you can now start developing your plugins! Go ahead and fire up the environment by running the following:

```bash
docker-compose up -d

```

You can now access Grafana on [community.grafana.com](http://grafana.staged-by-discourse.com).

## Build your plugin with debug information

Now that you have a developer environment running, the next step is to build the plugin with debug information. By including debug information in your build, you’ll drastically improve your debugging experience.

To build your plugin with debug information:

1. Connect to bash inside the running container:

2. Navigate to the plugin folder:

3. Build the backend plugin with debug information:

4. Reload the backend plugin:

Your backend plugin is now running with debug information!

## Attach the debugger to the plugin

Before you can start debugging, you need to attach delve to the plugin you want to debug.

To attach delve to a process, you need to know its process ID (PID).

1. In your terminal, run `top`.
2. In the list of processes, look for your plugin, e.g. `gpx_github-datasource`.
3. Copy the PID, and then exit `top`.

To attach the debugger to the plugin process, run the following command:

```bash
dlv attach --headless --listen=:2345 <PID>

```

- The `headless` flag tells delve to attach to the backend plugin process in _headless_ mode, which lets VS Code remotely connect to delve through a container port (:2345).

You’re now ready to return to your IDE and connect to the delve debugger!

## Connect to the debugger from VSCode

The following instructions are for VSCode, but many other IDEs support debugging Go applications using delve.

1. In the root folder of your plugin, create a file called `.vscode/launch.json` with the following content:

2. Replace `github-datasource` in the `remotePath` to the name of your plugin folder. This needs to be a path within the container.

3. In VSCode, click **Run and Debug** → **Debug in Container** to start debugging. Note that the plugin will hang whenever it arrives at a breakpoint until you resume execution.

For more information on how to debug an application using VSCode, refer to [Debugging](https://code.visualstudio.com/docs/editor/debugging).

## Improving the debugging experience for Grafana plugins

In this post, you’ve learned how to debug a running backend plugin using VSCode, using Docker and the delve debugger.

The main limitation of this workflow is the lack of support for hot reloading. I imagine that you could build some sort of watchdog that detects changes in your code and runs a script that rebuilds and reloads the plugin and then automatically reattaches delve to it. Or even better: a way for delve to manage the process lifecycle itself based on code changes. I’ve tried the latter approach in the past, without success.

If you have an idea of how to enable hot reloading, please let me know!

Happy debugging!