Unable to connect to MySQL source using Private Data Connector

Grafana Configuration
1

  • What Grafana version and what operating system are you using?
    Grafana Cloud 10.3

  • What are you trying to achieve?
    Use the MySQL Data Source with a DB that is in a private network, with the help of Grafana Private Data Connector.

  • How are you trying to achieve it?
    PDC agent has been setup and is online, test successful from Grafana Cloud side.

    image
    image1242×927 86.2 KB

MySQL data source is configured with valid database credentials, triple checked and tested from my local machine.

  • What happened?
    When running the test in the Data source configured with the PDC, two errors are shown

There was an error when testing this data source, please check PDC agent logs.

[sqleng.connectionError] failed to connect to server - Please try again. If the issue persists please open a support ticket that notes the time in UTC.

image
image1465×422 28.9 KB

No logs occur on the PDC agent.

  • What did you expect to happen?
    The Data source should have been able to connect to the private database successfully.

  • Can you copy/paste the configuration(s) that you are having problems with?
    N/A

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    Above.

  • Did you follow any online instructions? If so, what is the URL?
    I followed the instructions in the PDC screen. PDC docs are here Private data source connect (PDC) | Grafana Cloud documentation

1 Like
2

Hi @connorhsm, could you please explain how you deploy the pdc agent, which installation method you used like kubernetes or docker and also what are the checks i need to do to achieve the agent connection. I refer the official documentaton in grafana for PDC but i am not clear, so I am asking. It would be really helpful if you also provide me the deployment file you used to deploy the pdc-agent image.

1 Like
3

Hello did you solve this?
I am using windows binary.
but the datasource is not connecting, but different error than yours
image

image
image757×489 42.4 KB

image
image840×558 25.7 KB

It says check pdc logs, but dont know where to look exactly: the command promt where i launched the pdc binary gives this starter logs, but no error logs after failed to connect local mysql datasource.

pdc.exe -token <token created from pdc config page> -cluster prod-ap-south-0 -gcloud-hosted-grafana-id 744478 -ssh-flag='-o PermitRemoteOpen=localhost:3306'
level=info caller=main.go:109 ts=2024-03-01T12:32:21.168871+06:00 msg="PDC agent info" version=v0.0.23 commit=881abe89f842ec98b27d6bfc7371c8333c07d669 date=2023-12-18T13:52:47Z sshversion="OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2\r" os=windows arch=amd64
level=info caller=ssh.go:109 ts=2024-03-01T12:32:21.1697119+06:00 msg="starting ssh client"
level=info caller=keymanager.go:53 ts=2024-03-01T12:32:21.1697119+06:00 msg="starting key manager"
level=info caller=keymanager.go:179 ts=2024-03-01T12:32:21.1702594+06:00 msg="found existing valid certificate"
level=info caller=keymanager.go:192 ts=2024-03-01T12:32:21.1707646+06:00 msg="found valid grafana_pdc_known_hosts"
level=info caller=ssh.go:276 ts=2024-03-01T12:32:23.0818954+06:00 msg="Allocated port 1234 for remote forward to socks:0"
level=info caller=ssh.go:276 ts=2024-03-01T12:32:23.2494344+06:00 msg="This is Grafana Private Datasource Connect!\n"
4

Enable -log.level=debug for pdc agent first.

5

This is the log before hitting “Test Connection” in mysql data source setting:

pdc.exe -log.level=debug -token <my token> -cluster prod-ap-south-0 -gcloud-hosted-grafana-id 744478 -ssh-flag='-o PermitRemoteOpen=localhost:3306'
level=info caller=main.go:109 ts=2024-03-01T14:39:48.437305+06:00 msg="PDC agent info" version=v0.0.23 commit=881abe89f842ec98b27d6bfc7371c8333c07d669 date=2023-12-18T13:52:47Z sshversion="OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2\r" os=windows arch=amd64
level=info caller=ssh.go:109 ts=2024-03-01T14:39:48.4383022+06:00 msg="starting ssh client"
level=info caller=keymanager.go:53 ts=2024-03-01T14:39:48.4388096+06:00 msg="starting key manager"
level=info caller=keymanager.go:170 ts=2024-03-01T14:39:48.4396279+06:00 msg="new certificate required: certificate validity has expired"
level=info caller=keymanager.go:243 ts=2024-03-01T14:39:48.4397858+06:00 msg="generating new certificate"
level=debug caller=client.go:227 ts=2024-03-01T14:39:48.4462155+06:00 msg="performing request" method=POST url=https://private-datasource-connect-api-prod-ap-south-0.grafana.net/pdc/api/v1/sign-public-key
level=debug caller=ssh.go:128 ts=2024-03-01T14:39:48.87051+06:00 msg="parsed flags: [-i C:\\Users\\mahin/.ssh/grafana_pdc 744478@private-datasource-connect-prod-ap-south-0.grafana.net -p 22 -R 0 -o CertificateFile=C:\\Users\\mahin/.ssh/grafana_pdc-cert.pub -o ConnectTimeout=1 -o ServerAliveInterval=15 -o UserKnownHostsFile=C:\\Users\\mahin/.ssh/grafana_pdc_known_hosts -vvv '-o]"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:48.8910939+06:00 msg="OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:48.8910939+06:00 msg="debug3: Failed to open file:C:/Users/mahin/.ssh/config error:2"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:48.8917575+06:00 msg="debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:48.8953423+06:00 msg="debug2: resolving \"private-datasource-connect-prod-ap-south-0.grafana.net\" port 22"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:48.9586804+06:00 msg="debug2: ssh_connect_direct"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:48.9609392+06:00 msg="debug1: Connecting to private-datasource-connect-prod-ap-south-0.grafana.net [34.93.57.132] port 22."
level=info caller=ssh.go:276 ts=2024-03-01T14:39:48.9614453+06:00 msg="debug2: fd 3 setting O_NONBLOCK"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.9702+06:00 msg="debug1: fd 3 clearing O_NONBLOCK"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.9707048+06:00 msg="debug1: Connection established."
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.9726778+06:00 msg="debug1: identity file C:\\\\Users\\\\mahin/.ssh/grafana_pdc type 3"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.9744694+06:00 msg="debug1: certificate file C:\\\\Users\\\\mahin/.ssh/grafana_pdc-cert.pub type 7"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.9745246+06:00 msg="debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.9750898+06:00 msg="debug1: Remote protocol version 2.0, remote software version Go"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.9750898+06:00 msg="debug1: no match: Go"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.9755953+06:00 msg="debug2: fd 3 setting O_NONBLOCK"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.975644+06:00 msg="debug1: Authenticating to private-datasource-connect-prod-ap-south-0.grafana.net:22 as '744478'"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.975644+06:00 msg="debug3: hostkeys_foreach: reading file \"C:\\\\Users\\\\mahin/.ssh/grafana_pdc_known_hosts\""
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.9761667+06:00 msg="debug3: record_hostkey: found ca key type RSA in file C:\\\\Users\\\\mahin/.ssh/grafana_pdc_known_hosts:1"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.9766861+06:00 msg="debug3: load_hostkeys: loaded 1 keys from private-datasource-connect-prod-ap-south-0.grafana.net"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.9772112+06:00 msg="debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.9782753+06:00 msg="debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.9787988+06:00 msg="debug3: send packet: type 20"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:49.9793174+06:00 msg="debug1: SSH2_MSG_KEXINIT sent"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0366757+06:00 msg="debug3: receive packet: type 20"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0519637+06:00 msg="debug1: SSH2_MSG_KEXINIT received"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0541224+06:00 msg="debug2: local client KEXINIT proposal"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0546897+06:00 msg="debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0599325+06:00 msg="debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0605334+06:00 msg="debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0611859+06:00 msg="debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0622629+06:00 msg="debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0628846+06:00 msg="debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0628846+06:00 msg="debug2: compression ctos: none,zlib@openssh.com,zlib"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0640231+06:00 msg="debug2: compression stoc: none,zlib@openssh.com,zlib"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0645791+06:00 msg="debug2: languages ctos: "
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0651342+06:00 msg="debug2: languages stoc: "
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0656858+06:00 msg="debug2: first_kex_follows 0 "
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0662489+06:00 msg="debug2: reserved 0 "
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0668295+06:00 msg="debug2: peer server KEXINIT proposal"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0743076+06:00 msg="debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,kex-strict-s-v00@openssh.com"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0755983+06:00 msg="debug2: host key algorithms: rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0755983+06:00 msg="debug2: ciphers ctos: aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0772157+06:00 msg="debug2: ciphers stoc: aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0772157+06:00 msg="debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0772157+06:00 msg="debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0772157+06:00 msg="debug2: compression ctos: none"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0772157+06:00 msg="debug2: compression stoc: none"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0803286+06:00 msg="debug2: languages ctos: "
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0814101+06:00 msg="debug2: languages stoc: "
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0819492+06:00 msg="debug2: first_kex_follows 0 "
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0824765+06:00 msg="debug2: reserved 0 "
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.083148+06:00 msg="debug1: kex: algorithm: curve25519-sha256"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0910118+06:00 msg="debug1: kex: host key algorithm: rsa-sha2-512-cert-v01@openssh.com"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0929649+06:00 msg="debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0935293+06:00 msg="debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0952221+06:00 msg="debug3: send packet: type 30"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.0962648+06:00 msg="debug1: expecting SSH2_MSG_KEX_ECDH_REPLY"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2527846+06:00 msg="debug3: receive packet: type 31"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2549573+06:00 msg="debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com SHA256:JYZn+FpH6nXKlbBT4ER1lmiaEhzyC6zdAkUEF+hqOI8, serial 0 ID \"\" CA ssh-rsa SHA256:7Mrj7jER3YBUy5LsAL9gMiCE6v6aYFSTzRHye20AL8w valid forever"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2600488+06:00 msg="debug2: Server host certificate hostname: private-datasource-connect-prod-ap-south-0.grafana.net"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2605983+06:00 msg="debug3: hostkeys_foreach: reading file \"C:\\\\Users\\\\mahin/.ssh/grafana_pdc_known_hosts\""
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2617383+06:00 msg="debug3: record_hostkey: found ca key type RSA in file C:\\\\Users\\\\mahin/.ssh/grafana_pdc_known_hosts:1"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2622971+06:00 msg="debug3: load_hostkeys: loaded 1 keys from private-datasource-connect-prod-ap-south-0.grafana.net"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2628607+06:00 msg="debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2634673+06:00 msg="debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2646144+06:00 msg="debug1: Host 'private-datasource-connect-prod-ap-south-0.grafana.net' is known and matches the RSA-CERT host certificate."
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2651641+06:00 msg="debug1: Found CA key in C:\\\\Users\\\\mahin/.ssh/grafana_pdc_known_hosts:1"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2776844+06:00 msg="debug3: send packet: type 21"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2776844+06:00 msg="debug2: set_newkeys: mode 1"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2787836+06:00 msg="debug1: rekey out after 134217728 blocks"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2805897+06:00 msg="debug1: SSH2_MSG_NEWKEYS sent"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2811236+06:00 msg="debug1: expecting SSH2_MSG_NEWKEYS"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2820043+06:00 msg="debug3: receive packet: type 21"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2825461+06:00 msg="debug1: SSH2_MSG_NEWKEYS received"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2830888+06:00 msg="debug2: set_newkeys: mode 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2841652+06:00 msg="debug1: rekey in after 134217728 blocks"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2848246+06:00 msg="debug3: unable to connect to pipe \\\\\\\\.\\\\pipe\\\\openssh-ssh-agent, error: 2"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2864585+06:00 msg="debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2930971+06:00 msg="debug1: Will attempt key: C:\\\\Users\\\\mahin/.ssh/grafana_pdc-cert.pub ED25519-CERT SHA256:eN9i9NLI8B/6MEPmi/T72UEQkiDz28wBB3ntf+1n3hE explicit"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2976628+06:00 msg="debug1: Will attempt key: C:\\\\Users\\\\mahin/.ssh/grafana_pdc ED25519 SHA256:eN9i9NLI8B/6MEPmi/T72UEQkiDz28wBB3ntf+1n3hE explicit"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2982389+06:00 msg="debug2: pubkey_prepare: done"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.2999574+06:00 msg="debug3: send packet: type 5"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.3006118+06:00 msg="debug3: receive packet: type 7"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.3006118+06:00 msg="debug1: SSH2_MSG_EXT_INFO received"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.3095704+06:00 msg="debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss>"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.3121157+06:00 msg="debug1: kex_input_ext_info: ping@openssh.com (unrecognised)"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.4697361+06:00 msg="debug3: receive packet: type 6"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.4725292+06:00 msg="debug2: service_accept: ssh-userauth"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.4744459+06:00 msg="debug1: SSH2_MSG_SERVICE_ACCEPT received"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.4750559+06:00 msg="debug3: send packet: type 50"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.536384+06:00 msg="debug3: receive packet: type 51"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.5578091+06:00 msg="debug1: Authentications that can continue: publickey"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.5583639+06:00 msg="debug3: start over, passed a different list publickey"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.5594681+06:00 msg="debug3: preferred publickey,keyboard-interactive,password"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.5600092+06:00 msg="debug3: authmethod_lookup publickey"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.5605695+06:00 msg="debug3: remaining preferred: keyboard-interactive,password"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.5610731+06:00 msg="debug3: authmethod_is_enabled publickey"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.5617464+06:00 msg="debug1: Next authentication method: publickey"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.5623497+06:00 msg="debug1: Offering public key: C:\\\\Users\\\\mahin/.ssh/grafana_pdc-cert.pub ED25519-CERT SHA256:eN9i9NLI8B/6MEPmi/T72UEQkiDz28wBB3ntf+1n3hE explicit"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.5628752+06:00 msg="debug3: send packet: type 50"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.5633995+06:00 msg="debug2: we sent a publickey packet, wait for reply"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.6366069+06:00 msg="debug3: receive packet: type 60"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.6448742+06:00 msg="debug1: Server accepts key: C:\\\\Users\\\\mahin/.ssh/grafana_pdc-cert.pub ED25519-CERT SHA256:eN9i9NLI8B/6MEPmi/T72UEQkiDz28wBB3ntf+1n3hE explicit"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.6449352+06:00 msg="debug3: sign_and_send_pubkey: ED25519-CERT SHA256:eN9i9NLI8B/6MEPmi/T72UEQkiDz28wBB3ntf+1n3hE"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.649776+06:00 msg="debug2: sign_and_send_pubkey: using private key \"C:\\\\Users\\\\mahin/.ssh/grafana_pdc-cert.pub\" for certificate"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.6503359+06:00 msg="debug3: sign_and_send_pubkey: signing using ssh-ed25519-cert-v01@openssh.com"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.6537087+06:00 msg="debug3: send packet: type 50"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.7198008+06:00 msg="debug3: receive packet: type 52"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.7313969+06:00 msg="debug1: Authentication succeeded (publickey)."
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.7333875+06:00 msg="Authenticated to private-datasource-connect-prod-ap-south-0.grafana.net ([34.93.57.132]:22)."
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.7391939+06:00 msg="debug1: Remote connections from LOCALHOST:0 forwarded to local address socks:0"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.7391939+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.7405246+06:00 msg="debug2: fd 5 setting O_NONBLOCK"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.7410954+06:00 msg="debug2: fd 6 setting O_NONBLOCK"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.7418016+06:00 msg="debug1: channel 0: new [client-session]"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.7423667+06:00 msg="debug3: ssh_session2_open: channel_new: 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.742924+06:00 msg="debug2: channel 0: send open"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.7434474+06:00 msg="debug3: send packet: type 90"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.7445289+06:00 msg="debug1: Entering interactive session."
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.7450572+06:00 msg="debug1: pledge: network"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.8451784+06:00 msg="debug3: receive packet: type 81"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.8453777+06:00 msg="debug1: remote forward success for: listen 0, connect socks:0"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.8468383+06:00 msg="Allocated port 1234 for remote forward to socks:0"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.8475068+06:00 msg="debug1: Updating allowed port 1234 for forwarding to host socks port 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.8532069+06:00 msg="debug1: All remote forwarding requests processed"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.9199558+06:00 msg="debug3: receive packet: type 91"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.9216862+06:00 msg="debug2: channel_input_open_confirmation: channel 0: callback start"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.9228033+06:00 msg="debug2: fd 3 setting TCP_NODELAY"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.9280071+06:00 msg="debug2: client_session2_setup: id 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.9285454+06:00 msg="debug1: Sending command: '-o"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.9292174+06:00 msg="debug2: channel 0: request exec confirm 1"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.929778+06:00 msg="debug3: send packet: type 98"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.9303437+06:00 msg="debug2: channel_input_open_confirmation: channel 0: callback done"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:50.9309625+06:00 msg="debug2: channel 0: open confirm rwindow 2097152 rmax 32768"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:51.0074733+06:00 msg="debug3: receive packet: type 99"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:51.0074733+06:00 msg="debug2: channel_input_status_confirm: type 99 id 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:51.0142266+06:00 msg="debug2: exec request accepted on channel 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:39:51.0147958+06:00 msg="This is Grafana Private Datasource Connect!\n"
level=info caller=ssh.go:276 ts=2024-03-01T14:40:06.0305082+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:40:06.1046814+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:40:21.1080637+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:40:21.1850102+06:00 msg="debug3: receive packet: type 82"

(PDC config page in grafana says: 1 agent connected.)

The following log is after clicking Save and Test in ‘data source’ error: driver: bad connection.
I dont now what is going on but it seems like I am getting this erorr socketio shutdown - ERROR:10057. Any idea whats wrong? My firewall is turned off.

... ... ... 
level=info caller=ssh.go:276 ts=2024-03-01T14:39:51.0147958+06:00 msg="This is Grafana Private Datasource Connect!\n"
level=info caller=ssh.go:276 ts=2024-03-01T14:40:06.0305082+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:40:06.1046814+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:40:21.1080637+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:40:21.1850102+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:40:36.1869079+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:40:41.2049232+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:40:51.5697391+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:40:51.6460165+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:41:06.6549873+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:41:06.7314664+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:41:21.7336381+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:41:21.8101326+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:41:36.8113025+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:41:36.8871071+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:41:51.8896258+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:41:51.9542053+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:06.9812581+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:07.0486535+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:22.0753551+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:22.1515694+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:31.1721844+06:00 msg="debug3: receive packet: type 90"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:31.1749714+06:00 msg="debug1: client_input_channel_open: ctype forwarded-tcpip rchan 1 win 2097152 max 32768"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:31.1798162+06:00 msg="debug1: client_request_forwarded_tcpip: listen localhost port 1234, originator 1.2.3.4 port 4567"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:31.1803753+06:00 msg="debug1: channel 1: new [1.2.3.4]"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:31.180936+06:00 msg="debug1: confirm forwarded-tcpip"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:31.1820622+06:00 msg="debug3: send packet: type 91"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:31.1826818+06:00 msg="debug2: channel 1: pre_rdynamic: have 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:31.1837507+06:00 msg="debug2: channel 1: pre_rdynamic: have 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.2438992+06:00 msg="debug2: channel 1: pre_rdynamic: have 4"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.2438992+06:00 msg="debug2: channel 1: decode socks5"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.2462227+06:00 msg="debug2: channel 1: socks5 auth done"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.2462775+06:00 msg="debug2: channel 1: pre_rdynamic: need more"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.2513236+06:00 msg="debug2: channel 1: pre_rdynamic: have 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.7391706+06:00 msg="debug2: channel 1: pre_rdynamic: have 16"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.7485608+06:00 msg="debug2: channel 1: decode socks5"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.7504922+06:00 msg="debug2: channel 1: socks5 post auth"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.7510016+06:00 msg="debug2: channel 1: dynamic request: socks5 host localhost port 3306 command 1"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.7570224+06:00 msg="debug2: fd 7 setting O_NONBLOCK"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.7587918+06:00 msg="debug3: socketio_getsockopt - ERROR:10022"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.7594232+06:00 msg="debug1: getsockopt TCP_NODELAY: Invalid argument"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.7605313+06:00 msg="debug1: connect_next: host localhost ([::1]:3306) in progress, fd=7"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.7610369+06:00 msg="debug3: fd 7 is O_NONBLOCK"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.7616278+06:00 msg="debug3: fd 7 is O_NONBLOCK"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.7622874+06:00 msg="debug3: channel 1: waiting for connection"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8043279+06:00 msg="debug3: finish_connect - ERROR: async io completed with error: 10061, io:000002B168607E10"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8045985+06:00 msg="debug1: channel 1: connected to localhost port 3306"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8112265+06:00 msg="debug3: recv - from CB ERROR:10061, io:000002B168607E10"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8112265+06:00 msg="debug2: channel 1: read<=0 rfd 7 len 4294967295"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8128731+06:00 msg="debug2: channel 1: read failed"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8134278+06:00 msg="debug2: channel 1: chan_shutdown_read (i0 o0 sock 7 wfd 7 efd -1 [closed])"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8139696+06:00 msg="debug3: socketio_shutdown - ERROR:10057"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8150181+06:00 msg="debug2: channel 1: input open -> drain"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8156881+06:00 msg="debug2: channel 1: ibuf empty"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8162862+06:00 msg="debug2: channel 1: send eof"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8168187+06:00 msg="debug3: send packet: type 96"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8178667+06:00 msg="debug2: channel 1: input drain -> closed"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.887703+06:00 msg="debug3: receive packet: type 97"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8935178+06:00 msg="debug2: channel 1: rcvd close"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8958574+06:00 msg="debug2: channel 1: output open -> drain"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8970695+06:00 msg="debug3: channel 1: will not send data after close"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.901232+06:00 msg="debug2: channel 1: obuf empty"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.9018035+06:00 msg="debug2: channel 1: chan_shutdown_write (i3 o1 sock 7 wfd 7 efd -1 [closed])"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.9023692+06:00 msg="debug3: socketio_shutdown - ERROR:10057"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.9057596+06:00 msg="debug2: channel 1: chan_shutdown_write: shutdown() failed for fd 7 [i3 o1]: The socket is not connected"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.9060514+06:00 msg="debug2: channel 1: output drain -> closed"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.9068771+06:00 msg="debug2: channel 1: send close"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.9075973+06:00 msg="debug3: send packet: type 97"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.9083048+06:00 msg="debug2: channel 1: is dead"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.9091319+06:00 msg="debug2: channel 1: garbage collecting"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.909685+06:00 msg="debug1: channel 1: free: 1.2.3.4, nchannels 2"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.9103342+06:00 msg="debug3: channel 1: status: The following connections are open:"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.9108678+06:00 msg="  #0 client-session (t4 r0 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.9114781+06:00 msg="  #1 1.2.3.4 (t4 r1 i3/0 o3/0 e[closed]/0 fd 7/7/-1 sock 7 cc -1)"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.2993166+06:00 msg="debug3: receive packet: type 90"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.2993166+06:00 msg="debug1: client_input_channel_open: ctype forwarded-tcpip rchan 2 win 2097152 max 32768"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.3011894+06:00 msg="debug1: client_request_forwarded_tcpip: listen localhost port 1234, originator 1.2.3.4 port 4567"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.3064895+06:00 msg="debug1: channel 1: new [1.2.3.4]"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.3070277+06:00 msg="debug1: confirm forwarded-tcpip"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.3082024+06:00 msg="debug3: send packet: type 91"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.3087541+06:00 msg="debug2: channel 1: pre_rdynamic: have 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.3098799+06:00 msg="debug2: channel 1: pre_rdynamic: have 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.370766+06:00 msg="debug2: channel 1: pre_rdynamic: have 4"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.3853633+06:00 msg="debug2: channel 1: decode socks5"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.391752+06:00 msg="debug2: channel 1: socks5 auth done"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.3923251+06:00 msg="debug2: channel 1: pre_rdynamic: need more"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.3934623+06:00 msg="debug2: channel 1: pre_rdynamic: have 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.4543146+06:00 msg="debug2: channel 1: pre_rdynamic: have 16"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.4700016+06:00 msg="debug2: channel 1: decode socks5"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.4702868+06:00 msg="debug2: channel 1: socks5 post auth"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.4744454+06:00 msg="debug2: channel 1: dynamic request: socks5 host localhost port 3306 command 1"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.4761507+06:00 msg="debug2: fd 7 setting O_NONBLOCK"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.4767078+06:00 msg="debug3: socketio_getsockopt - ERROR:10022"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.4767078+06:00 msg="debug1: getsockopt TCP_NODELAY: Invalid argument"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.4778802+06:00 msg="debug1: connect_next: host localhost ([::1]:3306) in progress, fd=7"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.4784405+06:00 msg="debug3: fd 7 is O_NONBLOCK"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.4790008+06:00 msg="debug3: fd 7 is O_NONBLOCK"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:35.4795565+06:00 msg="debug3: channel 1: waiting for connection"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.5193972+06:00 msg="debug3: finish_connect - ERROR: async io completed with error: 10061, io:000002B168607E10"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.5198519+06:00 msg="debug1: channel 1: connected to localhost port 3306"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.5227941+06:00 msg="debug3: recv - from CB ERROR:10061, io:000002B168607E10"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.5228545+06:00 msg="debug2: channel 1: read<=0 rfd 7 len 4294967295"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.5280864+06:00 msg="debug2: channel 1: read failed"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.5291148+06:00 msg="debug2: channel 1: chan_shutdown_read (i0 o0 sock 7 wfd 7 efd -1 [closed])"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.5291148+06:00 msg="debug3: socketio_shutdown - ERROR:10057"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.5297733+06:00 msg="debug2: channel 1: input open -> drain"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.5307171+06:00 msg="debug2: channel 1: ibuf empty"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.5313042+06:00 msg="debug2: channel 1: send eof"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.5318247+06:00 msg="debug3: send packet: type 96"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.5323873+06:00 msg="debug2: channel 1: input drain -> closed"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6030267+06:00 msg="debug3: receive packet: type 97"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6104618+06:00 msg="debug2: channel 1: rcvd close"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6104618+06:00 msg="debug2: channel 1: output open -> drain"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6157132+06:00 msg="debug3: channel 1: will not send data after close"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6162789+06:00 msg="debug2: channel 1: obuf empty"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6168112+06:00 msg="debug2: channel 1: chan_shutdown_write (i3 o1 sock 7 wfd 7 efd -1 [closed])"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6173754+06:00 msg="debug3: socketio_shutdown - ERROR:10057"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6212737+06:00 msg="debug2: channel 1: chan_shutdown_write: shutdown() failed for fd 7 [i3 o1]: The socket is not connected"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6236279+06:00 msg="debug2: channel 1: output drain -> closed"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6241944+06:00 msg="debug2: channel 1: send close"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6291347+06:00 msg="debug3: send packet: type 97"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6291347+06:00 msg="debug2: channel 1: is dead"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6306802+06:00 msg="debug2: channel 1: garbage collecting"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6311884+06:00 msg="debug1: channel 1: free: 1.2.3.4, nchannels 2"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6319506+06:00 msg="debug3: channel 1: status: The following connections are open:"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6320896+06:00 msg="  #0 client-session (t4 r0 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6326227+06:00 msg="  #1 1.2.3.4 (t4 r2 i3/0 o3/0 e[closed]/0 fd 7/7/-1 sock 7 cc -1)"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6441113+06:00 msg="debug3: receive packet: type 90"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6452522+06:00 msg="debug1: client_input_channel_open: ctype forwarded-tcpip rchan 1 win 2097152 max 32768"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6463809+06:00 msg="debug1: client_request_forwarded_tcpip: listen localhost port 1234, originator 1.2.3.4 port 4567"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6471563+06:00 msg="debug1: channel 1: new [1.2.3.4]"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6477357+06:00 msg="debug1: confirm forwarded-tcpip"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6482895+06:00 msg="debug3: send packet: type 91"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6489171+06:00 msg="debug2: channel 1: pre_rdynamic: have 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.6501899+06:00 msg="debug2: channel 1: pre_rdynamic: have 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.720034+06:00 msg="debug2: channel 1: pre_rdynamic: have 4"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.7254549+06:00 msg="debug2: channel 1: decode socks5"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.7270735+06:00 msg="debug2: channel 1: socks5 auth done"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.7316884+06:00 msg="debug2: channel 1: pre_rdynamic: need more"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.7333531+06:00 msg="debug2: channel 1: pre_rdynamic: have 0"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.803395+06:00 msg="debug2: channel 1: pre_rdynamic: have 16"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.8099654+06:00 msg="debug2: channel 1: decode socks5"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.8099654+06:00 msg="debug2: channel 1: socks5 post auth"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.8148666+06:00 msg="debug2: channel 1: dynamic request: socks5 host localhost port 3306 command 1"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.8157012+06:00 msg="debug2: fd 7 setting O_NONBLOCK"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.8170604+06:00 msg="debug3: socketio_getsockopt - ERROR:10022"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.8177324+06:00 msg="debug1: getsockopt TCP_NODELAY: Invalid argument"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.8184138+06:00 msg="debug1: connect_next: host localhost ([::1]:3306) in progress, fd=7"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.8273509+06:00 msg="debug3: fd 7 is O_NONBLOCK"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.8278589+06:00 msg="debug3: fd 7 is O_NONBLOCK"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:37.8284845+06:00 msg="debug3: channel 1: waiting for connection"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.8516045+06:00 msg="debug3: finish_connect - ERROR: async io completed with error: 10061, io:000002B168607E10"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.8545903+06:00 msg="debug1: channel 1: connected to localhost port 3306"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.8545903+06:00 msg="debug3: recv - from CB ERROR:10061, io:000002B168607E10"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.8594237+06:00 msg="debug2: channel 1: read<=0 rfd 7 len 4294967295"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.8602876+06:00 msg="debug2: channel 1: read failed"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.8609829+06:00 msg="debug2: channel 1: chan_shutdown_read (i0 o0 sock 7 wfd 7 efd -1 [closed])"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.8618834+06:00 msg="debug3: socketio_shutdown - ERROR:10057"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.8625126+06:00 msg="debug2: channel 1: input open -> drain"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.8630441+06:00 msg="debug2: channel 1: ibuf empty"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.8636539+06:00 msg="debug2: channel 1: send eof"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.8641767+06:00 msg="debug3: send packet: type 96"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.864848+06:00 msg="debug2: channel 1: input drain -> closed"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9410561+06:00 msg="debug3: receive packet: type 97"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9410561+06:00 msg="debug2: channel 1: rcvd close"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9431136+06:00 msg="debug2: channel 1: output open -> drain"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9482733+06:00 msg="debug3: channel 1: will not send data after close"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.94885+06:00 msg="debug2: channel 1: obuf empty"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9500123+06:00 msg="debug2: channel 1: chan_shutdown_write (i3 o1 sock 7 wfd 7 efd -1 [closed])"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9505701+06:00 msg="debug3: socketio_shutdown - ERROR:10057"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9586531+06:00 msg="debug2: channel 1: chan_shutdown_write: shutdown() failed for fd 7 [i3 o1]: The socket is not connected"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9592207+06:00 msg="debug2: channel 1: output drain -> closed"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9601396+06:00 msg="debug2: channel 1: send close"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9607074+06:00 msg="debug3: send packet: type 97"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9612969+06:00 msg="debug2: channel 1: is dead"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9622697+06:00 msg="debug2: channel 1: garbage collecting"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9627759+06:00 msg="debug1: channel 1: free: 1.2.3.4, nchannels 2"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9634167+06:00 msg="debug3: channel 1: status: The following connections are open:"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9641354+06:00 msg="  #0 client-session (t4 r0 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:39.9646444+06:00 msg="  #1 1.2.3.4 (t4 r1 i3/0 o3/0 e[closed]/0 fd 7/7/-1 sock 7 cc -1)"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:54.9773181+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:55.0532086+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:43:10.0729546+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:43:10.4505359+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:43:25.467107+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:43:25.8341197+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:43:40.8447203+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:43:40.9114072+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:43:55.9217633+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:44:10.5822411+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:44:11.016089+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:44:11.0825267+06:00 msg="debug3: receive packet: type 82"
level=info caller=ssh.go:276 ts=2024-03-01T14:44:26.0922518+06:00 msg="debug3: send packet: type 80"
level=info caller=ssh.go:276 ts=2024-03-01T14:44:26.7763459+06:00 msg="debug3: receive packet: type 82"
6

Are you sure that MySQL is listening on localhost interface and on the same localhost as PDC agent has available.
It may look simple, but each container, VM,… has own localhost usually,so it can be very complex from the network perspective.

7

Hello,

I changed localhost:3306 to 127.0.0.1:3306 and it worked, dont know why this worked.

What I found confusing was the logs. the 3rd line already said that “connected to localhost”, how one should interpret this as “lets change localhost to 127.0.0.1 and see what happens” ?

level=info caller=ssh.go:276 ts=2024-03-01T14:42:32.7622874+06:00 msg="debug3: channel 1: waiting for connection"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8043279+06:00 msg="debug3: finish_connect - ERROR: async io completed with error: 10061, io:000002B168607E10"
-----------------level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8045985+06:00 msg="debug1: channel 1: connected to localhost port 3306"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8112265+06:00 msg="debug3: recv - from CB ERROR:10061, io:000002B168607E10"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8112265+06:00 msg="debug2: channel 1: read<=0 rfd 7 len 4294967295"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8128731+06:00 msg="debug2: channel 1: read failed"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8134278+06:00 msg="debug2: channel 1: chan_shutdown_read (i0 o0 sock 7 wfd 7 efd -1 [closed])"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8139696+06:00 msg="debug3: socketio_shutdown - ERROR:10057"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8150181+06:00 msg="debug2: channel 1: input open -> drain"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8156881+06:00 msg="debug2: channel 1: ibuf empty"
level=info caller=ssh.go:276 ts=2024-03-01T14:42:34.8162862+06:00 msg="debug2: channel 1: send eof"

Note that in pdc configuration “localhost” was explicitly mentioned:

image
image1235×494 48.5 KB

Another minor thing is the level=info part. After specifying -log.level=debug for pdc agent logs, I do get detailed logs but All the lines are prefixed with level=info :thinking:

8

Always restart grafana for config changes

That is the case with amost any product where you change some configuration

9

Not sure what you mean by restarting grafana, I am running on grafana cloud… kindly consider elaborating.

I seem to have found the issue is name resolution issue for “localhost” and my confusion caused by the log that said “connected to localhost”.

I just tried data source configuration again with localhost:3306 and it fails, as before. only 127.0.0.1:3306 works

Update: localhost resolution issue seems to be in my laptop only. Tested with another laptop, both localhost:3306 and 127.0.0.1:3306 works.

If anyone can provide insights that would be very helpful…

10

Look at /etc/hosts on the two machines and see whether localhost is defined
differently.

Antony.

1 Like
11

Machine where localhost resolution is not working but 127001 works =>

# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

# localhost name resolution is handled within DNS itself.
#	127.0.0.1       localhost
#	::1             localhost
# Added by Docker Desktop
192.168.0.111 host.docker.internal
192.168.0.111 gateway.docker.internal
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section

This is from the machine that PDC had to no problem resolving both localhost and 127001

# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host
192.168.19.221		VM-IIOT						#VM-IIOT
# localhost name resolution is handled within DNS itself.
#	127.0.0.1       localhost
#	::1             localhost

Didn’t know about etc/hosts. Thanks. But looks like the files are same.
My main purpose of getting PDC working is done, this is just extra.
If you have any insights for me that would be swell!

1 Like
12

Oh, I did not realise these were Windows machines. Sorry, I’ll keep quiet
again now.

Antony.

13

Oh no, didn’t mean to imply anything… unless you are comfortable with linux troubleshooting…

14

I’m certainly comfortable with Linux troubleshooting. It’s Windows I know
nothing about.

What are these machines (and if they’re Linux, how do they get a /etc/hosts
file containing “This is a sample HOSTS file used by Microsoft TCP/IP for
Windows”)?

Antony.

15

windows sorry

my laptop is the problematic one

my colleague’s laptop also windows, didn’t have issues with localhost resolution, mine did.

16

Oops sorry totally missed cloud part.

Look at your hosts file in windows.

17

So I would say it is not Grafana related issue, but your Windows issue.

18

@yosiasz You seem to be just echo-ing what has already been discussed…

1 Like
19

Apologies @hasan8 you are correct

20

Hey @hasan8
can you please explain me how to solve your errors?
I have successfully connect PDC connect.
I have to similar errors after save&test button click in MSSQL.

image
image1920×916 97.1 KB

Here it is my Docker Errors

2024-03-06 14:02:06 level=info caller=main.go:109 ts=2024-03-06T08:32:06.639532236Z msg="PDC agent info" version=v0.0.25 commit=448d27eba56ebac42de334a0a21398140f8b1113 date=2024-03-05T16:29:01Z sshversion="OpenSSH_9.6p1, OpenSSL 3.1.2 1 Aug 2023" os=linux arch=amd64
2024-03-06 14:02:06 level=info caller=ssh.go:114 ts=2024-03-06T08:32:06.639898136Z msg="starting ssh client"
2024-03-06 14:02:06 level=info caller=keymanager.go:53 ts=2024-03-06T08:32:06.643678536Z msg="starting key manager"
2024-03-06 14:02:06 level=info caller=keymanager.go:126 ts=2024-03-06T08:32:06.643746336Z msg="new keys required: could not read private key file"
2024-03-06 14:02:06 level=info caller=keymanager.go:62 ts=2024-03-06T08:32:06.646583235Z msg="fetching new certificate: agent arguments changed hash=1834297cef769ca031cb28494f66fed70a1b06ce60afa40e84e9a4006d4b2dd4"
2024-03-06 14:02:06 level=info caller=keymanager.go:243 ts=2024-03-06T08:32:06.646640635Z msg="generating new certificate"
2024-03-06 14:02:07 level=info caller=ssh.go:292 ts=2024-03-06T08:32:07.875135051Z msg="Pseudo-terminal will not be allocated because stdin is not a terminal."
2024-03-06 14:02:10 level=info caller=ssh.go:292 ts=2024-03-06T08:32:10.281793091Z msg="Allocated port 1234 for remote forward to socks:0"
2024-03-06 14:02:10 level=info caller=ssh.go:292 ts=2024-03-06T08:32:10.475310662Z msg="This is Grafana Private Datasource Connect!\n"
2024-03-06 14:04:47 level=info caller=ssh.go:292 ts=2024-03-06T08:34:47.28822002Z msg="connect_to DESKTOP-BC7IDV9: unknown host (Name has no usable address)"
2024-03-06 14:05:11 level=info caller=ssh.go:292 ts=2024-03-06T08:35:11.682633309Z msg="connect_to DESKTOP-BC7IDV9: unknown host (Name has no usable address)"
2024-03-06 14:05:39 level=info caller=ssh.go:292 ts=2024-03-06T08:35:39.508755914Z msg="connect_to DESKTOP-BC7IDV9: unknown host (Name has no usable address)"
2024-03-06 15:20:28 level=info caller=ssh.go:292 ts=2024-03-06T09:50:28.836998161Z msg="connect_to DESKTOP-BC7IDV9: unknown host (Name has no usable address)"
2024-03-06 15:28:27 level=info caller=ssh.go:292 ts=2024-03-06T09:58:27.331336571Z msg="connect_to DESKTOP-BC7IDV9: unknown host (Name has no usable address)"
2024-03-06 15:54:29 level=info caller=ssh.go:292 ts=2024-03-06T10:24:29.209472238Z msg="connect_to DESKTOP-BC7IDV9: unknown host (Name has no usable address)"
2024-03-06 16:04:11 level=info caller=ssh.go:182 ts=2024-03-06T10:34:11.469030677Z msg="stopping ssh client"
2024-03-07 12:26:25 level=info caller=main.go:109 ts=2024-03-07T06:56:25.676213Z msg="PDC agent info" version=v0.0.25 commit=448d27eba56ebac42de334a0a21398140f8b1113 date=2024-03-05T16:29:01Z sshversion="OpenSSH_9.6p1, OpenSSL 3.1.2 1 Aug 2023" os=linux arch=amd64
2024-03-07 12:26:25 level=info caller=ssh.go:114 ts=2024-03-07T06:56:25.6767893Z msg="starting ssh client"
2024-03-07 12:26:25 level=info caller=keymanager.go:53 ts=2024-03-07T06:56:25.6805008Z msg="starting key manager"
2024-03-07 12:26:25 level=info caller=keymanager.go:170 ts=2024-03-07T06:56:25.6844628Z msg="new certificate required: certificate validity has expired"
2024-03-07 12:26:25 level=info caller=keymanager.go:243 ts=2024-03-07T06:56:25.6845036Z msg="generating new certificate"
2024-03-07 12:26:26 level=info caller=ssh.go:292 ts=2024-03-07T06:56:26.965086Z msg="Pseudo-terminal will not be allocated because stdin is not a terminal."
2024-03-07 12:26:30 level=info caller=ssh.go:292 ts=2024-03-07T06:56:30.1948442Z msg="Allocated port 1234 for remote forward to socks:0"
2024-03-07 12:26:30 level=info caller=ssh.go:292 ts=2024-03-07T06:56:30.5431656Z msg="This is Grafana Private Datasource Connect!\n"
2024-03-07 12:27:49 level=info caller=ssh.go:292 ts=2024-03-07T06:57:49.631910456Z msg="connect_to DESKTOP-BC7IDV9: unknown host (Name has no usable address)"
2024-03-07 12:33:54 level=info caller=ssh.go:292 ts=2024-03-07T07:03:54.454768174Z msg="connect_to localhost port 3306: failed."

Can you please help me out of this issue.