When signing custom plugin I get the error self-signed certificate in chain

hakunamattata · July 1, 2023, 5:17pm

Hi,
I am trying to sign a custom plugin using the command npx @grafana/toolkit plugin:sign --rootUrls http://grafana.staged-by-discourse.com, I am behind a corporate proxy that does SSL inspection of traffic. I have tried everything listed below and still keep getting the message “Self-signed certificate in chain” when trying to sign the plugin.

I have done the following so far

npm config set constrict-ssl false
npm config set cafile /path/to/cert.pem
export NODE_EXTRA_CA_CERTS=\bundle.pem
set my proxy correctly i.e. proxy and https-proxy for npm
npm config set ca “cert string”
Tried setting NODE_TLS_REJECT_UNAUTHORIZED=0

I also ran NODE_DEBUG=‘tls,https’ npx @grafana/toolkit plugin:sign --rootUrls http://grafana.staged-by-discourse.com to see if my certificates are being picked up correctly. I am 100% my certificates are correct.

Nothing seems to solve the problem, any suggestions would be highly appreciated

jangaraj · July 1, 2023, 5:29pm

Did you add your corporate certs to CA certs bundle.pem?

hakunamattata · July 1, 2023, 7:45pm

Yes I did and they are correct certificates verified that too. Also I cleared my npm cache to make sure there was nothing going on there

jangaraj · July 1, 2023, 8:03pm

Could you check if curl -v.... https://grafana.com is able to create verified tls connection with that bundle.

hakunamattata · July 1, 2023, 9:02pm

That is what baffles me that the certificates work using CURL, following is the output

Command - curl -v --cacert bundle.pem https://grafana.com > curl.txt

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0*

Trying 34.120.177.193:443...

* Connected to [grafana.com](http://grafana.com/) (34.120.177.193) port 443 (#0)

* ALPN: offers h2

* ALPN: offers http/1.1

* CAfile: bundle.pem

* CApath: none

} [5 bytes data]

* TLSv1.3 (OUT), TLS handshake, Client hello (1):

} [512 bytes data]

* TLSv1.3 (IN), TLS handshake, Server hello (2):

{ [88 bytes data]

* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):

} [1 bytes data]

* TLSv1.3 (OUT), TLS handshake, Client hello (1):

} [512 bytes data]

* TLSv1.3 (IN), TLS handshake, Server hello (2):

{ [155 bytes data]

* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):

{ [6 bytes data]

* TLSv1.3 (IN), TLS handshake, Certificate (11):

{ [4287 bytes data]

* TLSv1.3 (IN), TLS handshake, CERT verify (15):

{ [264 bytes data]

* TLSv1.3 (IN), TLS handshake, Finished (20):

{ [52 bytes data]

* TLSv1.3 (OUT), TLS handshake, Finished (20):

} [52 bytes data]

* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384

* ALPN: server did not agree on a protocol. Uses default.

* Server certificate:

* subject: C=US; ST=New York; L=New York; O=Raintank Inc.; CN=[grafana.com](http://grafana.com/)

* start date: Jun 27 00:00:00 2023 GMT

* expire date: Jan 3 16:58:01 2024 GMT

* subjectAltName: host "[grafana.com](http://grafana.com/)" matched cert's "[grafana.com](http://grafana.com/)"

* issuer: C=XX; ST=XXX; L=XXXX; O=AAA; OU=BBB; CN=CCC

* SSL certificate verify ok.

} [5 bytes data]

> GET / HTTP/1.1

> Host: [grafana.com](http://grafana.com/)

> User-Agent: curl/7.84.0

> Accept: */*

>

{ [5 bytes data]

* Mark bundle as not supporting multiuse

< HTTP/1.1 200 OK

< Server: nginx/1.23.3

< Date: Sat, 01 Jul 2023 20:41:33 GMT

< Content-Type: text/html

< Transfer-Encoding: chunked

< Vary: Accept-Encoding

< Last-Modified: Sat, 01 Jul 2023 19:21:35 GMT

< Vary: Accept-Encoding

< ETag: W/"64a07cbf-38b63"

< Build: 7ddd35becf6083a0426bdf738d2c62068db9386f

< X-UA-Compatible: IE=Edge,chrome=1

< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload

< Referrer-Policy: origin-when-cross-origin

< Content-Security-Policy: object-src 'none'; script-src 'self' 'unsafe-eval' 'unsafe-inline' [https://go2.grafana.com](https://go2.grafana.com/) [https://cdn.jsdelivr.net](https://cdn.jsdelivr.net/) [https://cdnjs.cloudflare.com](https://cdnjs.cloudflare.com/) [https://unpkg.com](https://unpkg.com/) [https://x.clearbitjs.com](https://x.clearbitjs.com/) [https://app.clearbit.com](https://app.clearbit.com/) .......

< Cache-Control: public, max-age=0, must-revalidate

< X-Frame-Options: DENY

< Access-Control-Allow-Origin: *

< Access-Control-Allow-Methods: GET, OPTIONS

< Access-Control-Allow-Headers: DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range

< Access-Control-Expose-Headers: Content-Length,Content-Range

< x-cache-status: HIT

< X-Cached: HIT

< X-Frame-Options: DENY

< Via: 1.1 google

< Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

<

{ [13831 bytes data]

100 363k 0 363k 0 0 826k 0 --:--:-- --:--:-- --:--:-- 827k

* Connection #0 to host [grafana.com](http://grafana.com/) left intact

jangaraj · July 2, 2023, 7:00am

Time for source code plugin-tools/packages/sign-plugin/src/utils/manifest.ts at main · grafana/plugin-tools · GitHub
Find if axios supports that custom CA certs.

But I would say it is not worth it. I would sign it outside of corp network or I would execute the same API call as that CLI is doing manually.

hakunamattata · July 2, 2023, 11:09am

Thanks, unfortunately I cannot bypass the proxy and move the code out of corp network. Going the manual route might be the best option, do you know which API calls it is making and the respective payloads? If not, I can dig through the source code to figure things out.

jangaraj · July 3, 2023, 7:11am

See linked source code.

hakunamattata · July 3, 2023, 11:01am

Thanks for all your help

jangaraj · July 3, 2023, 11:14am

Did you find a solution? It will be nice if you will post it here for others with the same problem.

hakunamattata · July 3, 2023, 1:34pm

Right now, I am trying the manual method which is cumbersome. For a long term solution, the axios pieces will need to be modified. Once I am successful in the manual method I will post the solution here. On a side note, I am doing the following steps

npm run dev
sign the plugin manually and updating the file dist/MANIFEST.txt
npm run build

I see the file MANIFEST.txt being removed and when I restart Grafana it shows the plugin is unsigned, what am I doing wrong here

hakunamattata · July 5, 2023, 9:08pm

I don’t have a very elegant manual workaround, I did the following to make things work.

run the command npx @grafana/sign-plugin@latest --rootUrls=http://grafana.staged-by-discourse.com
Let the command fail and find out the location where the npm module was installed
opened the file packages/sign-plugin/src/commands/sign.command.ts in Visual Studio code
Set a breakpoint on line 20 i.e. console.log(‘Signing manifest…’);
Copy the value of the variable manifest
Convert the contents of manifest into proper json
Using Postman do an HTTP POST to https://grafana.com/api//plugins/ci/sign , set your Authorization to Bearer Token which is the GRAFANA API KEY, set your JSON payload to the JSON output and make sure Postman does not do SSL verification

Copy the response into dist/MANIFEST.txt file.

Not the best way to do it. I know the exact axios change required, if someone can explain to me how I can run a local copy of the packages then I can test the change

jangaraj · July 5, 2023, 11:30pm

Yeah, that’s corporate env. I have similar experience with mimirtool. But I’m lucky, because I’m able to disable corporate VPN client for a moment.

Ref: When signing custom plugin I get the error self-signed certificate in chain · Issue #290 · grafana/plugin-tools · GitHub

hakunamattata · July 9, 2023, 12:00pm

Just a FYI, this version does not use axios and works behind a corporate proxy, plugin-tools/ at academo/do-not-use-axios · grafana/plugin-tools · GitHub