Grafana and graylog connection error

Grafana Elasticsearch
1

Hello guys

I’m having problem with connecting grafana and graylog.

It says “Elasticsearch error: Bad Gateway

I’m using graylog and elastic search on same machine which ubuntu.

Any who can help please?

image
image1920×2232 175 KB

2

Hello @laale10 and welcome to the :grafana: community

Here are a couple of links [1] [2] that could be helpful.

Otherwise, please share the whole Elasticserach settings

4

Hello @antonio thanks for the replay.
I have seen the links and it didn’t help.
here is my setup
I’m using Graylag and Elasticsearch on the same machine, Graylog is the master.

and Grafana in another machine.

Elastic configuration:-
======================== Elasticsearch Configuration =========================
NOTE: Elasticsearch comes with reasonable defaults for most settings.
Before you set out to tweak and tune the configuration, make sure you
understand what are you trying to accomplish and the consequences.
The primary way of configuring a node is via this file. This template lists
the most important settings you may want to configure for a production cluster.
Please consult the documentation for further information on configuration options:
---------------------------------- Cluster -----------------------------------
Use a descriptive name for your cluster:
cluster.name: graylog

------------------------------------ Node ------------------------------------
Use a descriptive name for the node:
#node.name: node-1

Add custom attributes to the node:
#node.attr.rack: r1

----------------------------------- Paths ------------------------------------
Path to directory where to store the data (separate multiple locations by comma):
path.data: /var/lib/elasticsearch

Path to log files:
path.logs: /var/log/elasticsearch

----------------------------------- Memory -----------------------------------
Lock the memory on startup:
#bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available
on the system and that the owner of the process is allowed to use this
limit.
Elasticsearch performs poorly when the system is swapping the memory.
---------------------------------- Network -----------------------------------
Set the bind address to a specific IP (IPv4 or IPv6):
#network.host: 192.168.0.1

Set a custom port for HTTP:
#http.port: 9200

For more information, consult the network module documentation.
--------------------------------- Discovery ----------------------------------
Pass an initial list of hosts to perform discovery when new node is started:
The default list of hosts is [“127.0.0.1”, “[::1]”]
#discovery.zen.ping.unicast.hosts: [“host1”, “host2”]

Prevent the “split brain” by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#discovery.zen.minimum_master_nodes:

For more information, consult the zen discovery module documentation.
---------------------------------- Gateway -----------------------------------
Block initial recovery after a full cluster restart until N nodes are started:
#gateway.recover_after_nodes: 3

For more information, consult the gateway module documentation.
---------------------------------- Various -----------------------------------
Require explicit names when deleting indices:
action.destructive_requires_name: false

Graylog configuration:-
############################

GRAYLOG CONFIGURATION FILE
############################

This is the Graylog configuration file. The file has to use ISO 8859-1/Latin-1 character encoding.
Characters that cannot be directly represented in this encoding can be written using Unicode escapes
as defined in .
For example, \u002c.

  • Entries are generally expected to be a single line of the form, one of the following:
    propertyName=propertyValue
    propertyName:propertyValue
  • White space that appears between the property name and property value is ignored,
    so the following are equivalent:
    name=Stephen
    name = Stephen
  • White space at the beginning of the line is also ignored.
  • Lines that start with the comment characters ! or # are ignored. Blank lines are also ignored.
  • The property value is generally terminated by the end of the line. White space following the
    property value is not ignored, and is treated as part of the property value.
  • A property value can span several lines if each line is terminated by a backslash (‘\’) character.
    For example:
    targetCities=
    Detroit,
    Chicago,
    Los Angeles
    This is equivalent to targetCities=Detroit,Chicago,Los Angeles (white space at the beginning of lines is ignored).
  • The characters newline, carriage return, and tab can be inserted with characters \n, \r, and \t, respectively.
  • The backslash character must be escaped as a double backslash. For example:
    path=c:\docs\doc1
    If you are running more than one instances of Graylog server you have to select one of these
    instances as leader. The leader will perform some periodical tasks that non-leaders won’t perform.
    is_master = true

The auto-generated node ID will be stored in this file and read after restarts. It is a good idea
to use an absolute file path here if you are starting Graylog server from init scripts or similar.
node_id_file = /etc/graylog/server/node-id

You MUST set a secret to secure/pepper the stored user passwords here. Use at least 64 characters.
Generate one by using for example: pwgen -N 1 -s 96
ATTENTION: This value must be the same on all Graylog nodes in the cluster.
Changing this value after installation will render all user sessions and encrypted values in the database invalid. (e.g. encrypted access tokens)
password_secret = VBC9MiBZZ2Mkye7VYq1D3zCCCvdxQJmzxsmBvdBMwsHcbtSxsg9kGL6YSIjDPHQxMMFzdoBhNPfmJ0Y3Vllx6RR3yjowcw4o

The default root user is named ‘admin’
#root_username = admin

You MUST specify a hash password for the root user (which you only need to initially set up the
system and in case you lose connectivity to your authentication backend)
This password cannot be changed using the API or via the web interface. If you need to change it,
modify it in this file.
Create one by using for example: echo -n yourpassword | shasum -a 256
and put the resulting hash value into the following line
root_password_sha2 = 66c74b936306c9927d418c5e1ccb3fa137d36c90f52ebf71100a189b9010f85c

The email address of the root user.
Default is empty
#root_email = “”

The time zone setting of the root user. See http:/www.joda.org/joda-time/timezones.html for a list of valid time zones.
Default is UTC
#root_timezone = UTC

Set the bin directory here (relative or absolute)
This directory contains binaries that are used by the Graylog server.
Default: bin
bin_dir = /usr/share/graylog-server/bin

Set the data directory here (relative or absolute)
This directory is used to store Graylog server state.
Default: data
data_dir = /var/lib/graylog-server

Set plugin directory here (relative or absolute)
plugin_dir = /usr/share/graylog-server/plugin

###############

HTTP settings
###############

HTTP bind address
The network interface used by the Graylog HTTP interface.
This network interface must be accessible by all Graylog nodes in the cluster and by all clients
using the Graylog web interface.
If the port is omitted, Graylog will use port 9000 by default.
Default: 127.0.0.1:9000
http_bind_address = 0.0.0.0:9000
#http_bind_address = [2001:db8::1]:9000

HTTP publish URI
The HTTP URI of this Graylog node which is used to communicate with the other Graylog nodes in the cluster and by all
clients using the Graylog web interface.
The URI will be published in the cluster discovery APIs, so that other Graylog nodes will be able to find and connect to this Graylog node.
This configuration setting has to be used if this Graylog node is available on another network interface than $http_bind_address,
for example if the machine has multiple network interfaces or is behind a NAT gateway.
If $http_bind_address contains a wildcard IPv4 address (0.0.0.0), the first non-loopback IPv4 address of this machine will be used.
This configuration setting must not contain a wildcard address!
Default:
#http_publish_uri = http:/192.168.1.1:9000/

External Graylog URI
The public URI of Graylog which will be used by the Graylog web interface to communicate with the Graylog REST API.
The external Graylog URI usually has to be specified, if Graylog is running behind a reverse proxy or load-balancer
and it will be used to generate URLs addressing entities in the Graylog REST API (see $http_bind_address).
When using Graylog Collector, this URI will be used to receive heartbeat messages and must be accessible for all collectors.
This setting can be overriden on a per-request basis with the “X-Graylog-Server-URL” HTTP request header.
Default: $http_publish_uri
#http_external_uri =

Enable CORS headers for HTTP interface
This allows browsers to make Cross-Origin requests from any origin.
This is disabled for security reasons and typically only needed if running graylog
with a separate server for frontend development.
Default: false
#http_enable_cors = false

Enable GZIP support for HTTP interface
This compresses API responses and therefore helps to reduce
overall round trip times. This is enabled by default. Uncomment the next line to disable it.
#http_enable_gzip = false

The maximum size of the HTTP request headers in bytes.
#http_max_header_size = 8192

The size of the thread pool used exclusively for serving the HTTP interface.
#http_thread_pool_size = 16

################

HTTPS settings
################

Enable HTTPS support for the HTTP interface
This secures the communication with the HTTP interface with TLS to prevent request forgery and eavesdropping.
Default: false
#http_enable_tls = true

The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
#http_tls_cert_file = /path/to/graylog.crt

The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
#http_tls_key_file = /path/to/graylog.key

The password to unlock the private key used for securing the HTTP interface.
#http_tls_key_password = secret

Comma separated list of trusted proxies that are allowed to set the client address with X-Forwarded-For
header. May be subnets, or hosts.
#trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128

List of Elasticsearch hosts Graylog should connect to.
Need to be specified as a comma-separated list of valid URIs for the http ports of your elasticsearch nodes.
If one or more of your elasticsearch hosts require authentication, include the credentials in each node URI that
requires authentication.
Default: http:/127.0.0.1:9200
#elasticsearch_hosts = http:/node1:9200,http:/user:password@node2:19200
#elasticsearch_hosts =

Maximum number of attempts to connect to elasticsearch on boot for the version probe.
Default: 0, retry indefinitely with the given delay until a connection could be established
#elasticsearch_version_probe_attempts = 5

Waiting time in between connection attempts for elasticsearch_version_probe_attempts
Default: 5s
#elasticsearch_version_probe_delay = 5s

Maximum amount of time to wait for successful connection to Elasticsearch HTTP port.
Default: 10 Seconds
#elasticsearch_connect_timeout = 10s

Maximum amount of time to wait for reading back a response from an Elasticsearch server.
(e. g. during search, index creation, or index time-range calculations)
Default: 60 seconds
#elasticsearch_socket_timeout = 60s

Maximum idle time for an Elasticsearch connection. If this is exceeded, this connection will
be tore down.
Default: inf
#elasticsearch_idle_timeout = -1s

Maximum number of total connections to Elasticsearch.
Default: 200
#elasticsearch_max_total_connections = 200

Maximum number of total connections per Elasticsearch route (normally this means per
elasticsearch server).
Default: 20
#elasticsearch_max_total_connections_per_route = 20

Maximum number of times Graylog will retry failed requests to Elasticsearch.
Default: 2
#elasticsearch_max_retries = 2

Enable automatic Elasticsearch node discovery through Nodes Info,
WARNING: Automatic node discovery does not work if Elasticsearch requires authentication, e. g. with Shield.
Default: false
#elasticsearch_discovery_enabled = true

Filter for including/excluding Elasticsearch nodes in discovery according to their custom attributes,
Default: empty
#elasticsearch_discovery_filter = rack:42

Frequency of the Elasticsearch node discovery.
Default: 30s
elasticsearch_discovery_frequency = 30s
Set the default scheme when connecting to Elasticsearch discovered nodes
Default: http (available options: http, https)
#elasticsearch_discovery_default_scheme = http
#elasticsearch_discovery_default_scheme = https

Enable payload compression for Elasticsearch requests.
Default: false
#elasticsearch_compression_enabled = true

Enable use of “Expect: 100-continue” Header for Elasticsearch index requests.
If this is disabled, Graylog cannot properly handle HTTP 413 Request Entity Too Large errors.
Default: true
#elasticsearch_use_expect_continue = true

Graylog will use multiple indices to store documents in. You can configure the strategy it uses to determine
when to rotate the currently active write index.
It supports multiple rotation strategies, the default being “count”:

  • “count” of messages per index, use elasticsearch_max_docs_per_index below to configure
  • “size” per index, use elasticsearch_max_size_per_index below to configure
  • “time” interval between index rotations, use elasticsearch_max_time_per_index to configure
    A strategy may be disabled by specifying the optional enabled_index_rotation_strategies list and excluding that strategy.
    #enabled_index_rotation_strategies = count,size,time

Provides a hard upper limit for the retention period of any index set at configuration time.
This setting is used to validate the value a user chooses for the maximum number of retained indexes, when configuring
an index set. However, it is only in effect, when a time-based rotation strategy is chosen.
If a rotation strategy other than time-based is selected and/or no value is provided for this setting, no upper limit
for index retention will be enforced. This is also the default.
max_index_retention_period = P90d
ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
to your previous 1.x settings so they will be migrated to the database!
This configuration setting is only used on the first start of Graylog. After that,
index related settings can be changed in the Graylog web interface on the ‘System / Indices’ page.
(Approximate) maximum number of documents in an Elasticsearch index before a new index
is being created, also see no_retention and elasticsearch_max_number_of_indices.
Configure this if you used ‘rotation_strategy = count’ above.
ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
to your previous 1.x settings so they will be migrated to the database!
This configuration setting is only used on the first start of Graylog. After that,
index related settings can be changed in the Graylog web interface on the ‘System / Indices’ page.
elasticsearch_max_docs_per_index = 20000000

(Approximate) maximum size in bytes per Elasticsearch index on disk before a new index is being created, also see
no_retention and elasticsearch_max_number_of_indices. Default is 1GB.
Configure this if you used ‘rotation_strategy = size’ above.
ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
to your previous 1.x settings so they will be migrated to the database!
This configuration setting is only used on the first start of Graylog. After that,
index related settings can be changed in the Graylog web interface on the ‘System / Indices’ page.
#elasticsearch_max_size_per_index = 1073741824

(Approximate) maximum time before a new Elasticsearch index is being created, also see
no_retention and elasticsearch_max_number_of_indices. Default is 1 day.
Configure this if you used ‘rotation_strategy = time’ above.
Please note that this rotation period does not look at the time specified in the received messages, but is
using the real clock value to decide when to rotate the index!
Specify the time using a duration and a suffix indicating which unit you want:
1w = 1 week
1d = 1 day
12h = 12 hours
Permitted suffixes are: d for day, h for hour, m for minute, s for second.
ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
to your previous 1.x settings so they will be migrated to the database!
This configuration setting is only used on the first start of Graylog. After that,
index related settings can be changed in the Graylog web interface on the ‘System / Indices’ page.
#elasticsearch_max_time_per_index = 1d

Optional upper bound on elasticsearch_max_time_per_index
elasticsearch_max_write_index_age = 1d
Disable checking the version of Elasticsearch for being compatible with this Graylog release.
WARNING: Using Graylog with unsupported and untested versions of Elasticsearch may lead to data loss!
#elasticsearch_disable_version_check = true
elasticsearch_disable_version_check = true
elasticsearch_version = 7

Disable message retention on this node, i. e. disable Elasticsearch index rotation.
#no_retention = false

How many indices do you want to keep?
ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
to your previous 1.x settings so they will be migrated to the database!
This configuration setting is only used on the first start of Graylog. After that,
index related settings can be changed in the Graylog web interface on the ‘System / Indices’ page.
elasticsearch_max_number_of_indices = 20

Decide what happens with the oldest indices when the maximum number of indices is reached.
The following strategies are availble:

  • delete # Deletes the index completely (Default)
  • close # Closes the index and hides it from the system. Can be re-opened later.
    ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
    to your previous 1.x settings so they will be migrated to the database!
    This configuration setting is only used on the first start of Graylog. After that,
    index related settings can be changed in the Graylog web interface on the ‘System / Indices’ page.
    retention_strategy = delete

How many Elasticsearch shards and replicas should be used per index? Note that this only applies to newly created indices.
ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
to your previous settings so they will be migrated to the database!
This configuration setting is only used on the first start of Graylog. After that,
index related settings can be changed in the Graylog web interface on the ‘System / Indices’ page.
elasticsearch_shards = 4
elasticsearch_replicas = 0

Prefix for all Elasticsearch indices and index aliases managed by Graylog.
ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
to your previous settings so they will be migrated to the database!
This configuration setting is only used on the first start of Graylog. After that,
index related settings can be changed in the Graylog web interface on the ‘System / Indices’ page.
Also see
elasticsearch_index_prefix = graylog

Name of the Elasticsearch index template used by Graylog to apply the mandatory index mapping.
Default: graylog-internal
ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
to your previous settings so they will be migrated to the database!
This configuration setting is only used on the first start of Graylog. After that,
index related settings can be changed in the Graylog web interface on the ‘System / Indices’ page.
#elasticsearch_template_name = graylog-internal

Do you want to allow searches with leading wildcards? This can be extremely resource hungry and should only
be enabled with care. See also:
allow_leading_wildcard_searches = false

Do you want to allow searches to be highlighted? Depending on the size of your messages this can be memory hungry and
should only be enabled after making sure your Elasticsearch cluster has enough memory.
allow_highlighting = false

Analyzer (tokenizer) to use for message and full_message field. The “standard” filter usually is a good idea.
All supported analyzers are: standard, simple, whitespace, stop, keyword, pattern, language, snowball, custom
Elasticsearch documentation:
Note that this setting only takes effect on newly created indices.

5

try increasing the verbosity of the Grafana server logs to debug and note any errors. For printing to console, set the console logs to debug as well.

6

Also, when you try to save and test the datasource connection, you can open up your browser’s developer tools and look at the network tab. that can often include useful information about network connection issues