Split single coord field in to two with lat and lon

HI,

I’m trying to create a map from data in elastic search, I’ve connected Grafana to elastic search and I can pull data from it, the field that contains coordinate data has lat,long in it, where as It looks like the GEO Map plugin doesn’t recognise that as a field it can use and wants two fields instead, one with latitude and one with longitude. I’ve been trying to work out how ot do it and failing terribly.

In my geo map, I have a query type of logs, with the following to pull the data

flow.client.geo.loc.coord:*

This pulls only the rows that have the flow.client.geo.loc.coord field populated, the field is populated with data such as 27.7431,-82.6909

Coords field in the GeoMap panel won’t accept that, in fact it doesn’t even show it as a valid available field, probably due to the comma. So I need to some how split that field in to two separate fields. Something like flow.client.geo.loc.coord.lon and flow.client.geo.loc.coord.lat.

I see a number of threads about combining fields, but not for splitting one with a comma in the middle. As far as I can tell, transformations should be able to do it but I can’t seem to work out how :cry:

Could anybody please give me some pointers in how I could achieve this to allow the geomap to use the coord data to plot points on the map.

Thanks all!

I’ve spent days trying to transform a single field into two, but it seems like a complex task. The field’s contents are all similar to 38.6666,-86.996. After extensive experimentation, attempting to use ChatGPT for assistance, which seemed to be exploring options that no longer exist in Grafana or were entirely incorrect, I’ve come to realise that this transformation may be far from simple and might even be impossible. If anyone has any ideas or suggestions, please let me know.

Yes, Grafana is for data visualization. You need data manipulation, which is limited (and there is IMHO a good reason for that).

So save data in the format, which is suitable for Grafana. Or add middleware, which will prepare data for a Grafana.

HI,

Thanks for the response, currently we don’t have access to manipulate the data at the elastiflow source, do you have any suggestions for a middleware that may be able to handle the job?

Grafana (infinity datasource) → (simple server app in your favorite language to split field) → Elasticsearch

2 Likes

I’ll look in to it, thanks :slight_smile:

1 Like

Please post raw sample data and what data source plugin you are using

@timestamp flow.client.geo.city.name flow.client.geo.country.code Country flow.client.geo.loc.coord flow.client.ip.addr
11/11/24 13:28 Sydney AU Australia -33.8715,151.2006 146.75.190.38
11/11/24 13:27 Duque de Caxias BR Brazil -22.7886,-43.3175 138.97.13.118
11/11/24 13:26 Maceió BR Brazil -9.575,-35.6908 192.140.116.107
11/11/24 13:27 Nassau BS Bahamas 25.0565,-77.3524 69.4.164.122
11/11/24 13:26 Nassau BS Bahamas 25.0565,-77.3524 69.4.164.122
11/11/24 13:26 Burlington CA Canada 43.4327,-79.8709 24.141.162.104
11/11/24 13:27 Montreal CA Canada 45.4995,-73.5848 104.28.132.142
11/11/24 13:30 Kaifeng CN China 34.8062,114.2763 61.158.237.50
11/11/24 13:26 Zhengzhou CN China 34.7599,113.6459 39.162.16.230
11/11/24 13:30 Wrexham GB United Kingdom 53.0594,-3.1921 2.100.150.137
11/11/24 13:31 Accra GH Ghana 5.5486,-0.2012 154.161.49.79
11/11/24 13:31 Accra GH Ghana 5.5486,-0.2012 154.161.49.79
11/11/24 13:31 HK Hong Kong 22.2578,114.1657 62.204.41.106
11/11/24 13:29 Kolkata IN India 22.518,88.3832 117.205.251.108
11/11/24 13:30 Gangnam-gu KR South Korea 37.4838,127.0501 175.196.117.164
11/11/24 13:26 Bayamón PR Puerto Rico 18.3616,-66.1935 24.41.236.17
11/11/24 13:31 Jeddah SA Saudi Arabia 21.4849,39.192 37.106.10.132
11/11/24 13:30 San Salvador SV El Salvador 13.6806,-89.1803 190.99.43.168
11/11/24 13:30 Adkins US United States 29.393,-98.2372 206.109.194.74
11/11/24 13:28 Scranton US United States 41.39,-75.6636 73.230.50.216
11/11/24 13:28 Scranton US United States 41.39,-75.6636 73.230.50.216
11/11/24 13:31 Cape Town ZA South Africa -33.91,18.4304 41.242.163.12

Here’s a very small snippet of the data, as you can see it has a time with the location data, one of the fields contains coordinates, but both parts are in the same field.

The data is coming from the ElastiSearch connector.

So is this the data after it has been processed by grafana? I thought elastic search data was json

then leverage infinity jsonata query

https://try.jsonata.org/mfqPAW_9P

HI,

Thank you for your help/

This is the raw data displayed by Grafana when I just query it. There are admittedly more columns but they contain private information, so I blanked those out before sending.

As far as I can tell, Elastisearch does store in JSON but the plugin for Grafana handles all that in the back end and displays it differently. I’ll see if I can find a way of connecting the elastisearch database to the infinity data source, I’ve not used it before, so may take a while to work it out.

Please post the raw json data from es

HI,

Here is the JSON that gets returned. It is returned in a single line but I had to anonymise it first, so it’s now in correct layout too. flow.client.geo.loc.coord is the data I need, with time stamps so that I can pulled data from different times.

{
	"hits": [
		{
			"_id": "NsOH_pIBhYpifShZdNT_",
			"_index": "elastiflow-flow-codex-2.4-rollover-000007",
			"_score": 1,
			"_source": {
				"@timestamp": 1730847141153,
				"as.label": [
					"TS-XXXX-XXX",
					"Anonymised Org (XXXXX)"
				],
				"flow.bytes": 5914624,
				"flow.client.as.asn": "XXXXX",
				"flow.client.as.label": "Anonymised Org (XXXXX)",
				"flow.client.as.org": "Anonymised Org",
				"flow.client.geo.city.name": "Anonymised City",
				"flow.client.geo.country.code": "XX",
				"flow.client.geo.country.name": "Anonymised Country",
				"flow.client.geo.loc.coord": "XX.XXXX,-XX.XXXX",
				"flow.client.geo.tz.name": "Anonymised/Time_Zone",
				"flow.client.host.name": "anonymised-hostname.com",
				"flow.client.ip.addr": "XXX.XXX.XXX.XXX",
				"flow.client.ip.subnet.mask_size": "XX",
				"flow.client.l4.port.id": 63935,
				"flow.client.l4.port.name": "TCP/XXXX",
				"flow.client.mac.addr": "XX:XX:XX:XX:XX:XX",
				"flow.collect.timestamp": 1730847141153,
				"flow.community.id": "anonymised_id",
				"flow.conversation.id": "anonymised_conversation_id",
				"flow.direction.name": "Ingress",
				"flow.dst.as.asn": "XXXXX",
				"flow.dst.as.label": "Anonymised Org (XXXXX)",
				"flow.dst.as.org": "Anonymised Org",
				"flow.dst.geo.city.name": "Anonymised City",
				"flow.dst.geo.country.code": "XX",
				"flow.dst.geo.country.name": "Anonymised Country",
				"flow.dst.geo.loc.coord": "XX.XXXX,-XX.XXXX",
				"flow.dst.geo.tz.name": "Anonymised/Time_Zone",
				"flow.dst.host.name": "anonymised-hostname.com",
				"flow.dst.ip.addr": "XXX.XXX.XXX.XXX",
				"flow.dst.ip.subnet.mask_size": "XX",
				"flow.dst.l4.port.id": 63935,
				"flow.dst.l4.port.name": "TCP/XXXX",
				"flow.dst.mac.addr": "XX:XX:XX:XX:XX:XX",
				"flow.export.geo.city.name": "Anonymised City",
				"flow.export.geo.country.code": "XX",
				"flow.export.geo.country.name": "Anonymised Country",
				"flow.export.geo.loc.coord": "XX.XXXX,-XX.XXXX",
				"flow.export.geo.tz.name": "Anonymised/Time_Zone",
				"flow.export.host.name": "anonymised-hostname.com",
				"flow.export.ip.addr": "XXX.XXX.XXX.XXX",
				"flow.export.l4.port.id": 49772,
				"flow.export.sysuptime": 2763325000,
				"flow.export.type": "sflow",
				"flow.export.version.name": "sFlow vX",
				"flow.export.version.ver": 5,
				"flow.in.bytes": 5914624,
				"flow.in.netif.index": "anonymised_index",
				"flow.in.netif.name": "index: anonymised",
				"flow.in.packets": 4096,
				"flow.in.vlan.tag.id": "anonymised_vlan_id",
				"flow.in.vlan.tag.pcp.name": "BE",
				"flow.isServer": "source",
				"flow.locality": "public",
				"flow.meter.packet_select.interval.packets": 4096,
				"flow.meter.packets_drop": 0,
				"flow.meter.packets_total": 669868032,
				"flow.next_hop.as.asn": "anonymised_asn",
				"flow.next_hop.as.label": "Anonymised Org (anonymised_asn)",
				"flow.next_hop.as.org": "Anonymised Org",
				"flow.next_hop.host.name": "anonymised-link.net",
				"flow.next_hop.ip.addr": "XXX.XXX.XXX.XXX",
				"flow.out.netif.index": "anonymised_index",
				"flow.out.netif.name": "index: anonymised",
				"flow.out.vlan.tag.id": "anonymised_vlan_id",
				"flow.out.vlan.tag.pcp.name": "BE",
				"flow.packets": 4096,
				"flow.seq_num": "anonymised_seq_num",
				"flow.server.as.asn": "XXXXX",
				"flow.server.as.label": "TS-XXXX-XXX",
				"flow.server.as.org": "Anonymised Org",
				"flow.server.geo.city.name": "Anonymised City",
				"flow.server.geo.country.code": "XX",
				"flow.server.geo.country.name": "Anonymised Country",
				"flow.server.geo.loc.coord": "XX.XXXX,-XX.XXXX",
				"flow.server.geo.tz.name": "Anonymised/Time_Zone",
				"flow.server.host.name": "anonymised-hostname.com",
				"flow.server.ip.addr": "XXX.XXX.XXX.XXX",
				"flow.server.ip.subnet.mask_size": "XX",
				"flow.server.l4.port.id": 80,
				"flow.server.l4.port.name": "http (TCP/80)",
				"flow.server.mac.addr": "XX:XX:XX:XX:XX:XX",
				"flow.src.as.asn": "XXXXX",
				"flow.src.as.label": "Anonymised Org (XXXXX)",
				"flow.src.as.org": "Anonymised Org",
				"flow.src.geo.city.name": "Anonymised City",
				"flow.src.geo.country.code": "XX",
				"flow.src.geo.country.name": "Anonymised Country",
				"flow.src.geo.loc.coord": "XX.XXXX,-XX.XXXX",
				"flow.src.geo.tz.name": "Anonymised/Time_Zone",
				"flow.src.host.name": "anonymised-hostname.com",
				"flow.src.ip.addr": "XXX.XXX.XXX.XXX",
				"flow.src.ip.subnet.mask_size": "XX",
				"flow.src.l4.port.id": 80,
				"flow.src.l4.port.name": "http (TCP/80)",
				"flow.src.mac.addr": "XX:XX:XX:XX:XX:XX",
				"geo.city.name": [
					"Anonymised City",
					"Anonymised City"
				],
				"geo.country.name": [
					"Anonymised Country"
				],
				"ip.dscp.name": "DF",
				"ip.ecn.name": "Non-ECT",
				"ip.frag.flags.tags": [
					"DF"
				],
				"ip.packet.size": 1422,
				"ip.ttl": 64,
				"ip.version.name": "IPv4",
				"ip.version.ver": 4,
				"l2.frame.size": 1444,
				"l4.proto.name": "TCP",
				"l4.session.established": "true",
				"sflow.pen.id": 0,
				"sflow.pen.name": "iana",
				"sflow.sample.header_proto.name": "ETHERNET-ISO88023",
				"sflow.sample.seq_num": "anonymised_sample_seq",
				"sflow.sample.size": 128,
				"sflow.sample.strip_size": 4,
				"sflow.sample_type.name": "flow_sample",
				"sflow.source_id": "anonymised_source_id",
				"sflow.source_id_type.name": "Interface Index",
				"sflow.sub_agent_id": 0,
				"system.host.name": "anonymised-hostname.com",
				"system.ip.addr": "XXX.XXX.XXX.XXX",
				"tcp.ack_num": "anonymised_ack_num",
				"tcp.flags.bits": 16,
				"tcp.flags.tags": [
					"ACK"
				],
				"tcp.header.size": 32,
				"tcp.options.payload": "anonymised_payload",
				"tcp.seq_num": "anonymised_seq_num",
				"tcp.urgent_pointer": 0,
				"tcp.window.size": 31746,
				"vlan.c_tag.dei.state": "false",
				"vlan.c_tag.id": "anonymised_vlan_id",
				"vlan.c_tag.pcp.name": "BE",
				"vlan.tag.id": [
					"anonymised_vlan_id_1",
					"anonymised_vlan_id_2"
				]
			}
		}
	]
}

Plug in that JSON data in the link I provided for doing jsonata and post back what you get after applying proper jsonata query for your specific data

It didn’t work, however I foudn that this did

hits.{
“_id”: _id,
“timestamp”: _source.“@timestamp”,
“latitude”: $split(_source.“flow.export.geo.loc.coord”, “,”)[0],
“longitude”: $split(_source.“flow.export.geo.loc.coord”, “,”)[1]
}

Downside is, I can’t work out how to us eit in the infinity data source :frowning:

ip.of.elastic:9200

There are tons of examples with solutions in this forum.