Tshark packet capture pushing to Loki

I had an idea to build a portable network analyzer for troubleshooting things like Crestron automation controllers or other funky IoT devices that are misbehaving. Curious if anyone has done something similar. The idea is: A Raspberry Pi with 2 Ethernet ports, inserted in between the device and the switch. It runs Tshark to capture and analyze all network packets coming in, similar to a port mirror, and leave it there for a few days to find intermittent problems. Tshark would push its output to Loki in the cloud, for easy analysis.

Tshark has an output option called ek intended to send to ElasticSearch. I’m not exactly sure how I would push it to Loki.

I haven’t done anything like this before, and I could be way off in my comments below.

First, I believe that you cannot change the format of live capture. Which means that you’ll have to manually transform the pcap file into some other format before you can send it to Loki. You also need to worry about the rotation of files. With those considerations in mind, maybe something like this would work:

Write a script that starts tshark with a text output (say /opt/tshark/output/file1.pcap), and then do the following on an hourly basis (or an interval of your choice, likely constraint by the amount of space you have on your appliance):

  1. Stop tshark.
  2. Move file1.pcap to file_await_processing.pcap.
  3. Start tshark, with text output to file1.pcap (idea is to keep the restart as short as possible).
  4. Transform file_await_processing.pcap to json.
  5. Either process the json within the script and send results to Loki / Grafana Cloud, or use a grafana agent or promtail agent.
  6. Clean up and remove file_await_processing.pcap.

One other thing to consider is that ideally you’d want to remove the json file as soon as it’s done being sent to Loki, but if you are using a log agent it might not be easy to tell when they are done. In this case you might opt to just delete the json file on an hourly basis and accept the assumption that everything should be done sending by that point.