Process a local JSON file and send it to Loki for use in Grafana?

Grafana Alloy
1

Hi All,
I’m just getting started with Alloy so hopefully this might be an easy question to answer?
I’ve read various forum posts on the subject but I’m still hopelessly lost I’m afraid.

Here’s what I want to do…

I have a script on my unix box that generates a json file called “botstatus.json” every couple of minutes to show the current status of some bots we run as part of our internal instant messaging system.

The json file it produces looks like this:

[
  {
    "timestamp": "2025-09-19T14:01:54.041+02:00",
    "accountID": 9987663234964776,
    "userName": "bot1",
    "displayName": "Bot One",
    "status": "ENABLED",
    "presenceCategory": "OFFLINE"
  },
  {
    "timestamp": "2025-09-19T14:01:55.501+02:00",
    "accountID": 164467976335421,
    "userName": "bot2",
    "displayName": "Bot Two",
    "status": "ENABLED",
    "presenceCategory": "AVAILABLE",
  },
  {
    "timestamp": "2025-09-19T14:01:55.501+02:00",
    "accountID": 644569676335422,
    "userName": "bot3",
    "displayName": "Bot Three",
    "status": "ENABLED",
    "presenceCategory": "AVAILABLE"
  }
]

The timestamp is the last time the script checked each bot. The other values are for the status of the bots themselves.

I’m hoping that Alloy can read the file, make sense of it and send it to Loki+Grafana (running on another unix server).
I’d then like to create a panel in Grafana that shows the current status of each bot.

Is this possible?
Can someone give me an example of how to craft the alloy.config to do this?

All help and suggestions very gratefully received :slight_smile: !

2

I don’t think Alloy can parse multi-line JSON. You mentioned this is a script? If you have control over the script then I would recommend you to change the output to one json per line. Something like this:

{"timestamp": "2025-09-19T14:01:54.041+02:00","accountID": 9987663234964776,"userName": "bot1","displayName": "Bot One","status": "ENABLED","presenceCategory": "OFFLINE"}
{"timestamp": "2025-09-19T14:01:55.501+02:00","accountID": 164467976335421,"userName": "bot2","displayName": "Bot Two","status": "ENABLED","presenceCategory": "AVAILABLE"}
{"timestamp": "2025-09-19T14:01:55.501+02:00","accountID": 644569676335422,"userName": "bot3","displayName": "Bot Three","status": "ENABLED","presenceCategory": "AVAILABLE"}
3

Ah. Yes I can change the script to do that.
So Alloy should be able to understand and pick up all the information then?

I was worried that I’d have to do something with “stage.json” (which i don’t quite understand at the moment) after reading some other community posts like these:

4

Hi @tonyswumac and @marcozin,

Sure Alloy can collect multiline logs - no problem, take a look here: stage.multiline, you just have to define how to catch/define net log event, which often is done by pointing at the timestamp

5

I am aware of that, but you can’t use multiline to reliably determine first or last line of a JSON body, particularly if it’s nested.

6

OK. So I’ve changed the script so I’m now getting output like this:

[{"timestamp":"2025-09-25T19:01:25.771+02:00","accountID":349644697632776,"userName":"bot1","displayName":"bot one","status":"ENABLED","presenceCategory":"OFFLINE","presenceTimestamp":1758819687451},
{"timestamp":"2025-09-25T19:01:25.772+02:00","accountID":349644697632778,"userName":"bot2","displayName":"bot two","status":"ENABLED","presenceCategory":"OFFLINE","presenceTimestamp":1758819687540},
{"timestamp":"2025-09-25T19:01:25.772+02:00","accountID":349644697632780,"userName":"bot3","displayName":"bot three","status":"ENABLED","presenceCategory":"OFFLINE","presenceTimestamp":1758819687618},
{"timestamp":"2025-09-25T19:01:25.773+02:00","accountID":349644697632782,"userName":"bot4","displayName":"bot four","status":"ENABLED","presenceCategory":"OFFLINE","presenceTimestamp":1758819687699},
{"timestamp":"2025-09-25T19:01:27.252+02:00","accountID":349644697635422,"userName":"bot5","displayName":"bot five","status":"ENABLED","presenceCategory":"AVAILABLE","presenceTimestamp":1758819712061}]

I tried to put a config together that would pick it up but I’m obviously doing something wrong.
Here’s my Config:

livedebugging {
  enabled = true
//  enabled = false
}


 local.file_match "local_jsonstatus_files" {
     path_targets = [{"__path__" = "/opt/myapp/powershell/BotStatus_cleaned.json"}]
     sync_period = "5s"
 }


 loki.source.file "jsonstatus_scrape" {
    targets    = local.file_match.local_jsonstatus_files.targets
    forward_to = [loki.process.process_jsonstatuses.receiver]
    tail_from_end = false
  }

 loki.process "process_jsonstatuses" {
    stage.static_labels {
        values = {
            service = "SymphonyAccountSystemStatus",
        }
    }
    stage.json {
        expressions = {
            "_ts"                     = "timestamp",
            "_accountID"              = "accountID",
            "_userName"               = "userName",
            "_displayName"            = "displayName",
            "_status"                 = "status",
            "_presenceCategory"       = "presenceCategory",
            "_presenceTimestamp"      = "presenceTimestamp",
        }
    }
    stage.labels {
        values = {
             bot = "_userName",
             botStatus = "_status",
        }
    } 
    forward_to = [loki.write.grafana_loki.receiver]
 }


 loki.write "grafana_loki" {
    external_labels = {
      server = sys.env("HOSTNAME"),
      environment = "DEVELOPMENT",
    }
    endpoint {
      url = "http://internal-fqdn-removed:3100/loki/api/v1/push" 
    }
  }

I can confirm that that the “loki.write.grafana_loki” part works as theres another part of the config that does work that sends back *.log files (I’ve just removed it from this post for ease of reading).

The Alloy Web UI shows the process properly.

image
image1192×276 15.6 KB

However, it doesn’t seem to work. At one point I finally got it to send some data through but “bot” just contained the text “_userName” rather than any data. Since then I tried a few times more, changing the syntax after reading some more but now it doesn’t send anything through.

I’m a bit lost :worried:

Can someone point me in the right direction?

7

Because you want each log line to be a self contained entity, you want one json per line, which means you’ll want to remove avoid grouping multiple json lines into a list.

One self contained JSON per line, so alloy can treat each log line as one entity.

8

I thought I’ve done that now? No?
Do you mean that I need to remove the “[” on the first line and the “]” on the last line?

9

Ha! it worked. @tonyswumac - thank you :slight_smile:
I chopped out the square brackets and the new-lines and commas between the records like this:

# remove the line-breaks and commas between records           :  sed -e 's/},{/}\n{/g'
# remove the open-square-bracket at the beginning of the line :  sed -e 's/^\[//g'
# remove the close-square-bracket at the end of the line      :  sed -e 's/\]$//g'

cat ./BotStatus.json | sed -e 's/},{/}\n{/g' | sed -e 's/^\[//g' |  sed -e 's/\]$//g' > ./BotStatus_cleaned.json
10

I may have spoken too soon.
Alloy seems to be ignoring new data.
The JSON file only stores the current information. when the script runs it overwrites the file rather than appending it.

I had thought that the:

tail_from_end = false

specified in the:

loki.source.file "jsonstatus_scrape"

section would handle that but perhaps not? Can someone advise here?

Perhaps i need to do something with the max & min poll frequency settings?
loki.source.file | File Watch | Grafana Alloy documentation

11

Most log agents will not work properly if the log files are overwritten. I would recommend you to:

  1. Change your script to append rather than overwrite.
  2. Implement some sort of log rotation so your log files don’t grow indefinitely.
12

Hi,

Following you guys here, very interested chat an thread!

I agree with @tonyswumac on his last comments- quite important.

Now in regard to stage.multiline i still believe you should have a good chance here, could you try:

stage.mulitline {

firstline = `\{\“timestamp\”:\“\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}+\d{2}:\d{2}\”`

}

13

Thanks for the hint regarding the multiline. It’s good to have an example to show how that works.
I might need that for the future but not for this particular requirement.