For my setup, I used the cloudformation stack with connection from AWS Cloudwatch to Kinesis Firehose and finally to Grafana Cloud. The logs are getting properly ingested, but I’m noticing a particular issue with the logs when I turn on transforms.
When turning on transforms, LogEvent groups are conglomerated into a single large log. For example, if I have 3 logs that are as follows:
Log1
Log2
Log3
Grafana will merge these together into 1 mega-log:
Log1
Log2
Log3
This occurs even with the default cloudwatch firehose provided by AWS. This seems to be a grafana issue, but I’m a little lost while digging into this. Any assistance would be greatly appreciated.
What I’ve tried:
- Reingest the logs to the stream
- Try different structured formats for the output
- Use newlines and multiple newlines when rejoining the data
I’ve also investigated whether this was a timestamp issue, but I’m seeing logs get correctly separated without transform when the nano timestamps of two records are equivalent.