Logs Disappearing Immediately After Injest

[nicopampaloni]

Hello,
I’m having a strange issue where logs are dissapearing almost immediately after injest. I believe it’s related to the timestamp of the logs. I am using Logstash to parse my logs and send them to loki. If I add the Date filter to change the @timestamp field to one present in the logs, they get deleted from Loki almost immediately. When I injest the logs and refresh Grafana, I can see the various labels getting detected, but by the time I select a label and search for it, there are no results found. Within 30-60 seconds of inesting the logs, the labels dissapear from Loki as well.

Here is an example log line:

INX00086E1637,10304612822043760453|1654009231.8297705715,2,2022:05:31:15:00:31:849,2022:05:31:15:00:31:850

I am using the date filter in Logstash to set the @timestamp field to the 2022:05:31:15:00:31:849 field in the log file. I have verified this is being done by Logstash correctly by setting a test field to the same value. The field shows the timestamp being converted to 2022-05-31T15:00:31.849Z.

If I comment out the Date filter the logs are ingested and displayed correctly with timestamp field set to the ingest time.

Here is the filter snippet of my logstash config file (note the “timein” field corresponds to the timestamp referenced above):

date {
  match => "timein", "yyyy:MM:dd:HH:mm:ss:SSS"]
}

and my loki config:

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common: 
  path_prefix: /tmp/loki
  storage: 
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

limits_config:
  max_streams_per_user: 0
  max_global_streams_per_user: 0
  max_query_length: 0

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

Has anyone seen this before or have any ideas what might be causing this?
Thank you!