Logging presto queries on grafana dashboard using event-listener and fluent-bit

I have a presto instance running on a namespace and it has a custom plugin and event-listener configured that gets all the logs and forwards them to 24224 port.

I can see the logs if they are written as log.info("any string here") but fluency.emit(tag, map<>) is not working at all and i dont see any logs or event object being sent to fluentd instance.

can you please help in configuring how it can work or how i can debug this when on AWS EKS?

When I check the logs of daemonset fluentbit, I can see it is listening to TCP port 24224

[input:forward:forward.0] listening on 0.0.0.0:24224  

Here is the configuration of fluent-bit:

fluent-bit.conf: |-      [SERVICE]          HTTP_Server    On          HTTP_Listen    0.0.0.0          HTTP_PORT      2020          Flush          10          Daemon         Off          Log_Level      trace          [INPUT]          Name           forward          Port           24224          Listen         0.0.0.0            [Output]          Name grafana-loki          Match *          Url <MY LOKI URL HERE>          TenantID ""          BatchWait 1          BatchSize 1048576          Labels {job="fluent-bit"}          RemoveKeys kubernetes,stream          AutoKubernetesLabels false          LineFormat json          LogLevel warn    labelmap.json: |-      {        "kubernetes": {          "container_name": "container",          "host": "node",          "labels": {            "app": "app",            "release": "release"          },          "namespace_name": "namespace",          "pod_name": "instance"        },        "stream": "stream"      }  

Here is my event-listener.properties file contents from presto:

event-listener.name=presto-fluent-logger  event-listener.fluentd-host=127.0.0.1  event-listener.fluentd-port=24224  event-listener.fluentd-tag=presto.query  

and I have followed all the steps from this link: https://github.com/komamitsu/fluency

I am getting error in plugin ConnectionException: Connection Refused

I am sure that the presto plugin has got registered successfully and the error shows that it is not able to connect to my fluentbit instance may be. Not sure what host:port I should keep in plugin or event-listener.properties file

But I get no logs when I execute a query from presto cli. Can anyone please help? I am not very sure what all codes do I need to place here

https://stackoverflow.com/questions/66061794/logging-presto-queries-on-grafana-dashboard-using-event-listener-and-fluent-bit February 05, 2021 at 06:42PM