I’m not familiar with next.jdbc but you get back a map and maps are unordered

There is #sql channel and there are plenty of people familiar with next.jdbc (including the maintainer of that library) for further questions 🙂

@UNST81B9P If I understand your question correctly, you are surprised of the order of the returned columns. @U015FHSUCGM already pointed out the unordered nature of map here. I want to provide you the alternative solution in case you need different behaviour: You can use https://github.com/seancorfield/next-jdbc/blob/develop/doc/result-set-builders.md to modify the result behaviour and e.g. return an array of fields instead of a map.

@UNST81B9P Does that solve your problem somehow?

@U0135SQQC82 let me check that. Thanks

On my case it won't work bc i use a map for turn this into a excel

it sounds like you're logging in some format like:

ERROR: {"my":"json"}

if you can change the logging format, then it will reduce the complexity in the later stages of the pipeline

No. The whole log line is json. Leaving json log lines makes lots of other things trickier. Everything downstream would then need to implement a log line parser. Keeping logs as easily parsable data is quite valuable.

why would the json span multiple lines then?

can't any json be represented without any line breaks?

Afaik, all log aggregation tools limit line length at some number. If it crosses the threshold, the line will be broken into multiple pieces.

> Keeping logs as easily parsable data is quite valuable. that was the point I was trying to make.

breaking the log line into multiple lines seems undesirable.

This is how all log aggregation tools work, afaik.

that is not an enforced behavior that I'm aware of. anyway, at some point, your logging tool is converting the log message into json, you could have it truncate/shorten/summarize data at that point before it's serialized into json

Curious, which tool are you using that does not have limits like this? Yeah - that's the piece I'm after. It's not clear how to do that in an efficient manner. Walking the structure is expensive.

the structure must be walked to serialize it

for more traditional logging, my experience is using python tools. I've also written some myself. for similar use cases, i've also used tools like kafka

depending on your use case, simply using kafka might work

I'm not really sure what your setup is like so it's hard to say

I haven't typically used specialized "logging" tools. I typically just use regular data processing tools. the priorities usually are 1. get the logs to durable storage like s3 as soon as possible 2. have the logs searchable

Interesting. That's far different from my experience. We are currently using Datadog logs. I have used CloudWatch logs, Honeycomb, and, what I'd call, a traditional ELK logging stack. ELK does give you more configurability but at a steep management cost. I've never shipped logs to s3.

makes sense. I still pick and choose managed services. I've never been super happy with the logging managed services to date. there are some managed services that work well.

we've hit the same problem too with our logging code. we've resorted to simply truncating part of the data we know to be problematic. at the moment this works just fine as we get the necessary data from each log that we need for alerts/dashboards/etc

@UGRJKK74Y Do you always know the part that is problematic?

we do. we use structured logs and know exactly what goes into each log line. the logs that are troublesome are the one for generic errors that contain everything in the kafka record (which typically consists of vectors of 100s of things)

do have an ETL pipeline for data analytics? every company I've worked at ends up having some sort of data pipeline and we just end uphaving logs as another source rather than logs as a separate thing

at the moment most of our logs are more traditional logs. though the case i mentioned above is our first push towards more event-based logging

which lends itself to the "logs as a data source" idea you mentioned

We do the same. The problem is that, as it happens, some maps logged happen to contain arbitrarily large maps. Should certainly be more diligent about logging this stuff but I was hoping to protect against it happening.

yeah if you don't know what is arbitrarily large i'm not sure of a good solution off the top of my head. you could pipe into ES instead of a logging aggregator. that should be able to handle things well enough

if you're not having performance issues currently, then in the part of your logging code where it serializes the data into json. you should be able to just see how long the string is and simply decide if you want to reserialize it smaller/truncated/summarized at that point. that means you're only analyzing the messages when they're too large. if there aren't that many, then it shouldn't have much of a performance impact

additionally, if the messages that are bloated are similar, then you can potentially just check the problematic keys/values before serializing and that should be pretty quick

yeah that what we're doing:

(log/error exception ::exception (update data :value truncate))

@U083D6HK9 perhaps you can use fipp (directly or as an inspiration) for truncating the data payload?

(fipp.edn/pprint {:a (range 10)
                    :b (zipmap (range 10)
                               (range 10))}
                   {:print-length 3})
  ;; {:a (0 1 2 ...), :b {0 0, 7 7, 1 1, ...}}