add a (flush) call

to your go loop

Awesome, thanks! Is this an OS specific difference? (linux vs macos)

Look at the source of print

(defn print
  "Prints the object(s) to the output stream that is the current value
  of *out*.  print and println produce output for human consumption."
  {:added "1.0"
   :static true}
  [& more]
    (binding [*print-readably* nil]
      (apply pr more)))

(defn prn
  "Same as pr followed by (newline). Observes *flush-on-newline*"
  {:added "1.0"
   :static true}
  [& more]
    (apply pr more)
    (newline)
    (when *flush-on-newline*
      (flush)))

sorry. get source from print vs println . They bottom out in pr vs prn

but i think you are seeing impl different behavior. Basically writing to a buffer and need to push that buffer to the actual output

Oh I see

no. this comes up frequently but to have a sorted-map requires all of the keys to be sortable in relation to each other

https://ask.clojure.org/index.php/11409/somewhat-confusing-sorted-set-behavior?show=11409#q11409

hmm interesting, so if some piece of code was tested against maps, you can't just replace a hash-map by a sorted-map and expect it to work, even if both are considered maps from an interface point of view (map? (sorted-map)) => true

that is tricky

Not at a keyboard, but I’m pretty sure the error is not from the lookup but from the construction; i.e. any code you write that uses a map doesn’t care which kind of map it gets.

@U0522J5HN it is from the lookup, you can't get from a sorted-map a different type of key, unless you provide a comparator. Which makes sense, but it is unfortunate, since code working on hash-map will fail on sorted-map

Hm, you’re right. That is a bit surprising. :thinking_face:

In my opinion, the doc string of clojure.core/sorted-map leaves too much to the imagination. It is possible, with some effort and vast general knowledge, to rationalize the subsequent results. But that is a rather low standard for docstrings. This docstring could tell you what the default comparator is, or could tell you whose compareTo method will constrain the map's usage. If you're willing to champion the matter for a while, you could do a public service by starting a new question on http://ask.clojure.org .

I may have huge gaps in my understanding of "effects systems", but I thought that some of them had the goal of explicitly declaring, constraining, and/or making clear from reading the code, and checked by the language compiler/runtime, a delimited set of types of effects that executing a particular part of the code can possibly have.

eval has no limits on effects

Exactly that's all true I think but when you actually squint at the code it's just assoc/dissoc/update with maximal late binding, which is an s/f-expr, so it's like implementing eval with 3 operators, with the upsides and downsides [:set [:my :path] :my-val] === '(assoc-in [:my :path] :my-val) and the handler is just a reimplementation of eval-ing that. Eval with a guard pred could recover the "safety" element. I haven't thought about types since I have only seen them in dynamic languages.

I was considering a regex over the pr-str’d version but that was making my stomach turn a bit

The #function tag there isn't even how they print in a vanilla repl

(defmacro sanitize [schema]
  (walk/postwalk-replace
   {'int? :int 'pos-int? :pos-int 'keyword? :keyword 'string? :string}
   (pr-str schema)))

(sanitize [:or int? string?])

user=> clojure.core/int?
#object[clojure.core$int_QMARK_ 0x27f9e982 "clojure.core$int_QMARK_@27f9e982"]
user=>

(m/form [:or int? string?])
; => [:or int? string?]

Something is changing how the functions print in your environment, and if you don't know what is changing how they print l, seems dangerous to depend on it

(map type (m/form [:or int? string?]))
; => (clojure.lang.Keyword clojure.lang.Symbol clojure.lang.Symbol)

Thanks @U0NCTKEV8 for explaining what u mean. you’re right

also @U055NJ5CC thanks for showing m/form

I am going to try including links to http://malli.io in 400 status responses (at devtime only?)

(->malli-io-link [:schema
                  {:registry {"ConsCell" [:maybe [:tuple :int [:ref "ConsCell"]]]}}
                  "ConsCell"])
;;=> ""

Follow this example:

(class (into-array ["a" "b"]))
=> [Ljava.lang.String;

Type hint will be: ^"[Ljava.lang.String;"

If you care about when things happen, then I would strongly advise against using laziness.

Can you elaborate? I always care about when things happen. :)

you may need to cadd a call to seq in your when-let there at the bottom

These logs (the items fetched) are immutable, or so i believe. If that matters.

hard to say, but getting an infinite seq where you don't expect one is usually the same problem as getting an infinite loop where you don't expect one

In my experience, most of the lazy functions are intended for working with non-side effecting data in memory. If you try to use the lazy functions when you care about timing and side-effects, you're going to bump into all sorts of issues like chunking and caching that doesn't match your use case. It's almost always easier to just write eager functions that process side effects the way you want.

a problem in your base case

I doubt it the source of your problem, but your keyword has a typo :DesribeLogStreamsLazyList -> :DescribeLogStreamsLazyList

https://github.com/drewverlee/cloudwatch/blob/master/src/drewverlee/aws/cloudwatch.clj#L47-L51 doesn't pass in the pagination token

That's good feedback, I'll think about the lazy aspect here. I'll also look into my base case. Hopefully it's more obvious after a break.

so it seems like DescribeLogStreams->GetLogEvents! would always fetch the first page, and DescribeLogStreams->GetLogEventsLazyList! will produce an infinite seq of first pages

doh

👀

that doesn't explain why you would get a bunch of Ds printed out though

err, that typo is definitely an issue. Ouch and TY.

I guess part of the issue is i didn't have an easy way to debug it. When i evaled it, it crashed.

wrapping the iteration calls with lazy-concat seems very sus to me. You probably just want to do (seq (iteration ...))

(let [login-timeout (atom nil)]
  (reify javax.sql.DataSource
    (getConnection [_]
      (get-connection db-spec))
    (getConnection [_ _user _password])
    (getLoginTimeout [_]
      (or @login-timeout 0))
    (setLoginTimeout [_ seconds]
      (reset! login-timeout seconds))))

For reference: https://jdbc.postgresql.org/documentation/publicapi/org/postgresql/ds/PGSimpleDataSource.html

Note: I don't necessarily need to use Hikari here, but I do want connection pooling.

Maybe I need proxy here?

@U20LDTZBJ Are you using next.jdbc for this? You should be able to specify SSL with that and HikariCP...

The code you posted above with the login-timeout looks like what next.jdbc already provides...

i am using next.jdbc, and i've been having the sinking feeling im doing something Wrong™

the main thing I need to do is override getConnection such that the password for the connection is set to the result of reaching out to AWS every 14min to get IAM auth (tokens expire every 15min)

Ah... that's an... interesting requirement...

i did manage to do this:

(proxy [PGSimpleDataSource DataSource] []
             (getConnection
               ([] (get-connection db-spec))
               ([user pass] nil))
             (getLoginTimeout []
               (or @login-timeout 0))
             (setLoginTimeout [seconds]
               (reset! login-timeout seconds)))

I'm not certain that HikariCP supports that sort of usage does it?

what i can say is that with max-lifetime on the reify version, it was respecting my wishes and updating credentials according to the max-lifetime value

max-lifetime on hikari that is

https://blog.jdriven.com/2021/06/configure-hikari-connection-pool-when-using-rds-iam -- "The Hikari DataSource doesn’t support authentication token as the datasource expects credentials for the lifetime of the datasource. Meaning you’ll need to restart your service to be able to update the credentials, this isn’t a suitable situation when using RDS IAM. With the creation of a custom Hikari DataSource you can make it support requesting an authentication token."

so, i have read that, though i also found this https://kwill.dev/posts/hikari-rds-iam-auth/

I'm going to create an issue against next.jdbc and stick both of those links in it to have a think about how to make this easier out-of-the-box with next.jdbc...

you'll notice they originally override getPassword , but then switch things up a bit

my version is slightly different in that I call a function "get-connection", which calls jdbc/get-connection, and assoc :password onto the db-spec with the fresh token result

If you want to add notes to https://github.com/seancorfield/next-jdbc/issues/238 as you get things fleshed out, that would be very helpful.

though i will level with you, i have not been able to test this against an actual IAM DB yet. flying a bit blind, but trying to get something deployable while i wait

absolutely

(we're talking about RDS at work and so this may be something I need to tackle myself at some point)

i find it odd the posts i've seen don't seem to mention SSL, but then when you go and look at the amazon docs they're quite explicit about it

perhaps it's specific to Aurora or something, perhaps it's a somewhat recent change (AWS moves quick)

IME, the MySQL driver at least defaults to SSL and you have to explicitly turn it off if you don't need it...

(at work we explicitly specify useSSL=false for that reason)

a "fun" difference between reify'ing DataSource and the proxy version is that Hikari's max-lifetime does not seem to be honored in the proxied pgsimpledatasource situation and refreshes happen too frequently

oops, may have spoke too soon. also it appears that this SSL business may indeed be somewhat recent. either way, i appreciate you commenting and creating an issue.

@U04V70XH6 while i've been working through this, i've been wondering: do I even need Hikari in the mix? I noticed a pooled PG datasource class.

I know nothing about PG's built-in pooling (I don't use PG, only MySQL).

ah, gotcha

If a connection pool has to restart every 15 minutes then there seem less advantage in using Hikari or other 3rd party connection pool solutions. Hikari may still have some advantages for scaling connections, but suggest using postgres directly may fit into these constraints just as well