Fork me on GitHub
#clojure
<
2021-10-04
>
Lycheese12:10:21

While introducing caching to my app through clojure.core.cache.wrapped I stumbled upon behaviour, that I didn't find any documentation for:

user> (cache/lookup-or-miss cache-atom :test4 {:test4 4})
;; => 4
user> (cache/lookup-or-miss cache-atom :test5 {:test4 4})
;; => nil
user> (cache/lookup-or-miss cache-atom :test6 (fn [] {:test4 4}))
Execution error (ArityException) at clojure.core.cache.wrapped/default-wrapper-fn (wrapped.clj:36).
Wrong number of args (1) passed to: user/eval64637/fn--64638
user> (cache/lookup-or-miss cache-atom :test6 (fn [_] {:test4 4}))
;; => {:test4 4}
Docs:
clojure.core.cache.wrapped/lookup-or-miss
([cache-atom e value-fn] [cache-atom e wrap-fn value-fn])
  Retrieve the value associated with `e` if it exists, else compute the
  value (using value-fn, and optionally wrap-fn), update the cache for `e`
  and then perform the lookup again.

  value-fn (and wrap-fn) will only be called (at most) once even in the
  case of retries, so there is no risk of cache stampede.

  Since lookup can cause invalidation in some caches (such as TTL), we
  trap that case and retry (a maximum of ten times).
Maybe I'm just dumb, but it's not written anywhere that the value-fn gets e as an argument?

Lycheese12:10:11

Well the src explains it:

(def ^{:private true} default-wrapper-fn #(%1 %2))

(defn lookup-or-miss
  "Retrieve the value associated with `e` if it exists, else compute the
  value (using value-fn, and optionally wrap-fn), update the cache for `e`
  and then perform the lookup again.

  value-fn (and wrap-fn) will only be called (at most) once even in the
  case of retries, so there is no risk of cache stampede.

  Since lookup can cause invalidation in some caches (such as TTL), we
  trap that case and retry (a maximum of ten times)."
  ([cache-atom e value-fn]
   (lookup-or-miss cache-atom e default-wrapper-fn value-fn))
  ([cache-atom e wrap-fn value-fn]
   (let [d-new-value (delay (wrap-fn value-fn e))]
     (loop [n 0
            v (c/lookup (swap! cache-atom
                               c/through-cache
                               e
                               default-wrapper-fn
                               (fn [_] @d-new-value))
                        e
                        ::expired)]
       (when (< n 10)
         (if (= ::expired v)
           (recur (inc n)
                  (c/lookup (swap! cache-atom
                                   c/through-cache
                                   e
                                   default-wrapper-fn
                                   (fn [_] @d-new-value))
                            e
                            ::expired))
           v))))))
Would be nice to have complete docs though. Maybe I'll file a pr later. Are there any special rules for contributing to clojure.core libraries? (e.g. like the FSF copyright assignment for Emacs contributions)

Lycheese12:10:18

Thanks for the links I'll have to talk to legal about that 😅

Lycheese12:10:20

I wonder why they chose #(%1 %2) as default-wrap-fn and not just identity

p-himik12:10:49

Because they are not the same.

Lycheese12:10:21

yes I know I'm wondering why they introduce behaviour that does not follow the documentation

Lycheese12:10:35

to me the docs suggest that the return value of value-fn is taken as is

Lycheese12:10:46

at least when no wrap-fn is supplied

p-himik12:10:37

But the default wrapper does nothing but calling (value-fn e)? Not sure what you mean.

Lycheese12:10:51

which means that if you just supply a map it gets changed in my case I was supplying a map whose first key was the same as e which meant that my spec failed while I was expecting that the addition of a cache would not change the return value

Lycheese12:10:09

and if they don't match nil gets returned

Lycheese13:10:36

In my mind the default-wrapper-fn would need to be (fn [value-fn _] (value-fn)) to make sense together with the documentation.

Lycheese13:10:47

Wait a map isn't a fn I see now why you are confused

Lycheese13:10:34

So after my misunderstandings are stripped away the core problem is: the arguments for value-fn, if no default wrapper is specified, are not documented The stuff with maps is just undefined behaviour since it expects to deal with functions and not maps

p-himik13:10:55

> a map isn't a fn It actually is. > the arguments for value-fn, if no default wrapper is specified, are not documented Agree.

p-himik13:10:27

user=> (ifn? {})
true

Lycheese13:10:31

Huh. Is there a way -- apart from calling map? on the value-fn -- to allow both a "regular" function and a map to be passed to the wrapper function and have that wrapper function return the functions return value or the map itself respectively? But if there were, how would a function itself be put in the cache…the level of "unpeeling" the layers are different for a map and a function in this context, so it is probably impossible to get consistent since it requires guesswork on the program's side?

p-himik13:10:21

Just pass a function that, when needed, returns a map itself.

p-himik13:10:52

If value-fn returns something that can be called, it doesn't matter - its return value is never called.

Lycheese13:10:57

Yeah probably easier that way. Works just fine now. On to finding out how to parse a Java LocalDate in cljs

Lycheese13:10:23

Or figuring out how to pass a member function to update-in

Lycheese13:10:03

nvm figured it out

p-himik13:10:28

> how to parse a Java LocalDate in cljs CLJS wouldn't even know about Java. You have either a string or a number there - just parse it with goog or js/Date if the format is something those two can digest. > how to pass a member function to `update-in` Just wrap it in #(). Or full (fn [...] ...).

Lycheese14:10:14

It shows up as TaggedValue: LocalDate, 2020-10-28 when converted with str I thought it referred to Java's LocalDate since js doesn't have LocalDate, I think? I'm just converting the dates to strings before populating the API endpoint now

p-himik14:10:28

Don't use str on a Java date - use a proper formatter to get a proper ISO 8601 string, or in some other format you prefer.

p-himik14:10:25

In general, you usually don't want to call str on any Java object - instead, you want to call some other function to get strings in proper format. But there are exceptions of course. Especially in the simplest cases, like numbers.

Lycheese14:10:42

I only used str to debug since I had a component in cljs (`[:p date]` which caused my site to crash because it couldn't be rendered. The value above is in cljs not clj.

Lycheese14:10:46

That is also the reason for the member function in update-in . I call a function that gets me a map from the database and that map includes a LocalDate which until now was passed as is to a ring response/ok . I am now calling .toString on it after getting it from the db, but before passing it into the response.

Hagenek13:10:51

Anyone know of a good example of building a GraphQl API in clojure?

emccue14:10:55

look at lacinia and lacinia pedestal

Hagenek15:10:05

Thanks 😃

vlaaad16:10:58

That's how we do it, I don't know if it's a good example since it departs from what is usually advised in the docs, but I try to keep it as simple as possible

Ian Fernandez15:10:28

by tldr means, what's the difference between a flatland/ordered-map and a clojure.core/sorted-map ?

Lennart Buit15:10:29

I think the former is on insertion order, whereas the latter is on order of keys

Janet A. Carr16:10:30

Hey folks, can someone help me? I'm trying to use clj-http to request pdf binary and spit it into a file, but I can never seem to get the output right. I would like to do some akin to this with clojure

curl -X POST  \
 -H 'Content-Type: application/json' \
 -d '{
"neccessary_data": "here"
 }' \
 -o 'document.pdf'

Janet A. Carr16:10:28

So far I've tried directly spitting the response body into a file. Tried coercing as bytes. Tried coercing to an FilterInputStream and reading each byte into a byte array.

emilaasa16:10:16

Do you need to use clj http or can you use something else?

emilaasa16:10:41

Is (slurp "") ok?

p-himik16:10:56

Just in case - the PDF you're getting with curl is generated by curl itself - that request does not actually return a PDF.

Janet A. Carr16:10:56

I figured it out. of course as soon as I ask for help I figure it out. Classic. Thanks for the quick responses! 🙂

emilaasa16:10:36

I liked this SO suggestion:

(
 (:body (client/get "" {:as :stream}))
 (java.io.File. "test-file.gif"))

Nom Nom Mousse17:10:17

What is the best way to find the sha256 hash of a Clojure map? The below works, but feels a bit inefficient (it both sorts and calls toString):

(def sha256 (java.security.MessageDigest/getInstance "sha256"))
(def m {:a 1 :b 2})
(.digest sha256 (.getBytes (.toString (sort m)))

phronmophobic17:10:01

you can check the current hashing implementations. They don't use sha256, but the approach should be similar, https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/APersistentMap.java#L104

👍 1
kirill.salykin17:10:44

Sont think it is safe to rely on clojure hash implementation, it may be changes without notice

👍 1
Nom Nom Mousse17:10:18

I suspect so too, but he might have meant I should look at them for inspiration 🙂

👍 3
hiredman17:10:12

there are libraries that have been written for this kind of thing too, like https://github.com/arachne-framework/valuehash

🆒 3
Nom Nom Mousse17:10:54

That library seems like the best alternative 🙂

Nom Nom Mousse17:10:17

The readme does not contain lein coordinates XD

Nom Nom Mousse18:10:56

Yes, that is what I need, thanks. The valuehash one is not stable across versions :)

borkdude18:10:13

@U0232JK38BZ I've also seen people use a bencode representation and then sha-256-ing it. It works if you're not interesting in any other types than integers, strings, arrays (lists) and maps

Nom Nom Mousse18:10:14

Interesting approach. But I guess bencoding maps might give different results since maps are unsorted?

borkdude18:10:23

in bencode maps are sorted by key

borkdude18:10:57

only string keys are supported

Nom Nom Mousse18:10:48

Cool! I think I like the bencode-approach the best

Michael Mackenzie18:10:44

Hey guys! I've worked on a couple of hobby projects in Clojure(script), but only just recently got the opportunity to finally use Clojure professionally for a new project at my full-time job! The project involves a web API ingesting and serving analytics data to/from mongoDB. My hobby projects were not built to the same standard of engineering rigour as is expected in a professional, production environment, so now I'm starting to have to deal with different challenges, including testing. The last time I was doing backend development professionally was in Java using Guice, and I would have structured my app in such a way that the DB layer would be behind an interface, of which there would be a mongo implementation and a dummy implementation for tests. This implementation would be provided via @Inject and from the API code, data could be transparently accessed via something like client.getDataForThingyId(id) , which might hit the database or the mock implementation depending on how the injector is configured. In clojure, I am struggling to find good solutions for the following problems (which maybe aren't even problems but my mind is too accustomed to years of OO): • There's something really nice about being able to wrap a DB connection with an app-level "client" so that I don't constantly need to pass the DB connection to make queries. In Clojure, I always see conn just passed as a parameter, but it seems so repetitive. I'm just managing the connection via mount, so on every call i need to pass in the defstate value for the connection, or bake that connection into the functions themselves • Along with not having the state + behaviour together like a client object that wraps a DB connection, I am also not sure what the best way to change the implementation in a test situation, so that my API endpoint handlers don't hit the real database in the unit tests. Does anyone have any wisdom, recommendations or resources for how to think about these types of problems in Clojure? I am absolutely in love with Clojure, but am struggling a bit to understand how to design my application with production use and testing in mind

👀 1
👍 1
p-himik18:10:15

> I always see `conn` just passed as a parameter, but it seems so repetitive This is that case where repetition is good. It's explicit, it's obvious, it's simple, there's no magic. Anything else makes it easy on the surface but actually creates more problems. As you can guess, this is not a new problem and multiple people have tried all sorts of solutions. A lengthy discussion, if you're interested: https://groups.google.com/g/clojure/c/fRi554wbPSk

hiredman18:10:52

I prefer component because swapping in a different implementation for tests is as a simple as using assoc

vncz18:10:31

I know this is JavaScript but I usually solve this using the Reader: https://dev.to/vncz/from-0-to-reader-1g2d

Michael Mackenzie18:10:23

thanks, i'll take a look at those links and component

César Augusto18:10:56

Hi @UKKG61LVA how are you? • About the first point, from my experience working with Clojure, it is common to pass the conn as parameter to the function that will perform the database access... this way this class will not keep an instance of conn. I think the idea is to avoid keep state, this way you can test the functions using the instance of conn you want (mock test instance for example) • For different implementations of conn I normally use eh component https://github.com/stuartsierra/component. In the startup of the app you can decide if you are going to run for test (Local or CI) or prod environment, and depending on the environment you can have a certain implementation. I have this https://github.com/cesaralcancio/service-clojure using pedestal IO + components. It is not production ready but you can have an idea on how to use component and inject the DB instance as param to the function that handle the http requests.

hiredman18:10:42

I've been in so many mount vs. component discussions that someone actually saved one as a gist https://gist.github.com/pandeiro/9a1c8fd431e1b4c78c99

Michael Mackenzie18:10:49

I think I chose mount just because it was more recently updated on github, but I think this is likely just due to component being "done" basically

borkdude18:10:38

@UKKG61LVA recently updated isn't a good indicator of what to choose in clojure land. there are many libs that are close to done.

☝️ 2
borkdude18:10:04

component is a good default imo

Michael Mackenzie18:10:24

I'll give it a thorough read-through, thanks

borkdude18:10:43

integrant is also mentioned a lot

emccue18:10:44

Even if you don't use component or integrant or whatever - the usage site should just be "get a map, destructure what I need"

didibus19:10:05

The truth is, people have their own preference but they're all fine choices: mount, component, integrant, etc.

didibus19:10:21

I would say if passing in the DB connection seems repetitive to you, you just need to get over it. You mentioned OOP, well in OOP you always pass in the state on every method call:

state.method(method-args...)
In Clojure it is just as repetitive, no more, no less:
(function state function-args...)
Why are you not bothered by it in OO but are in Clojure?

1
Michael Mackenzie19:10:42

that's a very good point. I'm watching the Stuart Sierra talk "Components - Just enough structure" and I think i understand better how it solves this problem now

👍 2
didibus19:10:58

For mocking, you can either mock the DB itself, or you can mock your query functions themselves. My take though is that if you need to mock them for unit test, you've failed to design your application code properly. You shouldn't need to unit test your DB code, you should only need to integ test it with the real DB. This is achieved when you've properly seperated pure business logic and I/O code. When you've done that, your I/O code can be fully integ tested, and your pure business code can be fully unit tested, and nothing needs to be mocked. But yea, this can be difficult to achieve sometimes, so I'm not making it a black/white, but ideally you'd get as close to this as possible.and there's be less and less things that you need to mock in unit tests as you approach this ideal.

Michael Mackenzie19:10:49

Yes I agree. I wouldn't be actually mocking a database in a unit test, I would be mocking the interface that it provides data through. I think the solution inherent in using component (just associng a different implementation of the data provider that doesn't use a DB at all) would work very nicely for what I need to do

1
Michael Mackenzie19:10:22

I intend on having integration tests against an actual local mongo instance to test the functions that provide data from the DB and unit tests that use the ring test libraries to test the API side

didibus19:10:44

Ya you can do that. Though I'd say if you have your pure logic unit tested. The API might not benefit much from unit tests either, and you could just have an integ test on the API.

Michael Mackenzie19:10:55

like run an actual web server and hit the actual database for the whole thing?

Michael Mackenzie19:10:05

Yeah that probably makes sense and would be the highest fidelity

didibus19:10:50

Ya, it can be nice to also have some unit tests on the APIs, so not saying don't do it. But I think unit tests on business logic and integ tests on the APIs is already very good coverage.

didibus19:10:46

Cause basically you want something like this:

(defn api
  [request]
  (let [step1 (pure-request-processing request)
    step2 (fetch-additonal-stuff-from-db db)
    step3 (pure-processing step1 step2)]
  (case (pure-branching-logic step3)
    :persist (write-to-db step3)
    :else (pure-response-builder-logic step2))))

didibus19:10:40

So here you'd unit test all the pure pieces. You'd integ tests all the IO pieces. And now the last thing that needs to be tested would be the overall orchestration flow of the API itself. Which you can do with a unit test and mocking of all the functions in the API, or at least all the IO ones. But you can also just as well test it with a integ test over the API as well.

didibus20:10:57

I think I tend to favour the integ test for the API cause I find it easier to setup. It gets really hard mocking a lot of scenarios because you really need to understand ok for some input and some DB state, what am I supposed to be getting back at each step that I should mock? And over time, I find the mocking more brittle, as people refactor things internally, the mocking of the API is also broken, and needs to be refactored, where as the API integ test wouldn't be broken unless you actually broke existing functionality.

raspasov05:10:27

Use component, I’d recommend.

1
Patrice21:10:51

hello all, anybody knows if functions in clojure are late bound like in common lisp?

didibus22:10:17

I don't know if "late bound" is the correct term, but yes, the actual function to use is looked up at runtime from the Var mapped to the symbol.

didibus22:10:10

But unless you use a multi-method or protocol, there is no logic to select one of many possible functions, the Var always points to one and only one function, but the function it points too can change dynamically as the program runs.

didibus22:10:17

So it goes: Namespace -> Symbol -> Var -> Fn And at runtime, the arrow from Var -> Fn can be changed dynamically.

Patrice22:10:54

that's even what happens when the function is compiled right?

Patrice22:10:59

in the repl I mean

Patrice22:10:23

gotta love lisp!

didibus22:10:05

Everytime you call def or defn again for an existing Namespace -> Symbol -> Var -> Fn it swaps the Fn for the new one

Patrice22:10:31

I was kind of afraid clojure would only support early binding and just replace the function at runtime, so all good

Patrice22:10:48

really love the way it works

💯 1
didibus23:10:01

There are a few more advanced constructs that I'd say have some quirks in being hot-swapped, but there are ways around them. I'd say they are mostly defmulti, protocols, gen-interface, gen-class and such. They're used much less often, and for most of them you can still do it, just it sometimes is a bit more involved than just send to REPL.

didibus23:10:17

The one I encounter most often is defmulti, when you want to change the dispatch function, the defmulti call actually no-ops if it is already defined, so you need to set it to nil first, and then redef it. Or some people put the Var in the defmulti explictly to the dispatch function, instead of putting the dispatch function directly.

didibus23:10:23

And with protocols, it's more that if you redefine the protocol, you need to also redefine the type extensions for it cause they get invalidated when the protocol changes.

Patrice23:10:00

very good info thanks, do you know of any resource I can use on how the compilation works, I'm looking into the source but there's a lot in there, some doc would be useful

didibus23:10:53

Finally, there is one case where the Var indirection is removed, that is when you do an AOT compilation with direct linking. The "direct linking" which is for performance reasons, what it does is actually remove the indirection with Var, so it links things directly to the function which means they can't be rebound anymore. And its good to know that clojure.core is AOT compiled with direct linking for performance, so it means you can't rebind core functions, you can only shadow them, but the shadow won't be seen by other core functions that use it, only by your code.

didibus23:10:55

Hum... unfortunately the best I know is the official reference https://clojure.org/reference/reader but it is a little short on all the details. I've kind of gathered the knowledge over time and it lives in my head haha.

Patrice23:10:32

lol, thanks for sharing it!

Patrice23:10:11

you gave me a lot of pointers I can lookup, so thanks for that

didibus23:10:27

For the compilation, a general description which maybe can help you if you navigate how to find more info. Basically, Clojure has a custom class-loader, and a load function. The load function will look for clojure source files as libs on the classpath, and for the one to be loaded, it will read, compile and evaluate each top level form in the source file in order from top to bottom. If it encounters another call to load (which ns eventually delegates too), it will stop what its doing and go load that in turn and when done continue where it left. That's how it loads things in-order. So when loading, the file is parsed, and for each top level block of source, it is sent to the reader, that parses the string of text as an AST data-structure. Then the macros are evaluated, they are given their chunk of the parsed AST to process where they return a possibly transformed version of it that replaces the existing chunk of AST with whatever they returned. Next, the AST for the block is sent to the compiler for compilation. The compiler uses https://asm.ow2.io/ to create the byte-code from the AST. Generally what will happen is that each function or macro becomes its own Java class. Namespace are their own class as well. Etc. Finally, all of the new generated classes (as byte-code) by the compiler is dynamically loaded into the JVM run-time using Clojure's custom ClassLoader called the DynamicClassLoader (because it can load classes dynamically from memory).

didibus23:10:17

But notice how I said this all happens one top-level form at a time. That's the key part, the compiler does thing piecemeal, which is what enables the REPL to work. Because at the REPL you send a block of source, and so the compiler is able to do its job just on that, since that's the level it operates at all the time anyways.

didibus23:10:28

And off-course, all of this machinery to load, read, compile and evaluate is always available at runtime, unlike say in Java, where the compiler cannot be called into at runtime since it doesn't exist in the app, in Clojure, the compiler is always a part of your app.

Patrice00:10:55

I'm in the source for this exact part, beautiful code

Patrice00:10:31

again, great info, thanks a bunch

Joshua Suskalo21:10:12

You will get compiler errors if you reference functions you have not yet defined. We have the macro declare if you want a forward-reference.

hiredman21:10:53

there are two types of binding names to values in clojure (excluding creating named jvm classes), top level definitions created via def, which create vars, which are named mutable cells that contain something, and evaluating that name produces whatever is in the cell

hiredman21:10:45

the second is "locals" which are immutable bindings of names to values which are introduced via function application, let bindings, try/catch, basically everywhere else

hiredman21:10:42

locals basically don't exist at runtime

Patrice21:10:54

does that apply to function too?

hiredman21:10:09

functions are values

hiredman21:10:01

there are also optimization features that will remove levels of indirection in the var case when used

Patrice21:10:20

ok, makes sense, so say, when I compile a function and send it in the repl, the binding is changed from the old function to the "new" one, correct?

hiredman21:10:55

more or less

Joshua Suskalo22:10:59

Keep in mind that if you passed a function from a var as a value to some long-running process, then re-defining the function won't update the function as used by the long running process, because it's already had its value fetched from the var and is now just a value floating through your program.

Patrice22:10:35

here's why I ask, did a simple test with http-kit, I modified the handler, compiled it, send it to the repl, but I'd need to restart the server to get the change

Joshua Suskalo22:10:50

this is the "problem" I just referenced

Joshua Suskalo22:10:36

If you pass the var itself (by #'handler) then it will likely work as you expect. You can't generally use vars instead of values if you want them to live-update, but you can call a var like a function and it will deref it.

hiredman22:10:42

(http-kit/whatever my-handler) <- arguments are evaluated before functions are called, so my-handler is evaluated here

Patrice22:10:58

ok, and that.d be a real live-update, not with the server restarded in the background correct?

hiredman22:10:58

I believe this would be the same in common lisp, you are just less likely to encounter it because being a lisp-2 directly passing a function as an argument is annoying

Patrice22:10:54

in CL there's a symbol table associating the function name and the function itself, on update, only that binding is updated

hiredman22:10:17

I think a transliteration into common lisp is something like

(defun foo (x)
  x)

(defun bar (f)
  (lambda (x)
    (funcall f x)))

(setq q (bar #'foo))

(defun foo (x) (+ 1 x))

(funcall q 1)
where the final line is still evaluates to 1 even though foo has been redefined

Patrice22:10:05

was wondering if it's the same in clojure

hiredman22:10:20

it isn't about updating, the issue is evaluating arguments before function application

hiredman22:10:38

and then mutating the binding and expecting the value the function was applied to change

Joshua Suskalo22:10:46

it's like rewriting a lambda you passed to a function a while back and re-evaluating it. It won't update the existing value

Patrice22:10:30

yes, makes sense, all good

Patrice22:10:01

thanks for the info

Patrice22:10:07

(both of you)

hiredman22:10:37

you can also wrap an anonymous function around handler to cause it to re-evaluate the handler name every time the anonymous function is invoked

hiredman22:10:58

#(handler %)

hiredman22:10:42

but var quoting #'handler is far and away the common idiom