Fork me on GitHub
#clojure
<
2020-10-29
>
Jakub Holý (HolyJak)08:10:48

Hello! I struggle finding a good way how to perform parametrized initialization only once in the face of concurrent calls. What I have now is:

(defn get-or-init-adoc [atom params]
  (compare-and-set! atom nil (delay (some-expensive-init-reading-files params)))
  @@adoc)
Is there a better way (then delay inside an atom)? What I essentially want is (atomic-if (initialized? x) x (init+return x, params)) where x cannot be changed between I check whether it is initialized and initializing it. Then, inside a frequently called function I do (defn render [params file] (.convert (get-or-init-adoc adoc params) file))

Ed09:10:02

what's wrong with just having a (def x (delay (some-expensive-init params))) ?

Ed09:10:10

where are the params coming from?

Ed09:10:18

there's also promise which you can deliver once and will block all defref s until it's delivered ... but you need to be careful not to create deadlocks where the read calls block all the running threads before any thread can be started to deliver the promise

Ed09:10:06

or maybe (defn get-or-init-adoc [atom params] (swap! atom #(when (nil? %) (some-expenisive-init params)) ?

Ed09:10:33

I guess you're trying to make sure that the init only gets run once, and have the function lazily initialise some value?

3
Ed09:10:48

so maybe you could memoize the function?

borkdude09:10:17

note that memoize is not suited for non-referentially transparent functions

borkdude09:10:35

and it doesn't guarantee that the underlying function will be executed once

borkdude09:10:08

you could use locking to lock some sentinel, which will probably be the least headachy

3
noisesmith16:10:00

I don't really see the benefit of putting an atom outside the delay here, the delay already locks and will only execute once

Jakub Holý (HolyJak)18:10:26

The problem is I need to do the initialization inside a function that gets 'params' as it's argument. And I don't want to unnecessarily create a new delay upon each its call.

Jakub Holý (HolyJak)18:10:16

swap! doesn't work because it can execute the function multiple times.

noisesmith19:10:51

agents never retry, fwiw

noisesmith19:10:06

so sending an fnil to an agent should work

👍 6
Jakub Holý (HolyJak)20:10:48

Thanks! I guess careful use of locking is the simplest solution here...

Eugene Koontz14:11:38

I had the same problem (doing something slow and expensive that should only be done once per process) and came up with the following solution: https://github.com/ekoontz/menard/blob/master/src/menard/nederlands.cljc#L242

Eugene Koontz14:11:58

and then every usage of the model calls (load-model) , although the actual loading is only done once : subsequent calls to load-model simply deferences the ref

noisesmith01:11:36

@U017Y2J9F0W delay might be a simpler way to get that result (depending on whether you want to re-execute)

Eugene Koontz19:11:02

ahh, I do want to re-execute, so I can reload the models, so a delay won’t work, as you said. So my original code would be better.

noisesmith19:11:52

also, there are dedicated libraries for stateful initialization, which take things like cross-ns dependencies and re-initialization into account - eg stuartsierra/component and integrant

👍 3
Eugene Koontz19:11:09

i feel like I have to work through the primitives (what the core clojure libraries provide) to even appreciate the dedicated extra libraries

vncz13:10:23

Do you know what's a free service to quickly deploy a Clojure Deps.edn application?

vncz13:10:39

I would normally use Heroku but they do not support Clj tools out of the box and I need to do to many changes to my project

zilti13:10:09

Make an uberjar and deploy that one

vncz13:10:15

Yeah I guess that's the easiest way. Any chance you have a link about producing an uberjar with clj.tools @zilti

vncz14:10:46

2020-10-29T14:47:15.148369+00:00 app[web.1]: Error: Invalid or corrupt jarfile app.jar

vncz14:10:57

Interestingly enough, the deployed jar on Heroku seem to be broken or something :thinking_face:

borkdude14:10:32

@vincenz.chianese Does it work locally though?

vncz14:10:50

Yeah, I think I made a mistake in the jar execution, let me try again 🙂

vncz14:10:08

I was specifying a wrong namespace

vncz14:10:20

Ok that works now, but it seems like I'm having issues running datomic dev-local on heroku

borkdude15:10:10

the name doesn't suggest you should run it in production maybe? just guessing

vncz15:10:55

No, I think it's because I'm using a wrong directory name, let me triple check…

vncz15:10:08

Ok I got it working. It takes ages to startup but eventually it works

elton20:10:41

Hi does anyone have an idea of how I would do this? I’m trying to conditionally define functions using a global lein profile depending on whether I have certain libraries loaded, ex. something like this where I’m developing several libraries at once, some of which are childrens of the others. Say if the dependencies looked something like

[my.lib5 [my.lib1 my.lib2 my.lib4]
 my.lib4 [my.lib1 my.lib2]
 my.lib3 [my.lib2]]
and basically the workflow would be something like while i’m in a repl for my.lib3, to have all the custom function definitions defined using variables defined in my.lib3 and my.lib2 and so on. So I have something like this being loaded via {:repl {:repl-options {:init (load-file "~/.lein/startup.cljc")
;; load libs if they exist
(let [repl-deps ['[my.lib1 :as l1]
                 '[my.lib2 :as l2]
                 '[my.lib3 :as l3]
                 '[my.lib4 :as l4]
                 '[my.lib5 :as l5]
                 '[clj-async-profiler :as prof]
                 '[criterium.core :as crit]]]
  (doseq [dep repl-deps]
    (try
      (require dep)
      (catch Exception e nil))))

;; Create custom functions depending on what's loaded
(let [namespaces (->> (all-ns) (map ns-name) (map name) set)
      loaded? (fn [& args] (every? namespaces args))]
  (when (loaded? "my.lib1")
    (defn custom-fn-1 (l1/foo) )
  (when (loaded? "my.lib2")
    (l2/init)
    (defn custom-fn-2 (l2/bar)))
  (when (loaded? "my.lib2" "my.lib4")
    (l4/init l2/options)
    (defn custom-fn-3 (l4/baz)))
But it doesn’t work because I get a compile error No such namespace: l4 when in a repl for my.lib1, my.lib2 and my.lib3 . So I’m wondering, is there anyway to solve this without too much mucking about? Or is the only way to solve this to have a compiler option *allow-unresolved-namespace* in the compiler which would be used in resolveIn in Compiler.java?

borkdude20:10:47

@elton.law This is called the Gilardi scenario: https://technomancy.us/143

borkdude20:10:52

I could have sworn he was on Slack here as well as @sgilardi

Darin Douglass22:10:35

i actually asked him (since i was pairing with him at the time) if "Gilardi scenario" was a coincidence. totally wasn't 😛

borkdude22:10:02

why would it be a co-incidence?

Darin Douglass22:10:28

I figured given the community it’d be pretty unlikely it WASNT him. But still wanted to ask

scgilardi14:10:07

nice little bit of nostalgia courtesy of @technomancy and you two. :)

borkdude14:10:33

I'm honoured to finally meet the person behind the name :)

scgilardi14:10:35

🙂 thank you! I’m honored and happy to meet you! Thanks so much for your work and initiative in producing such useful and clever tools for us.

pez08:06:03

OMG. Mr Gilardi Scenario himself!

ghadi20:10:11

Gilardi issue is secondary symptom - the root issue is you are throwing away your exceptions @elton.law . Who knows what that is suppressing

ghadi20:10:49

But a good hypothesis is that lib4 is not loading initially, being skipped silently, causing a subsequent compilation error later (failure to resolve symbol l4/init )

ghadi20:10:22

but also good to understand the gilardi scenario

elton20:10:25

Umm forgive me if I’m misunderstanding but I don’t think the Gilardi scenario is related, I’m working on a project that has about 20 different individual projects, and basically, sometimes I will be able to do a require on that my.lib4 but sometimes I won’t (because it’s not a dependency)

elton20:10:06

and I want specific blocks using variables from my.lib4 to not be run

dpsutton20:10:15

you can't compile a form containing the symbol l4/init even if its behind a when form if the l4 alias isn't present

dpsutton20:10:56

but you are attempting to

elton20:10:15

I’d like to, that way I can have it all in one startup.clj file

elton20:10:30

and then say if I have multiple projects that use criterium, I can have those all load

elton20:10:37

only when its avaialble

elton20:10:42

if that makes sense

dpsutton20:10:57

right. and that seems to be exactly the Gilardi scenario

elton20:10:28

hmm okay, maybe i’ll take another look at it, thanks everyone

dpsutton20:10:23

i think you could easily get around this with requiring-resolve

👍 3
seancorfield20:10:49

^ I was about to suggest that.

seancorfield20:10:16

Bear in mind that it will throw an exception if the namespace can't be required.

elton20:10:48

Right right, because it would be a runtime thing and okay, after a second reading it totally makes sense now thanks again everyone

seancorfield20:10:08

(when-let [foo (resolve 'l1/foo)]
  (defn custom-fn-1 (foo)))
I think that would work

👍 3
seancorfield20:10:11

That way, it will only resolve if l1 is loaded. Well, if it's loaded and l1/foo resolves 🙂

joshkh22:10:27

with the understanding that it's not the best practice and for good reason, is there a way to fetch the latest commit of a branch via deps.edn, rather than a specific sha?

seancorfield22:10:30

@joshkh I have an example in my dot-clojure repo, if you've seen that?

joshkh22:10:59

nope! searching now

joshkh22:10:14

exactly what i was looking for. just to test my understanding, extra-deps brings in tda:add-lib3, which then handles the :add-libs key of deps.edn? nvm i misread the indentation

seancorfield22:10:49

(relies on the add-lib3 branch of t.d.a.)