Fork me on GitHub
#clojure
<
2020-01-09
>
Ivan Koz00:01:09

how to decide, i mean

noisesmith00:01:13

for functions that are mutually recursive and require closed-over data

Ivan Koz00:01:13

understood, ty

noisesmith00:01:01

I wonder if you could replace your nested conditionals with one cond - eg. (let [idx (mod n 3) (cond (and odd-n? (= idx 0)) ... (and odd-n? (= idx 1)) ... ... (and (not odd-n?) (= idx 0)) ...)

noisesmith00:01:40

just a thought - not sure that's an improvement

Ivan Koz00:01:42

well, i'm also looking for a way to make it less verbose but not too clever\\complex

noisesmith00:01:43

or even (case [odd-n? (mod n 3)] [true 0] .... [true 1] ... ... [false 0] ...)

noisesmith00:01:58

right, that's an important balance

noisesmith00:01:57

also you don't need odd-n? to be a captured value, it could be calculated in the let block of move

Ivan Koz00:01:10

yeah making cond cases based on odd-n? looks like a logical improvement

Ivan Koz00:01:17

well capturing odd-n? is a premature optimization from my side

noisesmith00:01:51

I wouldn't even think about "optimization" there, yeah - it's more a question of "does this make the code more straightforward, or not"

hiredman00:01:05

looks like iterate

hiredman00:01:59

by which I mean, you can rewrite it as single step, getting rid of building a seq and recursing, and use iterate to do that

Ivan Koz00:01:55

but iterate makes of inf seq isnt it?

noisesmith00:01:13

(take-while some-pred? (iterate ....))

Ivan Koz00:01:43

that's a great idea ty

noisesmith00:01:42

another thing to consider - instead of [from to spare] it could internally use {:keys [from to spare]} the advantage being that the new state can be something like (-> state (update :from pop) (update :to conj (peek from))) (spare doesn't change in this case, no need to mention it)

noisesmith00:01:03

I think that (plus some line breaks) make sit more clear what operation is actually being done

noisesmith00:01:41

of course the printed form should still be a vector, that's much better than a fully named hash map

Ivan Koz00:01:10

great, thank you so much for the valuable feedback, i'll get back to you as i'm done with above improvements

noisesmith00:01:56

hope it helps - these are all pretty subjective

Ivan Koz00:01:13

well i see a valid point in all of them

Ivan Koz02:01:43

Is there any difference (from the point of lazy evaluation) between lazy-cat lazy-seq + cons or just cons if i need to concatenate a value and a lazy sequence?

Ivan Koz02:01:00

i mean lazy-.. stuff will wrap first argument into LazySeq and using it in case of already existing first value has no sense right?

yuhan02:01:18

cons evaluates both its arguments whereas lazy-seq defers evaluation until the sequence is realized

yuhan02:01:03

((fn zeros []
   (lazy-seq (cons 0 (zeros)))))

yuhan02:01:46

Using cons for that would produce a StackOverflowError

Ivan Koz02:01:44

@qythium yeah i know that cons evaluates its argument, but my case is (cons 0 (lazy-seq ...

Ivan Koz02:01:38

since 0 is already there, using lazy- stuff instead of cons has no meaningful reason right?

yuhan02:01:29

yeah I suppose, since you don't care about 0 being evaluated

Ivan Koz02:01:19

exactly what i thought, ty

JAtkins03:01:44

Can defs be marked for running by the compiler? e.g. statically run this

JAtkins03:01:17

(def v1 {:key :val})
(def v2 {:key2 :val2})

(def v3 (merge v1 v2))
so that the program startup doesn't get hung up with some small calculations?

Ivan Koz03:01:11

you mean you want delayed merge?

JAtkins03:01:57

Not quite. I'd like to have the compiler replace the merge with {:key :val :key2 :val2} by running the code like macros are

JAtkins03:01:50

I have dozens of this kind of chained def running at the start of my cljs app, and it's starting to take a hit

Ivan Koz03:01:24

@jatkin you can also ask in #clojurescript, i never touched cljs so can't help you with compiletime

JAtkins03:01:59

Would that be redundant? Seems like it should be the same for both clj and cljs

JAtkins03:01:37

Macros are done by the clojure compiler typically.

Ivan Koz03:01:28

well on jvm compilation and evaluation are mixed, so technically there is no difference, it's just that macros are expanded before evaluation all happens on runtime

JAtkins03:01:22

So, this will happen by itself on the clojure compiler?

Ivan Koz03:01:27

thats why you should ask that question at #clojurescript most likely there is someone to help you who knows cljs specifics of macroexpansion.

JAtkins03:01:51

I may be misinterpreting. Ok, sounds different enough. Thanks!

seancorfield03:01:33

In Clojure, those chained def s will be evaluated when the namespace is loaded -- so in that respect it's going to be the same for both Clojure and ClojureScript @jatkin

seancorfield03:01:12

So if you do that a lot in Clojure, you will take a hit at startup, just like in cljs. My question would be: what problem are you solving to arrive at your current solution? (perhaps there are more performant alternatives?)

JAtkins03:01:18

I'm using code to save myself a bunch of typing for come config stuff. I have a tool with about 50 config parameters in one part, each one applicable for different items that may appear on the page. To keep it minimal I have code running in this way to dedupe alot of the config possibilities for the ~30 types of options.

seancorfield03:01:59

Is this just fixed data, computed just once at startup time?

seancorfield03:01:56

How about defining it in a separate process that you run to compute the full set of data that writes it to a cljs file in fully-expanded form, as a preprocessing step before building you cljs app to js?

Ivan Koz03:01:07

@seancorfield i think in that case it would be fixable by a macro

JAtkins03:01:26

All fixed. Possibly. I was about to write a thunk macro that forces the code in question to run in the compiler, maybe building up a map. If I actually needed the intermediate values I can have it write into a map and assign each def to one of the keys of the map

seancorfield04:01:49

@nxtk Do you want to have a stab at that macro? I think it's harder than it looks because it relies on combining runtime values -- it's not just compile-time stuff...

Ivan Koz04:01:14

oh i'm far from cljs, i only know iterop is different, and it translates to js by google closure

seancorfield04:01:43

@nxtk Even in pure Clojure...

Ivan Koz04:01:06

@seancorfield whats hard about it? I probably doesn't see what are you talking about.

(def x 5)
(def y 3)

(defmacro m-sum
  [a b]
  `(def z (+ ~a ~b)))

(m-sum x y)

(println z)
(println (str (macroexpand '(m-sum x y))))

Ivan Koz04:01:51

x y are runtime, and m-sum is at macro-expansion stage

Ivan Koz04:01:26

i mean lisp macros are just a functions which take forms as input and produce forms as output, so i'm not sure how you look at it in that case

Ivan Koz04:01:36

please elaborate, so we can be on the same page

hiredman04:01:57

You are are generating a form that does the sum at runtime

Ivan Koz04:01:20

ok that's true

hiredman04:01:40

Which would be the same as what you already have with the merges

emccue04:01:12

if your data is strictly a valid return for a macro then you can do something like this

Ivan Koz04:01:48

no i mean in cljs macro will be evaluated on compile time right?

hiredman04:01:58

The reason it is tricky is you have to manually do the resolving of symbols to values (at macroexpand time) and them call whatever function on the values. And it is even trickier because the macro language for cljs is clj not cljs

emccue04:01:19

(defmacro compile-eval [& body]
  (eval body))

(def [a b c] (compile-eval (let [x 3 y 5] [x y (+ x y)]))

hiredman04:01:32

The macro is expanded to the code, the code them is compiled an run as normal

emccue04:01:47

ymmv though

emccue04:01:20

ah, nvm, def doesnt destructure

Ivan Koz04:01:25

yeah i see 😃

seancorfield04:01:18

It's one of those "simple, obvious" problems that you tend to think just ought to be straightforward with a macro... but the compile-time / run-time divide is hard to cross...

JAtkins04:01:53

(defn run-chain [items]
  (reduce
    (fn [acc [k f]]
      (assoc acc k (f acc)))
    {}
    (partition 2 items)))

(defmacro compile-eval [& items]
  (eval `(run-chain ~items)))

(def x (compile-eval
         :x (constantly 1)
         :y (constantly 2)
         :combined (fn [{:keys [x y]}] (+ x y))))

(println x)

JAtkins04:01:05

I think this is working ok. At least for my use case.

JAtkins04:01:57

Hard to get working on cljs, since constantly has a different symbol and doesn't cross the divide safely.

seancorfield05:01:28

Yeah, eval... 🙂

hiredman05:01:56

Or just, I dunno, def one big map and don't do all the merges?

George Ciobanu05:01:01

Hello enlightened souls! Save me please from my Dart prison. I have to build a bunch of apps using Flutter and I'd really want to use Clojure. Anyone know of a transpiler to Dart? Interested in writing one? I'm definitely happy to put some of my own money toward it.

George Ciobanu05:01:17

Another question: any book you'd recommend for general good practices coming from an OO background? For example if not inheritance and encapsulation then what model should I use for naturally hierarchical entities? Think UI library with Buttons, Labels etc. Or any good examples of software I should look into that has source code available?

George Ciobanu17:01:25

thank you so much, this is exactly what I need (I think)(

seancorfield05:01:57

But a lot of how you think in OO isn't applicable in FP -- Clojure in particular favors plain, open data (often, hash maps), over closed, encapsulated data that is "typed"

George Ciobanu17:01:22

I'm familiar with that, but when I go into the stuff I need to do, specifically UI, it's hard not to use inheritance - nested maps are great but I can't tell if I'm doing a good thing by dropping the type guarantees that static oop inheritance brings (to be clear, I LOVE Clojure and am down with whatever path it takes me to)

Cora05:01:58

as far as books go I loved Getting Clojure. and there are clojurescript front-end frameworks you could look to for handling UI

👍 1
George Ciobanu17:01:00

thank you so much, checking it out

seancorfield05:01:13

Hierarchical entities are just nested hash maps. But UI stuff often still tends to be a bit OO because of the mutable state. Swing (use Seesaw) or JavaFX (use cljfx I think?).

seancorfield05:01:34

Getting Clojure is a good intro book, yes. As is Living Clojure.

Ivan Koz05:01:13

@geo.ciobanu why flutter and not say, react?

George Ciobanu17:01:59

Hi Ivan. Flutter because of multiple reasons, some of them I cannot share. Mostly has to do with performance and some legacy decisions

seancorfield05:01:21

React Native (and ClojureScript) for native mobile apps is a reasonable choice. I was only really addressing backend concerns.

Ivan Koz05:01:08

correct me if i'm wrong, but CLJS, specifically redux, makes react faster because of optimized dom updates based on immutable data structures and cheap dirty-checking?

seancorfield05:01:40

I wouldn't say that was specific to Redux but, yeah, the immutable data structures mean that React-style virtual DOM updates can be figured out faster.

Ivan Koz05:01:53

I mean we have flutter vs react performance comparisons, which have nothing to do with cljs, i can see how some new comer making his judgment based on such articles.

seancorfield05:01:58

I haven't seen Flutter vs React performance comparisons, but the cljs performance benefits come from immutability.

George Ciobanu17:01:03

Thank you so much for your reply! The main perf issue with React Native has to do with crossing the JS bridge. It doesn't matter how fast the JS code is, you are limited by the jump between JS world and native. So while Flutter is not as fast as native it is generally more consistent. Also in my case I need the ability to display the same thing on mobile (natively) and on the web, which means RN isn;t great (I know there's RN for the web, but it brings way too many layers)

George Ciobanu03:01:15

More specifically, it's not performance but other factors that make me have to use Flutter.

jumpnbrownweasel03:01:54

Since Dart is statically typed, how could it be generated from Clojure (which is not)?

George Ciobanu04:01:07

Or you can require hints

George Ciobanu04:01:14

Annotations etc

jumpnbrownweasel04:01:36

That's very ambitious.

George Ciobanu04:01:19

There's a similar project Clojure to ObjC

George Ciobanu04:01:32

So lots of potential reuse

seancorfield04:01:35

@geo.ciobanu You probably don't need to send all of those responses to the channel as well -- sort of defeats the point of a thread 🙂

George Ciobanu04:01:46

My apologies, I thought I was helping by sharing the info. But you made me realize that if people are interested they can just open the thread

seancorfield05:01:23

The issue is about how fast it is to compare subtrees of the virtual DOM. If you have an immutable DOM, identity comparisons are valid. If you have a mutable DOM, you have to use deep equality which is much slower.

seancorfield05:01:11

JS doesn't have immutability, although it has "immutable" libraries where you need to follow a bunch of conventions. ClojureScript has that immutability out of the box.

Ivan Koz06:01:47

do we have something like that in clj?

(fn [f pred val]
  (if (pred val)
    (f val)
    val))

seancorfield06:01:12

The closest is (cond-> val (pred val) (f))

seancorfield06:01:36

We have a variant at work (open sourced on GitHub): (condp-> val (pred) (f))

👍 1
Ivan Koz07:01:04

what do you think about latest move not to clever?

Ivan Koz07:01:29

@seancorfield i would appreciate if you take a look aswell, it's a classic iterative hanoi-tower algo as lazy seq

Lokanadham07:01:39

How can i write serialized java objects into a file and read them back into java objects in clojure? any serialization library for this?

Ivan Koz08:01:54

@lokanadham100 you mean java to json and back?

Lokanadham09:01:15

@nxtk I imported a java library into clojure. Now its returning me a java object. I want to store this object into file and read it back when necessary. For serializing if i use yaml/json its converting to string and writing to file is working fine. But while reading back and deserializing its throwing back java.classnotfoundException

Lokanadham09:01:41

So, how will i deserialize the java object back?

Ivan Koz08:01:04

or java serialization binary?

ido10:01:14

hey, when implementing a Java interface with overloading methods on a Clojure defrecord or via reify I need to call the a method from its overloaded implementation. is that possible?

(import '[com.github.benmanes.caffeine.cache.stats StatsCounter CacheStats])

(defrecord CacheMetricsNew 
  [hit-count miss-count load-success-count load-failure-count total-load-time eviction-count eviction-weight]
  StatsCounter
  (recordEviction [this]
    (recordEviction this 1)) ;; this is where the compiler yells
  (recordEviction [_this weight]
    (inc! eviction-count)
    (inc! eviction-weight 1))
  (recordHits [_this hits]
    (inc! hit-count hits))
  (recordLoadFailure [_this load-time]
    (inc! load-failure-count) 
    (update! total-load-time load-time TimeUnit/NANOSECONDS))
  (recordLoadSuccess [_this load-time]
    (inc! load-success-count) 
    (update! total-load-time load-time TimeUnit/NANOSECONDS))
  (recordMisses [_this hits]
    (inc! miss-count hits))
  (snapshot [_this]
    (CacheStats. hit-count miss-count load-success-count load-failure-count total-load-time eviction-count eviction-weight)))

alexmiller12:01:47

I don't think you can do that, but you can probably use a Java interop call there instead (.recordEviction ^StatsCounter this 1)

alexmiller12:01:07

or you could copy the code too of course, or you could use a secondary function and call to it from both

kosengan12:01:35

https://www.infoq.com/news/2019/07/rust-elixir-performance-at-scale/ Can this happen with Clojure? 'Interfacing Clojure with Rust'

alexmiller12:01:29

Clojure can use the JVM's support for native calls, just like Java (via JNA or JNI)

joost-diepenmaat12:01:10

is there any way to retrieve the name of an anonymous function object, as in

(fn foobar [_} ...)

joost-diepenmaat12:01:00

I can see the name as part of the printed object

alexmiller12:01:59

no, other than by demunging the class name

joost-diepenmaat12:01:11

closest I can get is

(clojure.repl/demunge (.getName (class (fn [] ...)))

Eamonn Sullivan12:01:32

Hi all, probably an embarrassing newbie question: I'm getting a baffling message when trying to use clojure.java-time: No single method: start of interface: java_time.interval.AnyInterval found for function: start of protocol: AnyInterval Do I need to construct something somewhere, maybe? The stack trace doesn't even get to my code, so I'm not sure the below is helpful, but here it is (assuming [java-time :as jt]: (defn parse-datetime [s] (when (string? s) (try (jt/offset-date-time s) (catch Throwable _ nil)))) (defn serialize-datetime [d] (when (jt/offset-date-time? d) (try (str (jt/offset-date-time d)) (catch Throwable _ nil))))

Lennart Buit12:01:29

Can you offer a sample of a date time you are trying to convert

Lennart Buit12:01:44

e.g. 2020-01-09T00:00:00+0100

Lennart Buit13:01:40

but that is working right

Eamonn Sullivan13:01:58

Yes, it all works fine in the repl. But when I try to (start) the service, I get this error.

Eamonn Sullivan13:01:45

I just can't make sense of the stack trace because it comes no where near my code, so not sure where to start.

Lennart Buit13:01:25

It looks very much like my custom same datetime scalar

Lennart Buit13:01:39

so I am leaning to ‘it gets a strange argument’

Eamonn Sullivan13:01:01

Ah, what are the arguments that scalar transformers are given? Is it just the value itself or are there other args?

Eamonn Sullivan13:01:08

like with resolvers.

Lennart Buit13:01:39

no its just the ‘raw’ value

Lennart Buit13:01:01

so whatever your scalar serialized to on the other side

Eamonn Sullivan13:01:06

I'm using a placeholder map as a stand-in for the data, and I probably haven't provided anything. Let me look in that direction. Thanks for taking a look and confirming that the scalar part of it looks reasonable.

Lennart Buit13:01:51

no worries — let me know if I can help more

akiel14:01:57

Hi. I like to switch to records for some of my maps. In that records, keys are still mostly optional. So fields can be nil. I still have specs for my maps. In that keys specs, I mark optional keys as optional (as usual). The keys spec op tests for keys with contains?. Now my problem: Records still “contain” nil keys. The value is just nil. So my spec’s don’t work anymore. Instead I have to wrap all optional keys in s/nilable. I think about an option for defrecord to change valAt to return else for nil fields. Anyone had this problem before? Are I miss something here? Related problem: chesire outputs nil values. Third, using __extmap for all optional keys would negate the performance improvements, I see from using records.

alexmiller14:01:53

if you can have records with nil fields, then yes, you'll need to modify your specs to be nilable

alexmiller14:01:38

alternately, you could s/conform the record to a plain map and strip nil-valued fields before validating the spec

akiel14:01:28

Yes using conform to change the record before validating is an idea worth exploring. Should have less friction as the nilable stuff. But I still like the idea to have records with “optional” fields which don’t count into contains?. What do you think? For maps it’s good practise to not include nil kvs. In records it’s not possible to do the same.

akiel15:01:17

@alexmiller I went the conformer route - thanks!

orestis16:01:47

Any success stories on doing Natural Language Processing with Clojure? Looking for categorization/topic modelling/similarity/clustering of hundreds of documents. I’m very new to the field of NLP.

littleli16:01:21

I remember Carin Meier was engaged in NLP/Clojure.

Joe Lane17:01:29

Not sure how open to cloud providers you are, but it may also be valuable to at least investigate using something like aws-api to call out to a service like https://aws.amazon.com/comprehend/features/. Especially if you are getting started but need to ship. Alternatively, a quick search yields several libraries for topic modeling. Also, remember, you have access to all java libraries too, so you could leverage them as well. • opennlp is a bit java apache project which has some of what you are looking for (https://github.com/dakrone/clojure-opennlp is a wrapper) • https://stanfordnlp.github.io/CoreNLP/ but be conscious of licensing • Use https://github.com/cnuernber/libpython-clj to call out to existing python ecosystem of your choice. example, https://github.com/cnuernber/facial-rec You may find different libraries that do some but not all of what you mentioned above. Have fun!

Joe Lane17:01:31

Also, check out https://mxnet.apache.org/ which I believe generates models that aws sagemaker can ingest, and you can invoke. I don't think invoking those models are language specific in any way, and you can write them in clojure if you choose (I know Carin Meier was pushing this forward). Then you can invoke your model via aws-api , for example, or via a lambda, etc.

orestis17:01:45

Thanks, AWS is on my list but I’d rather train my own model. They claim to take a subset of your data to further train your model, and you have to open a support request to delete. Also - you have to upload data in S3, then the results are written back to that - not exactly REPL friendly for experimentation!

Joe Lane17:01:13

I believe you can train the model yourself with mxnet, then deploy the model to sagemaker. Then you have the control.

orestis17:01:36

Yes, that is what I gathered too. I haven’t looked closely at MXNet yet, is it suitable for NLP too?

sogaiu18:01:41

there is a NLP stream(?) under(?) #data-science at zulipchat. may be that's worth checking out?

devn17:01:56

Two questions: 1. Does anyone remember that site that had comparisons between lisps in code? It had CL, emacs lisp, and Clojure IIRC. It was something like http://hypermetaobject.com or something. 2. Does anyone know where that ruby->clojure cheat sheet is? It’s out of date I’m sure, but I’m helping someone up their ruby game enough to be effective (don’t ask) and it had a table of clojure fn -> ruby enumerable method.

devn17:01:36

and I got the second. googling is hard 😉 https://gist.github.com/gus/245786

sogaiu18:01:17

i have been finding in recent years that stuff is harder than it used to be. possibly partly a result of seo?

seancorfield17:01:31

Bing found the first one as the third regular result for comparison common lisp emacs lisp clojure (in case Google isn't finding it based on a similar phrase). Finding the Gist is definitely harder 🙂

vemv18:01:14

Possibly an incorrect memory from my side, but didn't the clj compiler have some sort of flag to trace what it's doing? So one can debug the cases when it gets stuck (e.g. a top-level (while 1))

vemv19:01:05

...thanks to this channel I already know alternative approaches, e.g. kill -3 or Thread/dumpStack. Specifically curious about this supposed flag

bfabry19:01:04

also ctrl+\

noisesmith19:01:13

Ctrl-\, is equivalent to the kill -3 and easier, and the jstack command can do it by PID

noisesmith19:01:39

(and unlike kill, prints to the current termiinal iirc)

noisesmith19:01:05

I'm looking for a verbose arg to compile or one of its implementation details, not finding it so far

noisesmith19:01:38

there's *loading-verbosely* but that seems to just to make it print the name of each namespace as the compiler loads it

👍 1
alexmiller19:01:08

no such flag that I know of

👍 1
noisesmith19:01:36

haha if @alexmiller hasn't heard of it it likely doesn't exist

noisesmith19:01:37

the only invocations of getCompilerOption in Compiler.java are to get elideMetaKey and directLinkingKey and disableLocalsClearingKey - all options I've heard of

noisesmith19:01:58

and github search seems to indicate that's the only file where it is used

vemv19:01:02

thanks folks!

oliver20:01:50

Hi, I need an associative data structure with defined order… As of now I use nested vectors like (def vec [[:a "aa"] [:b "bb"]]) and then do the lookup like so (:a (into {} vec)). Is there any better/received way of doing this? Performance is secondary since the structures involved are small (< 20 elements).

lispyclouds20:01:50

works with all the standard functions

oliver20:01:37

Thanks! That looks useful… still I'd be interested in a solution without libs.

oliver20:01:55

Actually I haven't had any trouble using standard maps since the defined ordered doen't seem to change… it's just that I read one shouldn't rely on that. What I'm trying to do is define the Menu for a Website… it's not supposed to undergo any transformations at runtime… pretty much just lookup

oliver20:01:16

…and one mapping to format the menu, hence the need for order

lispyclouds20:01:34

If you're running on the JVM, you can use the LinkedHashMap? Would work without libs

oliver20:01:25

I'm using Clojurescript on Node.js actually… thought it wouldn't matter for this question, which is why I chose to ask here

lispyclouds20:01:18

well, no js expert here, but apparently since ES 2015, js Object keys are ordered by insertion. Since you dont have concurrent write issues, can try that?

enforser20:01:51

You could potentially do JS interop and use a Map object, https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map You could also use a normal hash-map for lookups by key, and keep another list/vector of the keys elsewhere when you need the actual order. This may work: http://clojuredocs.org/clojure.core/sorted-map-by

2
oliver20:01:03

Thanks, that' an interesting idea… I'll look into that!

bortex20:01:19

If you have somehow a way to compare the keys and know what goes before what, you could use a sorted-map which is standard clojure

oliver20:01:34

That would probably work, because the order is defined only once and is not supposed to change at runtime

bortex20:01:22

Items in a sorted-map are sorted by comparison, so if you have keys "a", "c", "b" , they will be “a, b, c” in the sorted map. You can specify the comparator to use. For something that keeps the order of inserting items into the map, instead of sorting by comparing items, maybe https://github.com/frankiesardo/linked or the already mentioned https://github.com/clj-commons/ordered

andy.fingerhut20:01:53

You can define your own custom comparator function, if that helps keep things in the order that you want.

👍 2
andy.fingerhut21:01:07

i.e. your own custom comparator function that you supply when you create a sorted map, using sorted-map-by

enforser21:01:27

I feel like since you're defining a website menu you likely won't notice any impact from going with easier to reason about but slightly less fast structures. I'd probably just do something like the following instead of introducing the sorted-map complexity or an external library:

(def menu-pages
  {:page1 "a"
   :page2 "b"
   :page3 "c"})

(def ordered-menu
  (map menu-pages [:page2 :page3 :page1]))

;; OR 

(def ordered-pages
  [{:name "page1"
    :href "foo"
    ...}
   {:name "page2"
    :href "bar"
    ...}])

👍 2
oliver22:01:47

Thanks, the second version is basically what I've been using so far. Now I want certain routehandlers to look up the names and use them as titles for pages in an effort to reduce redundancy... hence why I initially asked about looking up in vectors... However, I'm starting to think bigger now and I want to handle nested menus as well a composing them dynamically... I have to give that some more thought... Anyway, thanks to all of you for all the input!

λustin f(n)21:01:08

Hi. I have some existing tests where there are one or more (future (update-database data)) -type calls going on. Currently, we are doing things like (Thread/sleep 100) to give enough time for the futures to complete before checking they affected things as expected. However, this has unwanted effects like non-deterministic tests that fail randomly if the sleep is too short, and having to always wait that amount of time regardless of how fast it might have completed. Is there anything out of the box to help me do this better? Bar that, I am about to bite the bullet to learn macros, just so I can make something using with-redefs promise and timeouts that can block until the db functions are signaled to complete, or the operation times out.

λustin f(n)21:01:20

The future calls are usually a couple levels down, so I don't have access to their references.

enforser21:01:51

I've never done this, but I think you could potentially use a claypoole future which allows you to disable threading in your tests via a binding. https://github.com/TheClimateCorporation/claypoole#how-can-i-disable-threading

enforser21:01:25

and you can make the futures work identically to the built in core future by passing the :built-in keyword to them.

lispyclouds21:01:44

@U7Y7601B2 How are you continuing the computation in the code after the db calls? Thats where I'd put the tests. Like callbacks. Or is this a fire-and-forget?

λustin f(n)21:01:06

@U7ERLH6JX It is currently fire-and-forget, side-effecty stuff. There are a few places where things like this happen, so I was hoping to find something to patch over the issue for testing, rather than redesigning a bunch of things.

λustin f(n)21:01:19

@UCQL6E7PY that library looks nice, maybe I will just use that to make everything synchronous for the tests.

lispyclouds21:01:20

right, feels like an anti-pattern like @noisesmith mentioned on the other thread. maybe this one time refactoring goes a long way? 🙂

noisesmith21:01:45

@U7Y7601B2 btw futures are a bad way to do "fire and forget" - they silently throw away exceptions and you never see the error

λustin f(n)21:01:46

I will probably end up doing something like that eventually once I find a good model to move towards.

noisesmith21:01:12

user=> (def f (future (/ 1 0)))
#'user/f

noisesmith21:01:26

(not shown - total lack of any signal that an error occured)

λustin f(n)21:01:05

We have a future-try macro that just puts the body in a try block and logs the results ATM.

λustin f(n)21:01:29

What are the better ways to do this?

noisesmith21:01:51

a dedicated pool of workers (so you have a defined capacity / backpressure) and a queue of some sort - built in queue based options include ExecutorService via interop (can take a clojure function arg), core.async channels, and agents

noisesmith21:01:04

futures work great in proof of concept code but once workload increases and/ or logic gets more complex the gotchas start to pile up

noisesmith21:01:25

there's also pooled future options in the claypoole lib mentioned above

lispyclouds21:01:51

also, if you want to chain your side effects and catch the error(s) in one place you could try a lib like Failjure and its attempt-all/try-all functions. https://github.com/adambard/failjure#attempt-all https://github.com/adambard/failjure#try-all

lispyclouds21:01:11

and something like https://github.com/ztellman/manifold/blob/master/docs/deferred.md#let-flow allows concurrent let arms but wait on all of them in a clean way

λustin f(n)22:01:24

Cool, thanks for all the advice. I will need to take the time to get familiar with some of these tools, and to find some time to do the refactor to make a cleaner effects layer. For now, the workaround I am going with that looks like it works is to to make all futures become blocking using this:

(with-redefs [future-call (comp #(deref % 100 :fail) future-call)]

noisesmith22:01:38

as @alexmiller mentions with-redefs is very broken when combined with threaded code - what happens is that with-redefs creates a stack of old def / new def / old def

noisesmith22:01:21

if two with-redefs overlap, you can end up with a replacing f with g, b replacing g with h, a replacing h with f, and b replacing f with g - now f is lost

noisesmith22:01:30

tl;dr it's a race condition

λustin f(n)22:01:47

So if the with-redefs is inside competing threads, it is a problem?

noisesmith22:01:24

right - the only safe way to use it is in a single thread

λustin f(n)22:01:09

We use with-redefs only in testing, and only use out test suite syncronously, since much of the application already has shared state like db

λustin f(n)22:01:29

So I expect we should manage not shooting ourselves in the foot this time

codonnell23:01:03

We have some integration tests at work where we cannot (and do not want to) stub out asynchronous calls. I wrote a helper that automatically retries a predicate a number of times and returns a good error message on failure:

;; NOTE: This is modeled off `test/assert-predicate` () so that error messages on test failure are descriptive
(defmethod test/assert-expr 'with-retries [msg [_ pred-form {:keys [num-retries pause-ms]
                                                             :or {num-retries 10
                                                                  pause-ms 100}}]]
  (let [pred (first pred-form)
        args (rest pred-form)]
    `(reduce (fn [_# i#]
               (let [arg-values# (list [email protected])
                     result# (apply ~pred arg-values#)]
                 (if (or result# (= i# ~num-retries))
                   (do (test/do-report
                        {:type (if result# :pass :fail)
                         :message ~msg
                         :expected '~pred-form
                         :actual (cond->> (cons '~pred arg-values#)
                                   (not result#)
                                   (cons '~'not))})
                       (reduced result#))
                   (Thread/sleep ~pause-ms))))
             (range))))
which can be used like:
(deftest test-async
  (do-asynchronous-thing)
  (is (with-retries (verify-async-process-completed-successfully))))
Of course, this incurs a running time penalty on your tests that can add up quickly, so I'd only recommend using it if you cannot refactor your code in a way that you can guarantee when your async process is done (like let-flow).

alexmiller21:01:22

isn't just promise enough?

2
λustin f(n)21:01:32

I want to avoid changing the functions as they exist, and the future calls are usually a few levels down from the high-level behavior the tests are supposed to be for.

λustin f(n)21:01:47

There are places in the codebase where my coworkers have tried with-redefs on the future macro itself to try to make it synchronous, but that doesn't actually seem to work.

λustin f(n)21:01:06

My plan was to make something that uses with-redefs on the actual database functions in the body of the future so I can wire up some promises I can block on when the function calls complete. So I can kinda make this async code synchronous again for the test.

noisesmith21:01:43

nesting side effects deep under abstractions is an antipattern - there are a few good techniques for lifting the side effecting code and making it more directly observable (eg. a worker reading a queue)

1
noisesmith21:01:31

to make a future "sychronous", just deref it

noisesmith21:01:42

also, if you are calling future and never using deref to access the result, errors that should be throwing and being caught are instead preserved (waiting on a deref before re-throwing)

noisesmith21:01:48

so you get weird silent failures

λustin f(n)21:01:36

So... if I really want to get around this I will probably need to refactor all the way to the point of something like having an 'effects' layer that actually processes thing with a queue and workers, etc. Then having all the domain logic eventually put things into the queue, where I can check it in tests, etc?

noisesmith21:01:02

that tends to be the principled approach, yes

alexmiller21:01:02

with-redefs is not a reliable thing to do with multi-threaded code so probably taking on a different set of pain

1
colinkahn22:01:44

Is there such a thing as an in memory writer? I want to use the writer interface but not actually write anything to disk

bfabry22:01:10

StringBuffer, ByteBuffer

noisesmith22:01:41

or File of "/dev/null" on posix systems

colinkahn22:01:17

@noisesmith can I then read back what was written to the writer that way?

noisesmith22:01:32

no - I didn't realize that was a requirement

noisesmith22:01:08

btw the javadoc is great for questions like this for future reference - Writer has links to built in implementations, including the ones mentioned above by @U050MP39D https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/io/Writer.html

bfabry22:01:28

there's also (with-out-str in core which is great

colinkahn22:01:48

I’m trying to do:

(with-open [writer (SomeWriter.)]
  (some-thing-that-writes writer)
  ;; read contents here
)

noisesmith22:01:27

and clojure has

user=> (javadoc .Writer)
true
which works for both classes and instances, and opens directly in your web browser

noisesmith22:01:02

very handy when you have a random class in the repl, and want to read its docs

noisesmith22:01:51

user=> (with-open [writer (.StringWriter.)] (spit writer "hi!") (spit writer "bye!") (str writer))
"hi!bye!"

colinkahn22:01:53

Thanks, I forget to peek into the java world when I need stuff

colinkahn23:01:30

Nice! Thanks again

andy.fingerhut23:01:51

I think it would be more common for the ;; read contents here part of the code to be outside of the with-open after writing was complete and the "buffer was closed", but not sure whether it is important for your use case to read it, then conditionally write more. If so, that is a bit fancier of an interface in Java IO libs.

noisesmith23:01:59

yeah, I was definitely taking advantage of the flexibility and convenience of the StringWriter implementation to make a concise example there

noisesmith23:01:13

and other writers need flushing etc.

colinkahn23:01:06

@U0CMVHBL2 good to know, for this case i’m just going to be writing to it once