This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-01-09
Channels
- # announcements (28)
- # babashka (8)
- # bangalore-clj (1)
- # beginners (123)
- # boot (1)
- # bristol-clojurians (1)
- # calva (3)
- # cider (30)
- # clj-kondo (42)
- # cljs-dev (5)
- # clojure (260)
- # clojure-dev (11)
- # clojure-europe (7)
- # clojure-india (1)
- # clojure-italy (2)
- # clojure-losangeles (5)
- # clojure-nl (5)
- # clojure-portugal (15)
- # clojure-uk (51)
- # clojurescript (69)
- # cursive (6)
- # data-science (21)
- # datascript (17)
- # datomic (1)
- # emacs (29)
- # figwheel-main (11)
- # fulcro (89)
- # graphql (5)
- # hoplon (2)
- # hugsql (6)
- # jobs (11)
- # juxt (1)
- # leiningen (7)
- # luminus (1)
- # malli (3)
- # off-topic (64)
- # pathom (32)
- # project-updates (1)
- # re-frame (9)
- # reagent (10)
- # reitit (21)
- # ring (5)
- # ring-swagger (1)
- # shadow-cljs (8)
- # spacemacs (6)
- # xtdb (4)
for functions that are mutually recursive and require closed-over data
I wonder if you could replace your nested conditionals with one cond - eg. (let [idx (mod n 3) (cond (and odd-n? (= idx 0)) ... (and odd-n? (= idx 1)) ... ... (and (not odd-n?) (= idx 0)) ...)
just a thought - not sure that's an improvement
well, i'm also looking for a way to make it less verbose but not too clever\\complex
or even (case [odd-n? (mod n 3)] [true 0] .... [true 1] ... ... [false 0] ...)
right, that's an important balance
also you don't need odd-n? to be a captured value, it could be calculated in the let block of move
I wouldn't even think about "optimization" there, yeah - it's more a question of "does this make the code more straightforward, or not"
by which I mean, you can rewrite it as single step, getting rid of building a seq and recursing, and use iterate to do that
(take-while some-pred? (iterate ....))
another thing to consider - instead of [from to spare]
it could internally use {:keys [from to spare]}
the advantage being that the new state can be something like (-> state (update :from pop) (update :to conj (peek from)))
(spare doesn't change in this case, no need to mention it)
I think that (plus some line breaks) make sit more clear what operation is actually being done
of course the printed form should still be a vector, that's much better than a fully named hash map
great, thank you so much for the valuable feedback, i'll get back to you as i'm done with above improvements
hope it helps - these are all pretty subjective
Is there any difference (from the point of lazy evaluation) between lazy-cat
lazy-seq + cons
or just cons
if i need to concatenate a value and a lazy sequence?
i mean lazy-.. stuff will wrap first argument into LazySeq and using it in case of already existing first value has no sense right?
cons
evaluates both its arguments whereas lazy-seq
defers evaluation until the sequence is realized
@qythium yeah i know that cons evaluates its argument, but my case is (cons 0 (lazy-seq ...
since 0 is already there, using lazy-
stuff instead of cons
has no meaningful reason right?
(def v1 {:key :val})
(def v2 {:key2 :val2})
(def v3 (merge v1 v2))
so that the program startup doesn't get hung up with some small calculations?Not quite. I'd like to have the compiler replace the merge with {:key :val :key2 :val2}
by running the code like macros are
I have dozens of this kind of chained def running at the start of my cljs app, and it's starting to take a hit
@jatkin you can also ask in #clojurescript, i never touched cljs so can't help you with compiletime
well on jvm compilation and evaluation are mixed, so technically there is no difference, it's just that macros are expanded before evaluation all happens on runtime
thats why you should ask that question at #clojurescript most likely there is someone to help you who knows cljs specifics of macroexpansion.
In Clojure, those chained def
s will be evaluated when the namespace is loaded -- so in that respect it's going to be the same for both Clojure and ClojureScript @jatkin
So if you do that a lot in Clojure, you will take a hit at startup, just like in cljs. My question would be: what problem are you solving to arrive at your current solution? (perhaps there are more performant alternatives?)
I'm using code to save myself a bunch of typing for come config stuff. I have a tool with about 50 config parameters in one part, each one applicable for different items that may appear on the page. To keep it minimal I have code running in this way to dedupe alot of the config possibilities for the ~30 types of options.
Is this just fixed data, computed just once at startup time?
How about defining it in a separate process that you run to compute the full set of data that writes it to a cljs file in fully-expanded form, as a preprocessing step before building you cljs app to js?
@seancorfield i think in that case it would be fixable by a macro
All fixed. Possibly. I was about to write a thunk macro that forces the code in question to run in the compiler, maybe building up a map. If I actually needed the intermediate values I can have it write into a map and assign each def to one of the keys of the map
@nxtk Do you want to have a stab at that macro? I think it's harder than it looks because it relies on combining runtime values -- it's not just compile-time stuff...
oh i'm far from cljs, i only know iterop is different, and it translates to js by google closure
@nxtk Even in pure Clojure...
@seancorfield whats hard about it? I probably doesn't see what are you talking about.
(def x 5)
(def y 3)
(defmacro m-sum
[a b]
`(def z (+ ~a ~b)))
(m-sum x y)
(println z)
(println (str (macroexpand '(m-sum x y))))
i mean lisp macros are just a functions which take forms as input and produce forms as output, so i'm not sure how you look at it in that case
if your data is strictly a valid return for a macro then you can do something like this
The reason it is tricky is you have to manually do the resolving of symbols to values (at macroexpand time) and them call whatever function on the values. And it is even trickier because the macro language for cljs is clj not cljs
(defmacro compile-eval [& body]
(eval body))
(def [a b c] (compile-eval (let [x 3 y 5] [x y (+ x y)]))
It's one of those "simple, obvious" problems that you tend to think just ought to be straightforward with a macro... but the compile-time / run-time divide is hard to cross...
(defn run-chain [items]
(reduce
(fn [acc [k f]]
(assoc acc k (f acc)))
{}
(partition 2 items)))
(defmacro compile-eval [& items]
(eval `(run-chain ~items)))
(def x (compile-eval
:x (constantly 1)
:y (constantly 2)
:combined (fn [{:keys [x y]}] (+ x y))))
(println x)
Hard to get working on cljs, since constantly has a different symbol and doesn't cross the divide safely.
Yeah, eval
... 🙂
Hello enlightened souls! Save me please from my Dart prison. I have to build a bunch of apps using Flutter and I'd really want to use Clojure. Anyone know of a transpiler to Dart? Interested in writing one? I'm definitely happy to put some of my own money toward it.
Another question: any book you'd recommend for general good practices coming from an OO background? For example if not inheritance and encapsulation then what model should I use for naturally hierarchical entities? Think UI library with Buttons, Labels etc. Or any good examples of software I should look into that has source code available?
You might find https://leanpub.com/fp-oo helpful @geo.ciobanu
thank you so much, this is exactly what I need (I think)(
But a lot of how you think in OO isn't applicable in FP -- Clojure in particular favors plain, open data (often, hash maps), over closed, encapsulated data that is "typed"
I'm familiar with that, but when I go into the stuff I need to do, specifically UI, it's hard not to use inheritance - nested maps are great but I can't tell if I'm doing a good thing by dropping the type guarantees that static oop inheritance brings (to be clear, I LOVE Clojure and am down with whatever path it takes me to)
as far as books go I loved Getting Clojure. and there are clojurescript front-end frameworks you could look to for handling UI
thank you so much, checking it out
Hierarchical entities are just nested hash maps. But UI stuff often still tends to be a bit OO because of the mutable state. Swing (use Seesaw) or JavaFX (use cljfx I think?).
Getting Clojure is a good intro book, yes. As is Living Clojure.
@geo.ciobanu why flutter and not say, react?
Hi Ivan. Flutter because of multiple reasons, some of them I cannot share. Mostly has to do with performance and some legacy decisions
React Native (and ClojureScript) for native mobile apps is a reasonable choice. I was only really addressing backend concerns.
correct me if i'm wrong, but CLJS, specifically redux, makes react faster because of optimized dom updates based on immutable data structures and cheap dirty-checking?
I wouldn't say that was specific to Redux but, yeah, the immutable data structures mean that React-style virtual DOM updates can be figured out faster.
I mean we have flutter vs react performance comparisons, which have nothing to do with cljs, i can see how some new comer making his judgment based on such articles.
I haven't seen Flutter vs React performance comparisons, but the cljs performance benefits come from immutability.
Thank you so much for your reply! The main perf issue with React Native has to do with crossing the JS bridge. It doesn't matter how fast the JS code is, you are limited by the jump between JS world and native. So while Flutter is not as fast as native it is generally more consistent. Also in my case I need the ability to display the same thing on mobile (natively) and on the web, which means RN isn;t great (I know there's RN for the web, but it brings way too many layers)
More specifically, it's not performance but other factors that make me have to use Flutter.
Since Dart is statically typed, how could it be generated from Clojure (which is not)?
Type inference
Or you can require hints
Annotations etc
That's very ambitious.
There's a similar project Clojure to ObjC
So lots of potential reuse
Interesting
@geo.ciobanu You probably don't need to send all of those responses to the channel as well -- sort of defeats the point of a thread 🙂
My apologies, I thought I was helping by sharing the info. But you made me realize that if people are interested they can just open the thread
So thank you @seancorfield
The issue is about how fast it is to compare subtrees of the virtual DOM. If you have an immutable DOM, identity comparisons are valid. If you have a mutable DOM, you have to use deep equality which is much slower.
JS doesn't have immutability, although it has "immutable" libraries where you need to follow a bunch of conventions. ClojureScript has that immutability out of the box.
do we have something like that in clj?
(fn [f pred val]
(if (pred val)
(f val)
val))
The closest is (cond-> val (pred val) (f))
https://github.com/worldsingles/commons/blob/master/src/ws/clojure/extensions.clj#L19-L30 if you want to depend on it as a library.
@noisesmith @hiredman related to hanoi-tower, made few variants of move function old https://gist.github.com/nxtk/7c646697240fff2833d104211c8f36c1 new https://gist.github.com/nxtk/c4ea2267c3e865bcbe3d1e32371a38c1
@seancorfield i would appreciate if you take a look aswell, it's a classic iterative hanoi-tower algo as lazy seq
How can i write serialized java objects into a file and read them back into java objects in clojure? any serialization library for this?
@lokanadham100 you mean java to json and back?
@nxtk I imported a java library into clojure. Now its returning me a java object. I want to store this object into file and read it back when necessary. For serializing if i use yaml/json its converting to string and writing to file is working fine. But while reading back and deserializing its throwing back java.classnotfoundException
So, how will i deserialize the java object back?
There's https://github.com/sylvainhalle/Azrael and https://github.com/google/gson and http://genson.io/
hey, when implementing a Java interface with overloading methods on a Clojure defrecord
or via reify
I need to call the a method from its overloaded implementation. is that possible?
(import '[com.github.benmanes.caffeine.cache.stats StatsCounter CacheStats])
(defrecord CacheMetricsNew
[hit-count miss-count load-success-count load-failure-count total-load-time eviction-count eviction-weight]
StatsCounter
(recordEviction [this]
(recordEviction this 1)) ;; this is where the compiler yells
(recordEviction [_this weight]
(inc! eviction-count)
(inc! eviction-weight 1))
(recordHits [_this hits]
(inc! hit-count hits))
(recordLoadFailure [_this load-time]
(inc! load-failure-count)
(update! total-load-time load-time TimeUnit/NANOSECONDS))
(recordLoadSuccess [_this load-time]
(inc! load-success-count)
(update! total-load-time load-time TimeUnit/NANOSECONDS))
(recordMisses [_this hits]
(inc! miss-count hits))
(snapshot [_this]
(CacheStats. hit-count miss-count load-success-count load-failure-count total-load-time eviction-count eviction-weight)))
I don't think you can do that, but you can probably use a Java interop call there instead (.recordEviction ^StatsCounter this 1)
or you could copy the code too of course, or you could use a secondary function and call to it from both
https://www.infoq.com/news/2019/07/rust-elixir-performance-at-scale/ Can this happen with Clojure? 'Interfacing Clojure with Rust'
Clojure can use the JVM's support for native calls, just like Java (via JNA or JNI)
is there any way to retrieve the name of an anonymous function object, as in
(fn foobar [_} ...)
I can see the name as part of the printed object
no, other than by demunging the class name
closest I can get is
(clojure.repl/demunge (.getName (class (fn [] ...)))
right. thanks @alexmiller
Hi all, probably an embarrassing newbie question: I'm getting a baffling message when trying to use clojure.java-time:
No single method: start of interface:
java_time.interval.AnyInterval found for function: start of
protocol: AnyInterval
Do I need to construct something somewhere, maybe?
The stack trace doesn't even get to my code, so I'm not sure the below is helpful, but here it is (assuming [java-time :as jt]
:
(defn parse-datetime
[s]
(when (string? s)
(try
(jt/offset-date-time s)
(catch Throwable _
nil))))
(defn serialize-datetime
[d]
(when (jt/offset-date-time? d)
(try
(str (jt/offset-date-time d))
(catch Throwable _
nil))))
Can you offer a sample of a date time you are trying to convert
e.g. 2020-01-09T00:00:00+0100
but that is working right
Yes, it all works fine in the repl. But when I try to (start) the service, I get this error.
I just can't make sense of the stack trace because it comes no where near my code, so not sure where to start.
It looks very much like my custom same datetime scalar
so I am leaning to ‘it gets a strange argument’
Ah, what are the arguments that scalar transformers are given? Is it just the value itself or are there other args?
like with resolvers.
no its just the ‘raw’ value
so whatever your scalar serialized to on the other side
I'm using a placeholder map as a stand-in for the data, and I probably haven't provided anything. Let me look in that direction. Thanks for taking a look and confirming that the scalar part of it looks reasonable.
no worries — let me know if I can help more
@thegobinath there's a talk by lvh that touches on interfacing clojure with c: https://www.youtube.com/watch?v=Lf-M1ZH6KME libpython-clj interfaces with python via c iirc at least one thing from blueberry / dragan uses javacpp (though i'm not having success finding that bit now -- there is mention of org.bytedeco/dnnl-platform in deep-diamond, and iiuc that uses javacpp)
may be - iirc, there are some remarks by chrisn in his conj talk on libpython-clj regarding context / scope / rationale that might be worth checking out.
Hi. I like to switch to records for some of my maps. In that records, keys are still mostly optional. So fields can be nil. I still have specs for my maps. In that keys
specs, I mark optional keys as optional (as usual). The keys spec op tests for keys with contains?
.
Now my problem: Records still “contain” nil keys. The value is just nil. So my spec’s don’t work anymore. Instead I have to wrap all optional keys in s/nilable
. I think about an option for defrecord
to change valAt
to return else
for nil fields. Anyone had this problem before? Are I miss something here?
Related problem: chesire outputs nil values. Third, using __extmap
for all optional keys would negate the performance improvements, I see from using records.
if you can have records with nil fields, then yes, you'll need to modify your specs to be nilable
alternately, you could s/conform the record to a plain map and strip nil-valued fields before validating the spec
Yes using conform to change the record before validating is an idea worth exploring. Should have less friction as the nilable stuff. But I still like the idea to have records with “optional” fields which don’t count into contains?. What do you think? For maps it’s good practise to not include nil kvs. In records it’s not possible to do the same.
@alexmiller I went the conformer route - thanks!
Any success stories on doing Natural Language Processing with Clojure? Looking for categorization/topic modelling/similarity/clustering of hundreds of documents. I’m very new to the field of NLP.
Not sure how open to cloud providers you are, but it may also be valuable to at least investigate using something like aws-api
to call out to a service like https://aws.amazon.com/comprehend/features/.
Especially if you are getting started but need to ship.
Alternatively, a quick search yields several libraries for topic modeling. Also, remember, you have access to all java libraries too, so you could leverage them as well.
• opennlp is a bit java apache project which has some of what you are looking for (https://github.com/dakrone/clojure-opennlp is a wrapper)
• https://stanfordnlp.github.io/CoreNLP/ but be conscious of licensing
• Use https://github.com/cnuernber/libpython-clj to call out to existing python ecosystem of your choice. example, https://github.com/cnuernber/facial-rec
You may find different libraries that do some but not all of what you mentioned above. Have fun!
Also, check out https://mxnet.apache.org/ which I believe generates models that aws sagemaker can ingest, and you can invoke. I don't think invoking those models are language specific in any way, and you can write them in clojure if you choose (I know Carin Meier was pushing this forward).
Then you can invoke your model via aws-api
, for example, or via a lambda, etc.
Thanks, AWS is on my list but I’d rather train my own model. They claim to take a subset of your data to further train your model, and you have to open a support request to delete. Also - you have to upload data in S3, then the results are written back to that - not exactly REPL friendly for experimentation!
I believe you can train the model yourself with mxnet, then deploy the model to sagemaker. Then you have the control.
Yes, that is what I gathered too. I haven’t looked closely at MXNet yet, is it suitable for NLP too?
Yes: • https://medium.com/apache-mxnet/gluonnlp-v0-7-1-bert-reloaded-7b9450d33f4b • https://gluon-nlp.mxnet.io/examples/sentence_embedding/bert.html • https://mxnet.apache.org/api/python/docs/tutorials/packages/gluon/text/index.html My understanding is that BERT is currently state of the art.
there is a NLP stream(?) under(?) #data-science at zulipchat. may be that's worth checking out?
You may be able to utilize https://github.com/huggingface/transformers via https://github.com/cnuernber/libpython-clj
Two questions: 1. Does anyone remember that site that had comparisons between lisps in code? It had CL, emacs lisp, and Clojure IIRC. It was something like http://hypermetaobject.com or something. 2. Does anyone know where that ruby->clojure cheat sheet is? It’s out of date I’m sure, but I’m helping someone up their ruby game enough to be effective (don’t ask) and it had a table of clojure fn -> ruby enumerable method.
I found the first one: http://hyperpolyglot.org/lisp
and I got the second. googling is hard 😉 https://gist.github.com/gus/245786
i have been finding in recent years that stuff is harder than it used to be. possibly partly a result of seo?
Bing found the first one as the third regular result for comparison common lisp emacs lisp clojure
(in case Google isn't finding it based on a similar phrase). Finding the Gist is definitely harder 🙂
Possibly an incorrect memory from my side, but didn't the clj compiler have some sort of flag to trace what it's doing?
So one can debug the cases when it gets stuck (e.g. a top-level (while 1)
)
...thanks to this channel I already know alternative approaches, e.g. kill -3
or Thread/dumpStack
.
Specifically curious about this supposed flag
Ctrl-\,
is equivalent to the kill -3 and easier, and the jstack command can do it by PID
(and unlike kill, prints to the current termiinal iirc)
I'm looking for a verbose arg to compile or one of its implementation details, not finding it so far
there's *loading-verbosely*
but that seems to just to make it print the name of each namespace as the compiler loads it
haha if @alexmiller hasn't heard of it it likely doesn't exist
the only invocations of getCompilerOption in Compiler.java are to get elideMetaKey and directLinkingKey and disableLocalsClearingKey - all options I've heard of
and github search seems to indicate that's the only file where it is used
Hi, I need an associative data structure with defined order… As of now I use nested vectors like (def vec [[:a "aa"] [:b "bb"]])
and then do the lookup like so (:a (into {} vec))
. Is there any better/received way of doing this? Performance is secondary since the structures involved are small (< 20 elements).
works with all the standard functions
Actually I haven't had any trouble using standard maps since the defined ordered doen't seem to change… it's just that I read one shouldn't rely on that. What I'm trying to do is define the Menu for a Website… it's not supposed to undergo any transformations at runtime… pretty much just lookup
If you're running on the JVM, you can use the LinkedHashMap? Would work without libs
I'm using Clojurescript on Node.js actually… thought it wouldn't matter for this question, which is why I chose to ask here
well, no js expert here, but apparently since ES 2015, js Object keys are ordered by insertion. Since you dont have concurrent write issues, can try that?
You could potentially do JS interop and use a Map object, https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map You could also use a normal hash-map for lookups by key, and keep another list/vector of the keys elsewhere when you need the actual order. This may work: http://clojuredocs.org/clojure.core/sorted-map-by
If you have somehow a way to compare the keys and know what goes before what, you could use a sorted-map which is standard clojure
That would probably work, because the order is defined only once and is not supposed to change at runtime
Items in a sorted-map are sorted by comparison, so if you have keys "a", "c", "b"
, they will be “a, b, c” in the sorted map. You can specify the comparator to use. For something that keeps the order of inserting items into the map, instead of sorting by comparing items, maybe https://github.com/frankiesardo/linked or the already mentioned https://github.com/clj-commons/ordered
You can define your own custom comparator function, if that helps keep things in the order that you want.
i.e. your own custom comparator function that you supply when you create a sorted map, using sorted-map-by
I feel like since you're defining a website menu you likely won't notice any impact from going with easier to reason about but slightly less fast structures. I'd probably just do something like the following instead of introducing the sorted-map
complexity or an external library:
(def menu-pages
{:page1 "a"
:page2 "b"
:page3 "c"})
(def ordered-menu
(map menu-pages [:page2 :page3 :page1]))
;; OR
(def ordered-pages
[{:name "page1"
:href "foo"
...}
{:name "page2"
:href "bar"
...}])
Thanks, the second version is basically what I've been using so far. Now I want certain routehandlers to look up the names and use them as titles for pages in an effort to reduce redundancy... hence why I initially asked about looking up in vectors... However, I'm starting to think bigger now and I want to handle nested menus as well a composing them dynamically... I have to give that some more thought... Anyway, thanks to all of you for all the input!
Hi. I have some existing tests where there are one or more (future (update-database data))
-type calls going on. Currently, we are doing things like (Thread/sleep 100)
to give enough time for the futures to complete before checking they affected things as expected.
However, this has unwanted effects like non-deterministic tests that fail randomly if the sleep is too short, and having to always wait that amount of time regardless of how fast it might have completed.
Is there anything out of the box
to help me do this better?
Bar that, I am about to bite the bullet to learn macros, just so I can make something using with-redefs
promise
and timeouts that can block until the db functions are signaled to complete, or the operation times out.
The future calls are usually a couple levels down, so I don't have access to their references.
I've never done this, but I think you could potentially use a claypoole future which allows you to disable threading in your tests via a binding. https://github.com/TheClimateCorporation/claypoole#how-can-i-disable-threading
and you can make the futures work identically to the built in core future by passing the :built-in
keyword to them.
@U7Y7601B2 How are you continuing the computation in the code after the db calls? Thats where I'd put the tests. Like callbacks. Or is this a fire-and-forget?
@U7ERLH6JX It is currently fire-and-forget, side-effecty stuff. There are a few places where things like this happen, so I was hoping to find something to patch over the issue for testing, rather than redesigning a bunch of things.
@UCQL6E7PY that library looks nice, maybe I will just use that to make everything synchronous for the tests.
right, feels like an anti-pattern like @noisesmith mentioned on the other thread. maybe this one time refactoring goes a long way? 🙂
@U7Y7601B2 btw futures are a bad way to do "fire and forget" - they silently throw away exceptions and you never see the error
I will probably end up doing something like that eventually once I find a good model to move towards.
user=> (def f (future (/ 1 0)))
#'user/f
(not shown - total lack of any signal that an error occured)
We have a future-try
macro that just puts the body in a try block and logs the results ATM.
What are the better ways to do this?
a dedicated pool of workers (so you have a defined capacity / backpressure) and a queue of some sort - built in queue based options include ExecutorService via interop (can take a clojure function arg), core.async channels, and agents
futures work great in proof of concept code but once workload increases and/ or logic gets more complex the gotchas start to pile up
there's also pooled future options in the claypoole lib mentioned above
also, if you want to chain your side effects and catch the error(s) in one place you could try a lib like Failjure and its attempt-all/try-all functions. https://github.com/adambard/failjure#attempt-all https://github.com/adambard/failjure#try-all
and something like https://github.com/ztellman/manifold/blob/master/docs/deferred.md#let-flow allows concurrent let arms but wait on all of them in a clean way
Cool, thanks for all the advice. I will need to take the time to get familiar with some of these tools, and to find some time to do the refactor to make a cleaner effects layer. For now, the workaround I am going with that looks like it works is to to make all futures become blocking using this:
(with-redefs [future-call (comp #(deref % 100 :fail) future-call)]
as @alexmiller mentions with-redefs is very broken when combined with threaded code - what happens is that with-redefs creates a stack of old def / new def / old def
if two with-redefs overlap, you can end up with a replacing f with g, b replacing g with h, a replacing h with f, and b replacing f with g - now f is lost
tl;dr it's a race condition
So if the with-redefs is inside competing threads, it is a problem?
right - the only safe way to use it is in a single thread
We use with-redefs only in testing, and only use out test suite syncronously, since much of the application already has shared state like db
So I expect we should manage not shooting ourselves in the foot this time
We have some integration tests at work where we cannot (and do not want to) stub out asynchronous calls. I wrote a helper that automatically retries a predicate a number of times and returns a good error message on failure:
;; NOTE: This is modeled off `test/assert-predicate` () so that error messages on test failure are descriptive
(defmethod test/assert-expr 'with-retries [msg [_ pred-form {:keys [num-retries pause-ms]
:or {num-retries 10
pause-ms 100}}]]
(let [pred (first pred-form)
args (rest pred-form)]
`(reduce (fn [_# i#]
(let [arg-values# (list ~@args)
result# (apply ~pred arg-values#)]
(if (or result# (= i# ~num-retries))
(do (test/do-report
{:type (if result# :pass :fail)
:message ~msg
:expected '~pred-form
:actual (cond->> (cons '~pred arg-values#)
(not result#)
(cons '~'not))})
(reduced result#))
(Thread/sleep ~pause-ms))))
(range))))
which can be used like:
(deftest test-async
(do-asynchronous-thing)
(is (with-retries (verify-async-process-completed-successfully))))
Of course, this incurs a running time penalty on your tests that can add up quickly, so I'd only recommend using it if you cannot refactor your code in a way that you can guarantee when your async process is done (like let-flow
).I want to avoid changing the functions as they exist, and the future
calls are usually a few levels down from the high-level behavior the tests are supposed to be for.
There are places in the codebase where my coworkers have tried with-redefs
on the future macro itself to try to make it synchronous, but that doesn't actually seem to work.
My plan was to make something that uses with-redefs
on the actual database functions in the body of the future so I can wire up some promises I can block on when the function calls complete.
So I can kinda make this async code synchronous again for the test.
nesting side effects deep under abstractions is an antipattern - there are a few good techniques for lifting the side effecting code and making it more directly observable (eg. a worker reading a queue)
to make a future "sychronous", just deref it
also, if you are calling future and never using deref to access the result, errors that should be throwing and being caught are instead preserved (waiting on a deref before re-throwing)
so you get weird silent failures
So... if I really want to get around this I will probably need to refactor all the way to the point of something like having an 'effects' layer that actually processes thing with a queue and workers, etc. Then having all the domain logic eventually put things into the queue, where I can check it in tests, etc?
that tends to be the principled approach, yes
with-redefs
is not a reliable thing to do with multi-threaded code so probably taking on a different set of pain
Is there such a thing as an in memory writer? I want to use the writer interface but not actually write anything to disk
or File of "/dev/null" on posix systems
@noisesmith can I then read back what was written to the writer that way?
no - I didn't realize that was a requirement
btw the javadoc is great for questions like this for future reference - Writer has links to built in implementations, including the ones mentioned above by @U050MP39D https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/io/Writer.html
I’m trying to do:
(with-open [writer (SomeWriter.)]
(some-thing-that-writes writer)
;; read contents here
)
and clojure has
user=> (javadoc .Writer)
true
which works for both classes and instances, and opens directly in your web browservery handy when you have a random class in the repl, and want to read its docs
user=> (with-open [writer (.StringWriter.)] (spit writer "hi!") (spit writer "bye!") (str writer))
"hi!bye!"
I think it would be more common for the ;; read contents here
part of the code to be outside of the with-open
after writing was complete and the "buffer was closed", but not sure whether it is important for your use case to read it, then conditionally write more. If so, that is a bit fancier of an interface in Java IO libs.
yeah, I was definitely taking advantage of the flexibility and convenience of the StringWriter implementation to make a concise example there
and other writers need flushing etc.
@U0CMVHBL2 good to know, for this case i’m just going to be writing to it once
More specifically, it's not performance but other factors that make me have to use Flutter.
Type inference
Or you can require hints
There's a similar project Clojure to ObjC
So lots of potential reuse