Fork me on GitHub
#clojure
<
2021-08-06
>
tessocog01:08:11

i want to build a spec :foo/s3 that matches any value that conforms to either :foo/s1 or :foo/s2 is there a simple thing that goes at $ below?

(s/def :foo/s3 (s/$ :foo/s1 :foo/s2))

tessocog01:08:59

or is it correct to use s/or ?

✔️ 6
dpsutton05:08:25

i've never used trampoline. i always just catch and return a value and then recur considering normal values and the caught value

👍 3
hiredman06:08:54

I use trampoline to roll my own retry loops all the time

kirill.salykin11:08:14

Hi all, I am looking at the implementation of core.cache/lookup-or-miss (https://github.com/clojure/core.cache/blob/master/src/main/clojure/clojure/core/cache/wrapped.clj#L51) and this makes me wondering, does it make sense to use future instead of delay so one can specify timeout? The only difference between future and delay is that future starts immediately after the definition and delay starts on the deref. This doesnt make big difference for the lookup-or-miss implementation. Or do i miss anything and there is a good reason to use delay?

Ed12:08:30

a delay will calculate the value at most once, while a future will calculate the value even if it's not needed. If you look at https://github.com/clojure/core.cache/blob/63aeed08302e2b2c81faf59adade4ead82c87969/src/main/clojure/clojure/core/cache.clj#L57, it seems that the delay is only deref 'd if there's a cache miss.

kirill.salykin12:08:53

A future also caches the value, right?

Takes a body of expressions and yields a future object that will
invoke the body in another thread, and will cache the result and
return it on all subsequent calls to deref/@.
--- so basically delay is used just because it is lazy. did I understand it correctly?

kirill.salykin12:08:19

oke, thanks for clarification!

Ed12:08:01

yes, a future will also only run the calculation once, but it's eager not lazy

quoll18:08:18

The other difference is that futures are executed in another thread (which is the basis for allowing a timeout on them). Delays execute in the current thread when/if they’re needed (as mentioned).

Jakub Šťastný11:08:01

@seancorfield you said on your blog "We have about 111,000 lines of Clojure: about 88,000 is source code and the rest, 23,000, is test code." I'm curious how does this perform on a development machine? In terms of both CPU and MEM. Such as, could you run the whole test suite on a dev machine, or you only run relevant tests and the rest always on CI? How much MEM is needed? What would be the lowest configuration of a computer that'd allow running it? And how about builds? (I don't have an idea, since I'm not really very familiar with JVM. Thanks for any insights!)

seancorfield18:08:38

Our CI pipelines (on BitBucket) use a 4GB instance for one "job" and an 8GB instance for the bigger "job". That includes 2GB in each job used for Docker instances that run MySQL, Elastic Search, and Redis. The bigger "job" only just breaks the 4GB instance limit but 4/8 are our only choices. I do most of my dev on a Mac with 16GB (a 2012 27" iMac), running Docker locally for MySQL, Elastic Search (2 instances), and Redis. Our build.clj script runs tests in a subprocess (similar to how next.jdbc's build.clj script does it). I also do some dev work on my Windows laptop with 16GB (Surface Laptop 3), with everything on WSL2/Ubuntu -- and VS Code and Docker running on Windows (with the WSL2 integration). So that also runs MySQL, 2 ES, and Redis. In production right now we have three 24GB servers I believe, running all the member-facing apps. Most of our running processes have 1GB specified as their max heap but a couple of them are set higher. On staging, most processes are set to use just 256MB. We're in the process of moving to a more numerous cluster of smaller virtual servers -- those 24GB servers date back to when we ran our legacy monolithic web apps that used a lot more heap than the new (micro) services written in Clojure.

👍 3
Tomas Brejla12:08:56

@seancorfield do you use #lsp and #clj-kondo on those big projects? Do they still cope with large codebases okay?

seancorfield15:08:17

@U01LFP3LA6P Yes, I use VS Code/Calva + Clover for the REPL and that uses LSP and clj-kondo under the hood continuously. Every now and then, LSP spins up the CPU and the Clojure features in the editor "hang" for a few seconds but it's fairly rare (maybe once a day?) and I think the LSP team are tracking down a few possible causes.

seancorfield15:08:44

The benefits of LSP/clj-kondo far outweigh the inconvenience of occasional "hangs" like that in my opinion.

Tomas Brejla15:08:36

I'm asking because on one of my customers' project (classical stack of java, spring, hibernate etc.), IntelliJ IDEA is often on the edge (or behind) of usability. I often find myself waiting for Idea to keep up with my code changes. For example Idea is showing some syntax/compile error in my code, I fix it.. and it takes up to 10-15 seconds before Idea finally figures out that the code is now valid. :man-facepalming: That's on project with ~7000 java files, 1000 of which are generated. So I'm wondering how usable large clj(s) project are with calva/kondo/lsp. Good to know that it's usable :thumbsup: .

Tomas Brejla15:08:58

Btw yeah, I know about one of the "CPU spinners" as well. But hopefully those minor glitches will be solved soon. https://github.com/BetterThanTomorrow/calva/issues/1250

seancorfield15:08:30

Well, I don't know how VS Code would deal with such a large Java project -- I have the Java extensions installed but rarely work on any Java code -- but of course Clojure projects are always going to be smaller than their Java equivalent 🙂

truestory 2
Tomas Brejla15:08:58

Not just smaller.. not just smaller 🙂

😄 2
seancorfield15:08:34

That ticket doesn't apply to my setup because I'm not using Calva/nREPL for evaluation. And I keep Clover's REPL output trimmed when I'm working (and mostly hidden). I use Reveal to visualize all my results.

Tomas Brejla15:08:23

Yeah, I'll probably look into possibility of similar setup myself. Not sure if there's need of Clover in order to be able to do that (I hope not, but am afraid that it is). I like the idea of to using either reveal or portal to output every result of evaluation, but didn't have time to try it yet.

seancorfield15:08:30

There's middleware for nREPL I believe that tap>'s every result but we don't use nREPL at work, just Socket REPLs.

adi12:08:36

@jakub.stastny.pt_serv Not speaking for Sean, obviously, but I don't think developing against a 100K line Clojure codebase should be a problem. At a previous employer, the bread-and-butter monolith has grown to push about half a million lines of Clojure (all-inclusive source+test+docs, over the last 8-9 years). It does a lot of heavy lifting for the company. Everybody develops against it locally on their Macbook Pros. First time app startup with a REPL takes ten-odd seconds, but subsequently REPL-driven development is business as usual. Unit+integration test coverage is good. People frequently run unit tests pre-push. The git forge is configured to run unit tests on a per-commit basis. The challenge usually is running integration tests, against live DBs/services, where the problem is hardware resources for the live system as a whole, not the Clojure app. They put in a fair bit of work to optimize integration test run times, such as https://medium.com/helpshift-engineering/on-the-testability-of-ring-middlewares-in-clojure-6795eae60f2a, https://medium.com/helpshift-engineering/a-study-in-parallelising-tests-b5253817beae, and https://medium.com/helpshift-engineering/the-convoluted-magic-of-leiningen-test-selectors-2eb6c452dfcf.

👍 3
Jakub Šťastný13:08:36

That's good news definitely.

Jakub Šťastný13:08:49

How much RAM do these MB Pros have?

Jakub Šťastný13:08:13

Or better, how much RAM do you need to develop this app?

adi14:08:46

Hm, whatever the standard MBP is... 16 GB, I think. > Or better, how much RAM do you need to develop this app? As a ballpark estimate, I can say fairly confidently that at dev time, the app's JVM process itself requires far lower memory than certain Electron apps that we use for instant messaging :) However, developing is very different from running the app standalone. The latter requires appropriate memory + heap allocation, and extra capacity depending on the load it needs to handle. I don't know their actual infra, but I suspect each app node for this particular app would probably be https://aws.amazon.com/ec2/instance-types/.

Jakub Šťastný14:08:16

Yeah that's what I suspected. With Ruby/Node.js I was fine with a free AWS micro instance with 1GB, so this is a huge change.

jumar15:08:54

You can run clojure app on that - at least medium sized web app that doesn’t do heavy processing

adi16:08:23

@jakub.stastny.pt_serv To be clear, that node configuration is for the app running in a production service that serves several thousand requests per second.

adi16:08:03

And they're over-provisioned to absorb large intra-day swings in traffic.

Carlo14:08:06

how do you usually generate recursive data via spec/exercise on a recursive spec? I easily get stack overflows (understandably) and am wondering which trick does the community use

Russell Mull14:08:07

I recall having some issues with this as well, but it was many years ago now. In principle, https://clojuredocs.org/clojure.spec.alpha/*recursion-limit* should help, but I seem to remember that it doesn't always. I think in my case I ended up using s/gen to attach my own generator, written using gen/recursive-gen: https://github.com/clojure/test.check/blob/master/doc/intro.md#recursive-generators

🙌 3
Carlo16:08:44

thanks @U7ZL911B3, gen/recursive-gen was what I needed. I was confused because this process is automated, eg, in Haskell, and I was trying to understand if someone had an automatic way of doing it

Jim Newton14:08:55

I'm trying to experiment with http://clojure-goes-fast.com/blog/profiling-tool-async-profiler/ it said to include the line [com.clojure-goes-fast/clj-async-profiler "0.1.0"] in my project.clj/build.boot/: file. I'm not sure what that means, but when I add it to my project.clj file and try the example (prof/start {}) I get the following error

Execution error (IOException) at sun.tools.attach.HotSpotVirtualMachine/<init> (HotSpotVirtualMachine.java:75).
Can not attach to current VM
with a stacktrace
Show: Project-Only All 
  Hide: Clojure Java REPL Tooling Duplicates  (13 frames hidden)

2. Unhandled java.lang.reflect.InvocationTargetException
   (No message)

NativeMethodAccessorImpl.java:   -2  jdk.internal.reflect.NativeMethodAccessorImpl/invoke0
NativeMethodAccessorImpl.java:   62  jdk.internal.reflect.NativeMethodAccessorImpl/invoke
DelegatingMethodAccessorImpl.java:   43  jdk.internal.reflect.DelegatingMethodAccessorImpl/invoke
               Method.java:  566  java.lang.reflect.Method/invoke
NativeMethodAccessorImpl.java:   -2  jdk.internal.reflect.NativeMethodAccessorImpl/invoke0
NativeMethodAccessorImpl.java:   62  jdk.internal.reflect.NativeMethodAccessorImpl/invoke
DelegatingMethodAccessorImpl.java:   43  jdk.internal.reflect.DelegatingMethodAccessorImpl/invoke
               Method.java:  566  java.lang.reflect.Method/invoke
            Reflector.java:  167  clojure.lang.Reflector/invokeMatchingMethod
            Reflector.java:  102  clojure.lang.Reflector/invokeInstanceMethod
                  core.clj:  101  clj-async-profiler.core/mk-vm
                  core.clj:   97  clj-async-profiler.core/mk-vm
                  core.clj:  106  clj-async-profiler.core/attach-agent
                  core.clj:  103  clj-async-profiler.core/attach-agent
                  core.clj:  124  clj-async-profiler.core/start
                  core.clj:  116  clj-async-profiler.core/start
                  core.clj:  121  clj-async-profiler.core/start
                  core.clj:  116  clj-async-profiler.core/start
                      REPL: 2213  clojure-rte.rte-core/eval10871
                      REPL: 2213  clojure-rte.rte-core/eval10871
             Compiler.java: 7176  clojure.lang.Compiler/eval
             Compiler.java: 7131  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj: 1973  clojure.core/with-bindings*
                  core.clj: 1973  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  414  clojure.main/repl/read-eval-print/fn
                  main.clj:  414  clojure.main/repl/read-eval-print
                  main.clj:  435  clojure.main/repl/fn
                  main.clj:  435  clojure.main/repl
                  main.clj:  345  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  202  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  201  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  834  java.lang.Thread/run

1. Caused by java.io.IOException
   Can not attach to current VM

HotSpotVirtualMachine.java:   75  sun.tools.attach.HotSpotVirtualMachine/<init>
   VirtualMachineImpl.java:   56  sun.tools.attach.VirtualMachineImpl/<init>
   AttachProviderImpl.java:   58  sun.tools.attach.AttachProviderImpl/attachVirtualMachine
       VirtualMachine.java:  207  com.sun.tools.attach.VirtualMachine/attach
NativeMethodAccessorImpl.java:   -2  jdk.internal.reflect.NativeMethodAccessorImpl/invoke0
NativeMethodAccessorImpl.java:   62  jdk.internal.reflect.NativeMethodAccessorImpl/invoke
DelegatingMethodAccessorImpl.java:   43  jdk.internal.reflect.DelegatingMethodAccessorImpl/invoke
               Method.java:  566  java.lang.reflect.Method/invoke
NativeMethodAccessorImpl.java:   -2  jdk.internal.reflect.NativeMethodAccessorImpl/invoke0
NativeMethodAccessorImpl.java:   62  jdk.internal.reflect.NativeMethodAccessorImpl/invoke
DelegatingMethodAccessorImpl.java:   43  jdk.internal.reflect.DelegatingMethodAccessorImpl/invoke
               Method.java:  566  java.lang.reflect.Method/invoke
            Reflector.java:  167  clojure.lang.Reflector/invokeMatchingMethod
            Reflector.java:  102  clojure.lang.Reflector/invokeInstanceMethod
                  core.clj:  101  clj-async-profiler.core/mk-vm
   

Jim Newton14:08:39

anyone have an idea what's happening?

delaguardo14:08:13

which JVM are you running?

delaguardo14:08:40

and on which operating system?

Jim Newton14:08:27

it is macOS bigsur.

Jim Newton14:08:00

which JVM?

jnewton@Marcello Repos % java -version
java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)

delaguardo14:08:54

ok, then it should work. Could you try latest version of clj-async-profiler? https://github.com/clojure-goes-fast/clj-async-profiler from here it is 0.5.1

delaguardo14:08:37

the reason I asked about OS and JVM is that it wont work on Windows

Jim Newton15:08:26

OK, I updated my project.clj to have the line [com.clojure-goes-fast/clj-async-profiler "0.5.1"] , and restarted my repl. I still get what appears to me to be the same error.

Jim Newton15:08:32

haven't verified every line.

Jim Newton15:08:01

is there a way to ask at the repl, which version of a dependency was loaded?

delaguardo15:08:26

you can use lein deps :tree if you are using lein to start the repl

delaguardo15:08:40

but not from the repl

Ben Sless15:08:51

you need to allow attachment

Ben Sless15:08:20

> JDK9+: you must start the JVM with option `-Djdk.attach.allowAttachSelf`, otherwise the agent will not be able to dynamically attach to the running process

✔️ 3
Ben Sless15:08:48

The exception is Can not attach to current VM, that's likely the cause

Ben Sless15:08:13

also > Ensure that `perf_event_paranoid` is set to 1 or less

delaguardo15:08:10

last one is for linux, should not be a problem on mac os

Ben Sless16:08:31

yeah but -Djdk.attach.allowAttachSelf is critical

Jim Newton12:08:02

great! that seems to work well. It's amazing which random things you people know 🙂

Ben Sless14:08:04

At this point I probably went through all the JVM profiling tools, feel free to ask weird questions

Ben Sless14:08:12

besides yourkit

Jim Newton08:08:36

Hi Ben, do you have much experience looking at flame graphs. I'm trying to look at one and understand what it's telling me. or at least what it I should infer to make my program faster.

Jim Newton08:08:53

This is just a sample. I have not yet investigated whether it is a typical run. Is it telling me the number of times functions are called, or is it telling me the amount of time spent in functions? it's call cpu-flamegraph so I'd guess it probably is trying to imply something about total cpu time.

Jim Newton08:08:55

one thing I notice about the flame graph created by clj-async-profiler is that it is difficult to understand what is happening (profiling-wise) when a multi-function gets called. i.e., how much time is spent in any of the various multi-methods.

delaguardo09:08:14

there is a trick I used to identify which method of defmulti been called

(defmulti foo identity)

(defmethod foo 1 one [_] "one")

(defmethod foo 2 two [_]
  (throw (ex-info "From two" {})))

(foo 1) ;; => "one"

(foo 2) ;; => Exception with :at [spapi.client$eval6527$two__6528 invoke "client.clj" 331] in stacktrace
Note the symbol after dispatch value in defmethods. Because it is a part of & fn-tail it become the “name” of the implementation.

delaguardo09:08:15

the idea is the same as having the “name” for anonymouse function defined with fn

(fn foo [x] (+ x 1))

4
Jim Newton09:08:05

ahhh, that's clever!

delaguardo09:08:24

not ideal thou, because the name will be decorated with generated suffix ( so it will take some time to get used reading traces

4
Ben Sless09:08:31

Okay, flame graphs and you, and short intro

Ben Sless09:08:20

Flame graphs are two dimensional representation of % of time spent doing something on the X axis, and how deep you are in the stack (and where) on the Y axis

Ben Sless09:08:34

Thus it is a sort of histogram

Ben Sless09:08:22

Think of an entire stack as a bin, and stacks with similar roots are joined

Ben Sless09:08:39

thus the stack X->Y->Z: 3 X->Y->W: 2 Will be joined and displayed as: X->Y (5) -> {Z(3), W(2)}

Ben Sless09:08:54

What you see and how to read it

Ben Sless09:08:18

In clj-async-profiler, stack traces related to Clojure are in pretty Clojure colors, i.e blue and green

Ben Sless09:08:05

Native stuff is in red:

Ben Sless09:08:57

In unappealing yellow you'll see the JIT compiler and GC

Ben Sless09:08:14

looking at your graph, it's mostly compiler work

Ben Sless09:08:29

This means you need to let the JIT have more time to work

Ben Sless09:08:01

Assuming you profiled you function like so: (profile my-expr) let the JVM burn some CPU getting to know it better: (dotimes [_ 10] my-expr) Then try profiling again Also, I usually use dotimes when profiling to get more samples for more accurate results

Ben Sless09:08:40

Reading the actual results: I start by clicking the base of the pyramid of Clojure colored stacks, on java.lang.Thread.run, which zooms me in to that stack

Ben Sless09:08:32

From looking at these stacks, it looks like you ran your expression only once

Ben Sless09:08:57

Am I correct?

Ben Sless10:08:54

I suggest letting the code warm up then rerunning with plenty of samples

Jim Newton11:08:35

yes ran the expression only once.

Jim Newton11:08:15

but it worked over a pretty large data structure which contains lots of redundant code paths. at least that was my intent

Jim Newton11:08:44

BTW about the flame graph. If the same function is called from two completely different call trees, will it appear once or multiple times in the graph? I think it will appear multiple times, once per call-tree. that is of course sometimes what you want. but sometimes, you just want to see the functions independent of their call trees.

Ben Sless11:08:50

Multiple times

Ben Sless11:08:03

Joining is from the root of the call tree

Ben Sless11:08:47

From looking at the flame graph the data is insufficient. Let it run for at least one minute

Jim Newton11:08:20

BTW I was wrong about the create-or function. Actually I have two functions of the same name in two different name spaces. it was the other one which apparently appears in the flame graph. 😞

Jim Newton11:08:54

but that other one is very similar to this one, it works on a different type of data structure, but its structure is very similar.

(def rte/create-or
  (fn [operands]
    (cond (empty? operands)
          :empty-set

          (= 1 (count operands))
          (first operands)

          :else
          (cons :or operands))))

Ben Sless11:08:20

This will probably be useful in profiling the expression for an arbitrary duration:

(defmacro profile-for
  [ms & body]
  `(let [a# (atom true)
         p# (future (prof/profile (while @a# ~@body)))]
     (Thread/sleep ~ms)
     (reset! a# false)
     @p#))

Jim Newton12:08:13

is there something funny about profiling a test case using clojure.test? I mean does the test driver artificially re-trigger the compiler?

Ben Sless12:08:02

Don't think so, but it's one-shot. Profiling is empirical, you need lots of samples

Jim Newton12:08:33

yes I ran the test case 1000 times (previously only 1 time) and I still see lots of compiler overhead.

Jim Newton12:08:06

maybe I’m reading the graph wrong. let me study it a bit more before complaining too much

Ben Sless12:08:00

I just do it interactively

Ben Sless12:08:44

I also don't know how you run the tests

Ben Sless12:08:09

but generally, I wouldn't tie profiling and benchmarking to my tests, too different

Jim Newton12:08:21

Here is what I’m typing into the repl:

clojure-rte.rte-core> (prof/start {})
"Started [cpu] profiling\n"
clojure-rte.rte-core> (doseq [r (range 1000)]
                        (clojure.test/test-vars [#'clojure-rte.rte-canonicalize/t-canonicalize-pattern-714]))
[:count 100]
[:count 200]
[:count 300]
[:count 400]
[:count 500]
[:count 600]
[:count 700]
[:count 800]
[:count 900]
[:count 1000]
nil
clojure-rte.rte-core> (prof/stop {})
#object[java.io.File 0x64837b2 "/tmp/clj-async-profiler/results/05-cpu-flamegraph-2021-08-09-14-03-04.svg"]
clojure-rte.rte-core> 

Jim Newton12:08:56

(clojure.test/test-vars …) calls a test case by name

Ben Sless12:08:07

right, let me poke at it

Jim Newton12:08:35

I could of course copy the code out of the test into a normal function and see if the result is the same.

Ben Sless12:08:20

you don't even have to use testing

Ben Sless12:08:35

(prof/profile (dotimes [_ 1e3] (clojure-rte.rte-canonicalize/t-canonicalize-pattern-714)))

Ben Sless12:08:48

tests are just functions with extra metadata

Ben Sless12:08:00

Are there some fixtures you need to run?

Jim Newton12:08:39

by fixtures do you mean setup and tear down?

Jim Newton12:08:57

no. hey, when I copy the test code into just the repl, I see very different results from the profiler. bizare.

Jim Newton12:08:58

I wasn’t expecting that difference

Ben Sless12:08:24

by the way, how long does profiling take? Should be at least 30 seconds

Jim Newton12:08:26

no, its about 10 secs

Jim Newton12:08:18

I can up it by 4.

souenzzo14:08:46

@jakub.stastny.pt_serv I worked on two 100k cloc codebases, one with 5y of continuous development. Both require at least ~8Gb to have a good development experience. But this is mostly because of specific tools like #onyx or #shadow-cljs . If i start just the HTTP server, that uses most of the code, it should consume only 1Gb of RAM. On both codebases I have subsecond test feedback, like I edit a file, load it, run a test and get the output, everything in less than 1s (usually ~0.1s).

Jakub Šťastný14:08:23

Thanks. Yeah I'm hoping I could make it work on 1GB RAM on a free AWS micro instance.

valtteri15:08:55

Could you consider using ClojureScript? Node is less hungry for memory

jumar15:08:37

What I said in the other thread- you should be able to do that.

souenzzo15:08:41

@U6N4HSMFW this is not simple as "x is better than Y" Both #graalvm native-image tool and node can have faster startup times and less initial memory footprint But if you are processing a lot of data, JVM will be faster and consume less memory than the other alternatives, in the most cases. for sure you can craft a problem where @jakub.stastny.pt_serv to run in smaller machines, I recommend you to create an AOT Uberjar to have better startup times. Also, avoid "huge horizontal scaling". As JVM uses JIT and Clojure do structural sharing, 2x 4Gb machines will perform way better than 8x 1Gb machines. But for sure, 2x 1Gb machines is safer than 1x 2Gb machines

valtteri15:08:16

Sure it depends on what kind of work the app is doing among many other things. However in my experience node is less hassle on machines with limited amount of memory compared to jwm .

vemv16:08:02

re: RAM, to put things in perspective, it's not unheard of to have production JVMs running multi-TB workloads... So more often than not, consumer-grade laptops will skew your view of what low/high RAM looks like I launch all my JVMs with a Xmx of 24GB, enough to do arbitrary Clojure apps/tooling do their thing without running into superfluous OOMs. They can particularly happen given how many Clojure algorithms are developed (`walk` , laziness etc). Other flags can be necessary as well

indy17:08:04

Damn 24GB is quite a lot, when GC happens it must be quite the GC.

vemv18:08:13

not necessarily, GCs happen quite regularly. there are 'minor' and 'major' ones will depend on the GC of choice and its params of course

indy18:08:14

That was going to be my follow up question, which GC are you using? Because I'm pretty sure when I'm using G1 and set the memory tad too high, then the GC pause is very annoying.

vemv18:08:29

G1 and these settings https://github.com/reducecombine/.lein/blob/00468aefbfdfb3b4c9f4eec44a318dc3c3db377e/profiles.clj#L53-L57 don't recall a GC STW pause over the 2-3 years I've used it still I'm looking forward to try out the newer GCs :) promising sub-ms GC

indy18:08:09

Nice, I guess tuning the params matter as much.

indy18:08:54

Going to try these settings out

🙂 3
lilactown16:08:50

I decided to check and we have around 500k lines of clojure (clj/cljc/cljs) in the monorepo I work in 🥴

😮 3
lilactown16:08:38

of course this is split across many services which we rarely run all at once on our local machines

Nazral16:08:47

Behavior that I did not expect:

(select-keys {:a/b "foo"} [:a/b])
;; => #:a{:b "foo"}
Why is it not returning {:a/b "foo"} ?

Russell Mull16:08:39

It did.

☝️ 6
3
isak16:08:56

That is just a 'pretty-printed' version of the same data

dpsutton16:08:00

type {:a/b "foo"} in your repl

Russell Mull16:08:08

#:a{:b "foo"} is shorthand for "every key in the map has a ns of as"

Nazral16:08:55

I see, thank you

thumbnail17:08:06

You can (set! *print-namespace-maps* false) to disable this btw

🙏 3
seancorfield18:08:38

Our CI pipelines (on BitBucket) use a 4GB instance for one "job" and an 8GB instance for the bigger "job". That includes 2GB in each job used for Docker instances that run MySQL, Elastic Search, and Redis. The bigger "job" only just breaks the 4GB instance limit but 4/8 are our only choices. I do most of my dev on a Mac with 16GB (a 2012 27" iMac), running Docker locally for MySQL, Elastic Search (2 instances), and Redis. Our build.clj script runs tests in a subprocess (similar to how next.jdbc's build.clj script does it). I also do some dev work on my Windows laptop with 16GB (Surface Laptop 3), with everything on WSL2/Ubuntu -- and VS Code and Docker running on Windows (with the WSL2 integration). So that also runs MySQL, 2 ES, and Redis. In production right now we have three 24GB servers I believe, running all the member-facing apps. Most of our running processes have 1GB specified as their max heap but a couple of them are set higher. On staging, most processes are set to use just 256MB. We're in the process of moving to a more numerous cluster of smaller virtual servers -- those 24GB servers date back to when we ran our legacy monolithic web apps that used a lot more heap than the new (micro) services written in Clojure.

👍 3
meta-meta19:08:07

(ns foo.bar
  (:use [foo.baz :as b]))

(defn call-b [msg] (b msg)) 
What's an idiomatic way to test that (call-b 123) invokes foo.baz with msg?
(deftest call-b (with-redefs  ...?

Np19:08:40

You can use https://github.com/circleci/bond to spy on a fn. This example shows the spying on a fn and getting count for fn calls.

(ns test.foo
  (:require [bond.james :as bond :refer [with-spy]]))

(defn foo [x]
  (let [shaken (with-out-str (prn :martini))]
    [shaken])

(defn bar [y]
  (foo y))

(deftest foo-is-called
  (with-spy [foo]
    (bar 2)
    (is (= 1 (-> foo bond/calls count)))))

meta-meta20:08:45

I'd rather keep it vanilla clojure for now. Thank you though I'm sure that will be useful in the future.

didibus20:08:59

Ya I was going to recommend Bond as well. You can do something with with-redef, which would basically be re-implementing a lesser variant of Bond. But I'd say using an existing library would be better. ALTERNATIVELY I will say don't bother unit testing this. If b is pure, test b and test call-b. If b is impure, write an integ test

meta-meta20:08:28

Does bond spy and mock? because I don't want foo to be mocked as a no-op. Only interested in what my code is calling it with.

didibus20:08:47

If you're curious about with-redef, I've done this before:

(deftest call-b
  (let [b-spy (atom :not-called)
        orig-b b]
    (with-redefs
      [b (fn [& args]
           (reset! b-spy args)
           (apply orig-b args))]
      (is (do (call-b 123)
              (= [123] @b-spy))))
Typing this on my phone so sorry if there are some small mistakes, but that's the gist Edit: Rewrote to use vararg, since it's more applicable this way for spying on any function using with-redefs.

Np20:08:53

There are two main offerings of bond - 1. Spy - this will just spy on target function and won’t interfere with function definition. 2. Stub (mock) - you can use it to redefine a function. Additionally This also provides spy benefits

👍 3
Np20:08:38

Bond uses with-redefs internally to implement these functionalities. If you want to stick with Vanilla clojure. You will have to repeat the effort.

didibus20:08:44

Actually Bond had a commit a while back where it no longer use with-redefs, it uses something that is thread local and thread safe instead, so it's even better

👍 3
Np20:08:11

There is one major drawback of with-redefs. It redefines the function globally. This means that another test running in parallel will be able to see the new definition. To solve this problem, Helpshift has added new functions over bond to provide local-stub and local-spy . These fns will be mocked in the current thread only and this redefinition wouldn;t interfere with another thread. https://github.com/helpshift/bond

✔️ 3
Np20:08:45

Unfortunately the bond repo maintainer decided to not merge the request for code complexity/maintenance reasons. Checkout this https://github.com/circleci/bond/pull/47

didibus20:08:19

Ah, too bad. I'll keep an eye out for your fork.

noisesmith20:08:41

aside, I'd be suspicious of tests that make assertions about implementation details of the thing tested (did some f get called) as opposed to testing input / output values. in functional code you can reify the selection of function / arg as its own function (whose input / output you can test) if that's a property that's worth testing

hiredman19:08:43

(deftest call-b (is true))

hiredman19:08:26

kidding aside I would make b a parameter of call-b, and pass in something else in the test

hiredman19:08:32

it is usually better to directly test that what you want to happen happens, instead of testing that the function you think will do what you want gets called

3
💯 3
meta-meta19:08:34

Hmm, I don't totally follow. I think I want to mock foo.baz and test that it is called with msg

meta-meta19:08:49

I don't want to test foo.baz here. It's a networking library that I don't want to spin up.

dominicm19:08:58

@meta-meta Functions aren't usually defined as "calls foo.baz" though.

dominicm20:08:29

So that's not a useful property to test. You could change how it does it's job (use a different library), and you likely want the test to keep passing.

dominicm20:08:53

@meta-meta Why do you want to test that call-b calls foo.baz?

meta-meta20:08:05

It's hypothetical. I'm trying to test how my fn is using an imported library

dominicm20:08:06

If that's the exact property you're trying to test, then with-redefs is your best option. But beware it stops you from using test parallelism functionality.

hiredman20:08:12

oh, then for sure parameterize it

☝️ 3
dominicm20:08:28

Oh, I missed the "how".

hiredman20:08:44

(defn call-b [b msg] (b msg))

dominicm20:08:47

I read "I'm trying to test my fn is using an imported library". Oops!

😀 3
hiredman20:08:04

then in your test pass in a different b

dominicm20:08:55

@meta-meta Does call-b do some processing on it's parameters before calling foo.baz? And that's why you want to test it?

Edward Ciafardini20:08:46

Trying to use Twilio in a project, and I keep getting this error: Syntax error macroexpanding clojure.core/ns at (twilio/core.clj:1:1). ((import [ URLEncoder]) (:require [org.httpkit.client :as http] [clojure.string :as str])) - failed: Extra input spec: :clojure.core.specs.alpha/ns-form my dependencies: :dependencies [[org.clojure/clojure "1.10.1"] [twilio-api "1.0.0"]]

vemv20:08:22

There's likely a typo in your ns form, feel free to post it if the error persists

Edward Ciafardini20:08:50

(ns hotline.core (:require [twilio.core :as twilio]) (:gen-class))

vemv20:08:34

reading the error message more carefully, it might be a syntax error in their lib. It's not that rare; the ns form rules were strengthened in recent Clojure versions I'd check twilio/core.clj and raise an issue etc In the meantime you can disable spec-checking at macroexpansion time

Edward Ciafardini20:08:04

How do I disable spec-checking ?

meta-meta20:08:25

yes. exactly. I was trying to distill it down to ask my question.

dominicm20:08:42

@meta-meta I'd strongly recommend pulling out the processing code from call-b . Instead of:

(defn call-b [msg]
  (foo.baz (inc msg)))
Do
(defn awesomeify-msg [msg]
  (inc msg))

(defn call-b [msg]
  (foo.baz (awesomeify-msg msg)))
and then you can just test awesomeify-msg AND you can test it from the REPL easily too!

👍 3
didibus20:08:13

Since people are bringing up best practice idioms, I would recommend going one step further and never mixing pure and impure:

(defn awesomeify-msg
  [msg]
  (inc msg))

(defn call-b
  [msg]
  (io/b msg))

(defn -main 
  []
  (let
    [msg (make-msg 10)
     awesome-msg (awesomeify-msg msg)]
   (call-b awesome-msg)))
That way you just integ test call-b and unit test awesomify-msg and you integ test or spy with mocked call-b -main. Though I don't bother and just go with an integ test personally. So you end up with exclusively pure functions or exclusively impure functions, and a top level orchestrator function that is the only thing that mixes pure and impure.

👍 11
dominicm20:08:18

I almost pulled up the quote I saved from the clojure discourse of yours 🙂

😏 3
vemv21:08:38

I second the sentiment but never mixing pure and impure: is badly formulated. It would be unsustainable for an impure function not to be able to use inc or assoc Alt formulation: decouple pure from impure parts whenever possible

didibus22:08:00

The "whenever possible" is to the discretion of the developer. But I don't think it's that unsustainable to do so. Why would you need to do inc or assoc in an impure function? I can see assoc maybe like to construct the return value, but inc I can't see why.

vemv23:08:57

Perhaps moving on from a less fruitful discussion, is this a pattern you've used at scale? Especially considering never which is a quite unmistakable rule. As mentioned I like the overall intent but at scale, in a team it can predictably fall under its own weight. Especially when no tooling exists (e.g. a framework for functional architectures, or a linter for side-effect segregation)

didibus06:08:26

I get that "never" is too strong, we probably try to follow it as much as we can, and call it out on CRs when we spot it not followed. But generally I'd say we've done this at scale. The only difference is that you end up needing more than a single giant top level orchestrator function, you can create sub-orchestrators, this does make things clearer. But its trying to split between: 1. Only IO 2. Only pure 3. Only orchestration I have to explain 3 a bit, but basically 3 ends up just as a way to connect the input/ouput of other functions. So say you have a business process that's like modify the input somehow, then query the db for something, and using what you queried finish modifying the input to return a result. You have pure -> impure -> pure. You'd have a function that's literally:

(defn foo-process
  [input]
  (-> (pure/logic input)
      (impure/io)
      (pure/more-logic input)))
And if you happen to need this foo-process as part of other processes, that's fine. This becomes a sub-process.
(defn -main
  [input]
  (let [foo (foo-process input)
        bar (bar-process foo)]
    {:a (:result bar} :b (:result foo)}))
But what happens when you do things like this is you really maximize reuse, and decouple each pure/impure pieces from each-other.

didibus06:08:04

I wrote an example of this recently here: https://gist.github.com/didibus/ab6e15c83ef961e0b7171a2fa2fe925d Its a fake example, but I was trying to show a non-trivial use case. Now the process-bar would be top-level API, in this case I'd normally break it down into some sub-process, and also the "retry mechanism" I am using in real code would be hidden behind a macro (which you can find here: https://stackoverflow.com/a/68515868/172272), but still in my opinion, it is all quite readable, though feedback welcome 😄

vemv14:08:41

so yeah we think pretty similarly :) we agree this is a best-effort thing Normally in a code review (at least in a few teams) indeed we'd suggest to decouple IO, so that one could have a unit test and a more heavy test. I've never gone as far as having a concrete notion/architecture of 'orchestrators'. I'd say that the main difficulty lies in: • conveyance - how do you know a given defn is pure / impure? It's rarely as easy as eyeing one defn - defns can call each other, recursively. naming conventions can help but have their limits • enforcement - how do you know the codebase-wide segregation effort is veridic? Without enforcement there could be hidden impure corners which could foil the whole effort / might be hard to revert etc

vemv14:08:47

Perhaps an interesting read: https://github.com/clj-kondo/clj-kondo/pull/751 we left the effort as there were some pretty undecidable cases

didibus20:08:50

If I were to go the linter route, I.dont think I'd use the shebang, since it doesn't really indicate side-effect, more like warns you of caveats, which aren't always side-effectful. I think I'd go for a new meta tag, like ^:pure and assume all things not tagged with it are impure. And I'd have the linter make sure a ^:pure tagged function isn't using anything not tagged ^:pure. And I'd let people configure things to be pure in a config as well if you want to backfill libraries like clojure.core. But I'd also have it work like type hints:

(defn foo
  ^:pure [a b]
  ^:pure (+ a b))
So maybe clj-kondo doesn't know core/+ is pure but you can type hint it inside your pure fn to teach it. And there'd be a config (a bit like how you can teach clj-kondo types, in fact maybe the same one could be used) where you could mention it being pure so overtime you can backfill libs and don't have to add meta everywhere.

didibus20:08:26

Also interesting to look at ghostwheel: https://github.com/gnl/ghostwheel

tessocog22:08:20

where can i ask Oz and Vega related questions?

noisesmith23:08:48

I don't think there's anything simpler than what you have there

noisesmith23:08:58

IMHO binding things to names in a let block is underappreciated, the alternatives are often clever but not always clearer

hiredman23:08:50

((constantly (fn-that-we-want-the-return-value-of)) (log/info "finished getting stuff"))

noisesmith23:08:08

I didn't think doto was relevant here since the log statement doesn't use the calculated value

hiredman23:08:02

constantly is exactly the thing

hiredman23:08:56

depending on what kind of exceptional behavior you want there is also finally

noisesmith23:08:06

right - one might need to stop and think about constantly evaluating its arg before the args to the function it produces are evaluated for example

noisesmith23:08:51

oh yeah lol (try (fn-that-we-want-the-return-value-of) (finally (log/info "done"))) - cute

hiredman23:08:07

in common lisp you would have prog1, progn being the common lisp version of do

hiredman23:08:24

I've worked in code bases that had a do1 macro defined in that pattern

hiredman23:08:04

you may as well take & after-form and splice it

hiredman23:08:44

do1 leans pretty heavily on knowing the common lisp idiom and correspondence of progn and do