Fork me on GitHub
#clojure
<
2020-03-18
>
noisesmith00:03:41

@mac ( some-stream ( "path" "to" "file.suffix"))

noisesmith00:03:12

io/file also accepts "/path/to/file.suffix"

mac00:03:39

@noisesmith Excellent, thank you.

yuhan10:03:24

What's the idiomatic way of concatenating a transient collection?

yuhan10:03:43

Or a performant way of building up a result [b] given a a -> [b] function and a sequence of [a] as input, not caring about intermediate results

(let [f (fn [n] (repeat n n))]
  (loop [acc (transient [])
         xs [1 2 3]]
    (if (seq xs)
      (recur
        (concat! ;; <= doesn't exist
          acc (f (first xs)))
        (rest xs))
      (persistent! acc))))

p-himik10:03:12

(reduce conj! acc ...) probably.

yuhan10:03:10

Oh right! how did I not think of that

noisesmith13:03:18

or just (into [] (comp (map f) cat) [1 2 3]) - it uses transients for you under the hood

noisesmith13:03:48

I guess (mapcat f) is better, but (comp (map f) cat) should perform identically

noisesmith14:03:05

user=> (into [] (mapcat #(repeat % %)) [1 2 3])
[1 2 2 3 3 3]

noisesmith14:03:23

a simple perf comparison:

user=> (time (dotimes [_ 10000] (into [] (mapcat #(repeat % %)) (range 1000))))
"Elapsed time: 55650.88141 msecs"
vs.
user=> (time (dotimes [_ 10000]
(let [f (fn [n] (repeat n n))]
  (loop [acc (transient [])
         xs (range 1000)]
    (if (seq xs)
      (recur
        (reduce conj!
          acc (f (first xs)))
        (rest xs))
      (persistent! acc))))))
"Elapsed time: 54154.708998 msecs"

👍 4
noisesmith14:03:48

so for a 3% difference in perf at first measurement (that might be smaller than the margin of error(?)) you get a very big difference in code complexity

yuhan15:03:29

Yeah, for this case I'd definitely use a mapcat, but in my real example I had additional state that needed to be threaded through the functions

yuhan15:03:39

so more like (state, a -> state', [b]) , initial-state , [a] -> [b]

gv11:03:06

Hi. Which library can you recommend for watching filesystem changes in a given directory?

gv11:03:51

In particular I need to be notified about new files and want to read them as soon as they are copied completely

orestis11:03:25

https://github.com/wkf/hawk @gv this the most popular one AFAIK

👍 4
jpmonettas11:03:58

hi! Does anybody knows a function to efficiently calculate nested clojure data structures differences? Given some state and and a (fn [state]) => state' efficiently figure out what paths changed from state to state'

jpmonettas11:03:48

thanks @mike1452, will take a look!

slipset13:03:48

So we use RxJS on our frontend. And today this came up https://rxjs-dev.firebaseapp.com/api/operators/distinctUntilChanged Anyone knows if there are transducer implementations of this?

slipset13:03:09

dedupe it is.

slipset13:03:23

although dedupe doesn't let us provide a comparator. dedupe-by ?

Endre Bakken Stovner14:03:23

Is there a library that can report syntax and other errors in clojure files? Preferably with line-numbers and so-on, like shadow-cljs would do.

hindol15:03:25

If you are looking for a linter, clj-kondo helps.

Endre Bakken Stovner15:03:09

But they cannot give feedback on executed code failing, right?

Endre Bakken Stovner15:03:29

Bit more context: I want to create a make-like tool, where the DAG is described in clojure data-structures and perhaps helper functions in clojure. For this, I can just slurp a Make(-like) file written in clojure, read it and eval it, however, then any errors produced by the program will be cryptic.

borkdude15:03:49

clj-kondo can lint invalid code (e.g. {:a 1 :a 2}). if you are looking for a clojure interpreter: https://github.com/borkdude/sci/ you can have control over what functions are and aren't allowed in your clojure DSL while also getting good error messages about unexpected symbols etc the interpreter is compatible with GraalVM so you can make an executable out of it with amazing startup.

borkdude15:03:14

the list of projects in which it is used should give you some idea of what's possible

Endre Bakken Stovner15:03:22

@borkdude Thanks, will keep that in mind. I actually want to use a long-running server for my current program, but it is good to know 🙂

Endre Bakken Stovner15:03:41

Cool, I see that it does more than I thought. So sci is meant for use within long-running programs?

borkdude15:03:51

You can use it in long running programs. E.g. babashka uses sci and babashka has a socket REPL which you can leave running for a few days I guess 🙂

borkdude15:03:44

Note that interpretation with sci is likely slower than clojure.core/eval so if you're looking for performance with regards to hot loops, etc. then clojure.core/eval is faster

borkdude15:03:22

But most DSLs don't need that kind of performance I think

borkdude15:03:36

sci keeps everything isolated. clojure.core/eval also works but it can pollute your clojure RT (created namespaces, vars, etc.)

borkdude15:03:49

if you're looking for a clojure parser for the DAG tools.reader might just work. there's also https://github.com/borkdude/edamame which is configurable: you can allow optional code like features on top of EDN, limit it the way you like.

borkdude15:03:58

(edamame builds on tools.reader.edn, it doesn't evaluate any code)

Endre Bakken Stovner15:03:33

Okay, I will need to try it out. Performance in clojure might be a problem: I want users to be able to write custom clojure code for processing data. Would it be able to use metadata to annotate functions that should be interpreted with eval and for sci to do that?

borkdude15:03:17

this is basically the API of sci:

(sci/eval-string "(plus 1 2 3)" {:namespaces {'user {'plus +}}}) ;;=> 6
you can use this in any configuration you want

borkdude15:03:31

dad is an example of a Make-like tool that's made with sci: https://github.com/liquidz/dad

borkdude15:03:54

spire is a provisioning tool (like Ansible) also made with sci: https://github.com/epiccastle/spire

Endre Bakken Stovner15:03:00

Thanks, I will read those for inspiration. I am still a clojure-newb 🙂

borkdude15:03:07

I see you looking for a parser that can report syntax errors with location information: edamame also does this.

borkdude15:03:33

sorry to overwhelm you with data, let me know if you have any further questions 🙂

Endre Bakken Stovner15:03:33

Can I use arbitrary clojure-libraries from sci?

Endre Bakken Stovner15:03:45

Is it okay if I ask more questions?

borkdude15:03:15

yes and yes

borkdude15:03:46

you will have to bring those libraries in via the :namespaces option like I did in the example before with the plus function

Endre Bakken Stovner15:03:25

I am trying it out now 🙂

Endre Bakken Stovner15:03:48

Neat! It seems to be exactly what I wanted.

Endre Bakken Stovner15:03:23

I see that functions are represented as

#object[sci.impl.fns$parse_fn_args_PLUS_body$run_fn__962 0x6f3a2011
But if these are called from the JVM, will they not be optimized like regular java functions with java hotspot?

Endre Bakken Stovner15:03:29

The performance penalty is if you use GraalVM, right?

borkdude15:03:31

the difference is more like interpretation vs compilation. clojure compiles expressions to efficient JVM bytecode. the sci interpreter doesn't emit bytecode, but will invoke the interpreter to handle the code

Endre Bakken Stovner15:03:47

Understood 🙂 :thumbsup:

Endre Bakken Stovner16:03:51

Like, if sci allowed you to use arbitrary functions by default, would it still be safer than eval in some way?

borkdude16:03:53

all side effects that happen in sci are local, so vars and namespaces you define you won't see in your clojure.lang.RT

borkdude16:03:18

same for multimethods, derive, etc.

borkdude16:03:37

if that's not important to you, then I guess you might as well just use clojure.core/eval

Endre Bakken Stovner16:03:36

I will need to play around with both and see what I need 🙂 Thanks for all the help

Endre Bakken Stovner16:03:50

And either way, I will use edamame 🙂

borkdude19:03:39

@UCMNZLJ93 consider asking this in #spire. the author himself can explain it better to you than I can

ghadi15:03:03

Long-running programs should bias you away from any graal native-image tools (though they are cool)

ghadi15:03:37

because hotspot is really hard to beat

ghadi15:03:27

a lot of native-image web tooling (micronaut, etc.) has low memory usage and fast startup, but not the greatest GCs and no runtime optimization

p-himik15:03:51

Ah, was just about to ask about memory usage. Thanks!

borkdude15:03:12

You can use sci on the JVM though, not necessarily graalvm

ghadi15:03:31

yup, wasn't specifically speaking to sci

borkdude15:03:37

But good point, it's true

Akiz17:03:57

Hi everybody, should I ask about oAuth2 + Ring here or somewhere else? I can not move for two days so it is time to ask 🙂. Thanks

p-himik18:03:19

There's #ring but the question should also be fine here if it doesn't end up creating even more questions. In any case, EAFP. :)

Aleed20:03:06

Someone asked Paul Graham what programming language to learn 😜 Coincidently, it was searching twitter and seeing him tweet this same response years ago was what motivated me to start learning Clojure a few months back. https://twitter.com/paulg/status/1240218967865778177

👍 12
kwladyka20:03:13

Can you remember me why

(s/def ::uuid uuid?)
(gen/sample (s/gen ::uuid))
Unable to construct gen at: [] for: :spec.common/uuid while there is https://github.com/clojure/spec.alpha/blob/31165fec69ff86129a1ada8b3f50864922dfc88a/src/main/clojure/clojure/spec/gen/alpha.clj#L156

kwladyka20:03:30

this one works

(s/def ::uuid (s/with-gen uuid? gen/uuid))

kwladyka20:03:44

But I don’t remember why this doesn’t work by default?

ghadi20:03:24

do you have a local definition for uuid? in that namespace?

ghadi20:03:50

what does (resolve 'uuid?) return?

kwladyka20:03:16

(resolve 'uuid?)
=> #'clojure.core/uuid?

ghadi20:03:46

dunno your sample code works for me

kwladyka20:03:06

org.clojure/clojure {:mvn/version "1.10.1"}
?

ghadi20:03:16

you might have some bad repl state

kwladyka20:03:27

I killed REPL and restarted

ghadi20:03:29

works for me

kwladyka20:03:30

oh I have 1 idea

kwladyka20:03:54

ha I found this. you made me more confident it should work 🙂

kwladyka20:03:11

(:require [clojure.spec.alpha :as s]
            [clojure.spec.gen.alpha :as gen]
            [clj-uuid :as uuid])
This is because of the uuid alias in this ns

kwladyka20:03:20

quite not obviously issue

ghadi20:03:22

seems unrelated

kwladyka20:03:54

hmm now it works even with this ns. I don’t know what is happening, but I am sure I killed REPL and run again a few times to be sure.

kwladyka20:03:00

Well doesn’t matter, it works now.

kwladyka20:03:06

thanks to confirm is should work 👍