Fork me on GitHub
#clojure
<
2019-06-19
>
souenzzo00:06:52

(defprotocol IFib
  (fib-nth [this ^Long n]))
(deftype Fib [^BigInteger prev
              ^BigInteger current
              ^BigInteger tmp
              ^Long idx]
  IFib
  (fib-nth [this n]
    (set! (.-idx this) n)
    (while (pos? idx)
      (set! (.-idx this) (dec idx))
      (set! (.-tmp this) (.add prev current))
      (set! (.-prev this) current)
      (set! (.-current this) tmp))
    current))
(fib-nth (new Fib BigInteger/ZERO BigInteger/ONE BigInteger/ZERO (long 0)) (long 7))   
- Can I do that without protocols? (without performance hint) - Can I do it faster on JVM?

noisesmith00:06:31

all of those set! calls are expensive, because they require synchronization

souenzzo00:06:56

Oh god. Sorry

(defn fib-nth-v2
  [n]
  (loop [^BigInteger prev BigInteger/ZERO
         ^BigInteger current BigInteger/ONE
         ^Long idx n]
    (if (pos? idx)
      (recur current
             (.add prev current)
             (dec idx))
      current)))
It's simpler, easyier and ~20% faster.

hiredman00:06:04

type hinting with ^Long is almost always terrible

hiredman00:06:31

it means "this is a boxed long object" which if you are type hinting is never what you want

👍 4
souenzzo00:06:35

^long goes to the primitive java long?

noisesmith00:06:53

directly using the values in a loop/recur and not mutating is much faster

👍 12
noisesmith00:06:17

(long 7) is silly

hiredman00:06:47

like, just write a function

hiredman00:06:06

use loop recur, using set! is silly

hiredman00:06:01

the above just reads like someone who doesn't know what they are doing trying to blindly optimize the code

hiredman00:06:18

I should say, "someone who doesn't know clojure and doesn't have experience with functional programming"

souenzzo00:06:14

I'm just trying to play with some mutable state

admay13:06:45

If mutable state is what you're looking for, I'd suggest you check out ref, atom, var, and agent http://clojure.wladyka.eu/posts-output/2016-04-06-share-state.html#agents If you haven't heard of it, it's a part of what's called the Clojure STM and is the 'Clojure' way of doing anything involving mutation in a very controlled manner Here's the Brave Clojure chapter on it, https://www.braveclojure.com/zombie-metaphysics/

👍 4
admay13:06:36

However, if you're mutating things in Clojure, most of the time there's a simpler way to do it

hiredman00:06:32

fib is basically the worst way to do that, because fib is not a function that requires mutable state and is an almost universal example for teaching how to write a pure function

hiredman00:06:24

playing with mutable state and fib is like "how can I write fib in a way that is harder to understand, and will perform worse?"

hiredman00:06:50

something like

(defn fib ^long [^long n]
  (loop [prev 0N
         current 1N
         idx n]
    (if (pos? idx)
      (recur current
             (+ prev current)
             (dec idx))
      current)))

hiredman00:06:16

is easier to understand, and, well, I haven't bench marked, and the jvm can really chew through code, but I would be surprised if it isn't as fast or faster then that nonsense

jaihindhreddy04:06:26

^ It is faster than that "nonsense" 565.453129 ns vs the nonsense being 973.900396 ns.

asolovyov07:06:13

hey everyone! I'm trying to make a new data reader here, and my data_readers.cljc contain following: {t kasta.i18n/t} while src/kasta/i18n.clj contains

(ns kasta.i18n)

(defn t [arg1] arg1)
The problem is, I mark something in completely unrelated namespace and I get this error:
Syntax error reading source at (mk/fe/common/header.cljc:42:29).
Attempting to call unbound fn: #'kasta.i18n/t
I wonder what I could be doing wrong? I moved out i18n ns to a separate library since I thought that could be a problem, but no luck here. Any pointers?

p-himik08:06:36

Do you declare kasta.i18n/t anywhere? If so, is it possible that something tries to use it before it's actually defined?

asolovyov08:06:38

I don't declare, should I? Maybe I have to require it somewhere early?

asolovyov08:06:02

I just required it really early and it didn't help 😞

p-himik13:06:29

If you don't use declare then I have no idea, sorry. Maybe you could create a minimal working example and create a gist or something out of it that you could share?

asolovyov18:06:10

I guess I'll try to do that myself and work up to the situation it's not working anymore... It's pretty puzzling 😞

andrea.crotti09:06:10

does anyone know of a way to extract all the list of requires given a ns (without parsing ideally?) I thought ns-map would help but it doesn't atm sadly

andrea.crotti09:06:57

I could parse it with https://github.com/clojure/tools.analyzer maybe but the output doesn't seem so user friendly

andrea.crotti10:06:12

I ended up with some regexps for now, it's probably in this case, but would be nice to know if there a better way

Alex Miller (Clojure team)10:06:32

Have you tried tools.namespace?

andrea.crotti10:06:07

ah nice clojure.tools.namespace.file/read-file-ns-decl gives me the ns declaration at least

Olical12:06:23

I'm doing some weird stuff with it here https://github.com/Olical/conjure/blob/ea4b3090536669fac1f3bd58c2753af634373576/src/conjure/munge/main.clj Reading in all decls, getting their requires then sorting them topologically to get an order to load the files in. Might come in handy!

andrea.crotti10:06:24

so I can use that instead, thanks

temco13:06:33

Is there a way to call python in clojure repl? Directly or indirectly?

tavistock13:06:33

I havent tested it but https://clojuredocs.org/clojure.java.shell/sh should work, im sure there is a better way though

temco02:06:16

I found when using sh to "python" "xx.py", the full path of python must be provided, otherwise any third-party imported lib in xx.py couldn't be found.

tavistock13:06:31

i had trouble with stuff like this, it might because the path is getting mucked with due to this https://clojurians.slack.com/archives/C03S1KBA2/p1560950692262600 or because it doesnt use the same path as your typical shell

temco13:06:28

thanks, by bashing and then read the corresponding file to get the response

tavistock13:06:52

remember that :env can be wonky, if you override it you might need to get the current env and add to it instead of just setting it

mpenet13:06:15

there's also a lib using libpython that was mentioned a few days ago

temco13:06:13

and how can I wait for some asynchronous response from the outside program if it costs much time?

temco13:06:31

just wait and check the flag?

temco13:06:50

it seems interesting

temco13:06:27

my purpose is to call tensorflow-based python program and get the response in my clojure process, maybe libpython-clj is not enough, a http server might be ok

temco13:06:24

thanks for your help

Yehonathan Sharvit14:06:27

I’d like to deploy a library to clojars but only the files under the src folder. How can I do that?

lukasz14:06:09

@viebel by default only src and resources are included I think (if you're using Lein). You can rename resources dir to dev-resources and update your project.clj to add that directory to the classpath in dev profile. Is that what you're after ?

NoahTheDuke14:06:56

hey all, is there a way to get a Figwheel-style repl for clojure? One with auto-complete, colors, etc? The lein one is a little lacking

NoahTheDuke14:06:34

oh nice, I'll look into that!

Jakub Holý (HolyJak)16:06:36

Hi, how to parallelize I/O intensive operations (remote calls) while controlling the amount of parallelism? (other than core.async). Shouldn't https://clojure.github.io/clojure/branch-master/clojure.core-api.html#clojure.core.reducers/fold be able to do just that? (r/fold N-in-parallel r/cat r/append! (r/map remote-call (vec my-data))) ?

Jakub Holý (HolyJak)16:06:35

This is what I tried

(let [maxp (atom 0)
        cnt (atom 0)]
    (add-watch cnt :x (fn [_ _ _ current] (swap! maxp max current)))
    (->> (vec (range 100))
        (r/map #(do
                  (swap! cnt inc)
                  (try
                    (Thread/sleep 10) %
                    (finally (swap! cnt dec)))))
        (r/fold r/cat r/append!))
    (str "Done, max parallelism: " @maxp))
and expected it to print "parallelism" > 1 (ie around 100/10) but it does not. Why?

slipset16:06:49

The n the partition size

slipset16:06:47

As far as I can see from your code, you have not specified n, so it defaults to 512, and since your vector is 100 long, it’s all done in one partition

slipset16:06:18

What happens if you specify n to be 10?

Jakub Holý (HolyJak)16:06:33

OMG how could have I forgotten to put N in? Thanks! With n=1 I get max parall. = 4.

Jakub Holý (HolyJak)16:06:40

With nr items 1000, op duration 100ms and partition size 1 I still only get parallelism 4, expected much higher.

reborg16:06:01

it’s fork-join, you get the number of cores you have

Jakub Holý (HolyJak)16:06:03

(with pmap I get parallelism 37)

Jakub Holý (HolyJak)16:06:53

Thanks, @U054W022G! So there is no way to tell clojure to "do N operations in ||" other than using core.async or manual partitioning and using future on each partition, perhaps with a custom pool with a particular number threads?

reborg16:06:33

The snippet above doees it, unless you have some specific use case in mind.

reborg16:06:06

it’s not perfect approach (all blocking threads waiting on IO) but it depends on what you’re doing

hiredman16:06:43

I would suggest creating your own executor limited to exactly the number of threads you want to limit the parallelism to, and then schedule all your operations on that executor

hiredman16:06:07

using forkjoin for io stuff is not great

hiredman16:06:24

and pmap is terrible

hiredman16:06:23

executors are simple and easy to use and do exactly what you want. the big downside is by default they can act as infinite queues, which, as long as you have back pressure elsewhere shouldn't be an issue

noisesmith17:06:28

@U054W022G my understanding is that if you are bottlenecked by IO, limiting thread usage to core count isn't actually useful

👍 8
henrik17:06:15

People around here put me on to Claypoole. upmap turned out to be very useful for my particular case. https://github.com/TheClimateCorporation/claypoole

❤️ 8
lukasz18:06:45

Same - we use claypoole in a lot of places and after tuning pool sizes it's been working like a charm for us

Jakub Holý (HolyJak)18:06:14

Though I don't really need many threads if I can get the remote calls to release their threads while they wait for a response...

noisesmith18:06:46

threads are relatively cheap - eg. badly planned async / polling IO can be more costly via added code complexity compared to the overhead of heap usage for more idling threads

noisesmith18:06:01

make sure you are optimizing the right things

noisesmith18:06:53

it seems like I often see people coming from languages / vms where threads are expensive or hard to use applying criteria that don't neccessarily apply for clojure on the jvm

eraserhd18:06:12

Is there no utility that can find cyclic namespace dependencies for you?

vemv18:06:39

tools.namespace's refresh will refuse to do its job if it found cyclic deps behorehand

noisesmith18:06:48

@eraserhd the clojure compiler finds them on initial load :D

noisesmith18:06:42

if you create a set of namespaces that form a cycle, the compiler will error and show you precisely which namespaces were in the cycle

noisesmith18:06:58

a task like lein check could be run as you add deps to namespaces

noisesmith18:06:27

ideally one starts a project with a "big picture" understanding of how namespaces should relate to each other, but of course with a bigger team that gets lax

eraserhd18:06:14

We have stuff partially loaded on first refresh, but the compiler isn't complaining. I think it might be related to Clara rules, which does some magic.

noisesmith18:06:55

you can't trust a refresh - you need a full load from scratch to find cycles

noisesmith18:06:10

maybe I'm misunderstanding you though

eraserhd18:06:35

Currently: Kill repl -> boot system -> no cycles -> system broken -> refresh -> system works.

hiredman18:06:00

how are you refreshing?

hiredman18:06:39

almost certainly what you have is a missing require

eraserhd18:06:48

a small wrapper around clojure.tools.namespace.repl/refresh that prevents losing the reference to the running system

hiredman18:06:39

refresh is pretty good at detecting cycles and complaining about it as well, as long as you have correct dependency information in your ns forms

hiredman18:06:48

which is way I say you are likely missing a require

hiredman18:06:12

so repl/refresh isn't getting correct dependency information when it parses the ns forms

hiredman18:06:50

further, I will guess that you are importing a defrecord class instead of requiring its namespace and using the factory function

noisesmith18:06:05

what are the symptoms of "system broken" above?

eraserhd18:06:52

Our GraphQL endpoint returns zero entries for a kind of object that's created by a Clara rule with a null left-hand-side (meaning it should be created at compile time).

hiredman18:06:20

you are keeping the old system around and reloading so the types don't match

noisesmith18:06:16

deftype / defrecord / defprotocol are not idempotent - calling twice creates two distinct classes with the same name and package

eraserhd18:06:28

not actually - I misspoke. That's the "refresh" wrapper, but we are using the "restart" wrapper in this case, which does not keep the running system.

hiredman18:06:17

are you aot compiling ever?

hiredman18:06:55

are you requiring the namespace that defines your defrecords every place you construct them?

hiredman18:06:07

(or refer to them)

andy.fingerhut18:06:35

I do not know if it will find anything regarding ns forms or dependencies that you are not already seeing with the suggestion above, but if your project is Clojure/JVM you can try running Eastwood on it: https://github.com/jonase/eastwood

andy.fingerhut18:06:37

One thing it does that a lot of other tools do not is find mismatches between namespace names and the file name they are defined within, which causes trouble for some tools.

eraserhd18:06:34

@hiredman hmmm, this could be it. Clara does some magic here, too... and I think I remember something that could have broken this.

hiredman18:06:25

my guess is somehow in the normal order of loading your project, you get a defrecord defined twice, and a clara rule is creating an instance check will the first class type, and then the record is re-def'ed, and at runtime you get instances of a different class with the same name

hiredman18:06:55

and refreshing is causing everything to get reloaded without the double def'ing which makes it work

borkdude19:06:33

when analyzing clojure code, should this be supported?

(defmulti xyz (fn ([x y] x) ([x y z] x)))
(defmethod xyz "z" ([x y] "z") ([x y z] "z"))
so, multi-arity defmulti/defmethod, in this syntax?

borkdude19:06:26

I think this is correct, but correct me if I’m wrong

andy.fingerhut19:06:05

A bit of REPL testing with Clojure 1.10 seems to indicate the implementation is happy with that. Whether it should be supported is above my pay grade 🙂

Alex Miller (Clojure team)20:06:29

defmethod takes a “fn-tail” iirc - basically everything in an fn form except literally the fn symbol (including name iirc)

bronsa22:06:34

yep, including name, which is massively helpful for making readable stack traces for multimethods

👌 4
andy.fingerhut23:06:39

I know a few people have created libraries/tools for formatting/streamlining//etc. Java stack traces. I also know that sometimes these tools remove stuff that makes it more difficult to debug things in some situations, so much so that Stuart Sierra created a Clojure lib that actively tries to disable hooking/patching of such things: https://github.com/stuartsierra/stacktrace.raw

andy.fingerhut23:06:55

Has anyone written up documentation or articles on reading and interpreting Java stack traces, the raw ones, targeted at Clojure developers. I say targeted at Clojure developers, because there are definitely some Clojure-specific tips that would be helpful to know there, including the kind of thing bronsa just mentioned above.