Fork me on GitHub
#clojure
<
2020-09-04
>
emccue03:09:17

What is the equivalent of provided dependencies in leiningen?

seancorfield03:09:43

@emccue I'm pretty sure you can just say :scope "provided" inside a dependency for Leiningen since it generates the pom.xml and preserves that as I recall. On the other hand, there's no equivalent in deps.edn (but you can of course add such dependencies into a pom.xml file, generated via clojure -Spom).

seancorfield03:09:09

Is that answering the question you asked, or did I misunderstand?

dominicm07:09:01

You can do the above, but it's discouraged. In lein you should create a :provided profile.

dominicm07:09:31

https://cljdoc.org/d/leiningen/leiningen/2.9.3/doc/profiles#default-profiles mentioned here, but I can't find the document with rationale. Maybe because profiles can be toggled.

stephenmhopper13:09:59

I make heavy use of ->> when operating over a collection of data. I’m looking at something I wrote that has 4 remove steps, 1 filter step, followed by a map and finally sort-by. Assuming I’m using this code to process a sufficiently large set of data, what benefits (if any) would I see by switching to transducers here? My understanding is that a new data structure is being created at each of those steps when thread last is used, but with transducers, it’d just be one structure?

p-himik13:09:13

There won't be any intermediary lazy collections, yes.

👍 3
Alex Miller (Clojure team)13:09:42

the number of elements is important too

Alex Miller (Clojure team)13:09:00

transducers win as you increase both steps and seq size

Alex Miller (Clojure team)13:09:40

if you're talking a seq of 10 elements or something, it's unlikely to matter

Alex Miller (Clojure team)13:09:57

if you have 1000 it probably will

gekkostate13:09:05

So, in that case should we just default to transducers then?

dpsutton13:09:29

and a great phrase is "why guess when you can measure".

Alex Miller (Clojure team)13:09:58

a lot of times, the seq version is closer at hand and I know enough about the expected size that I still use seqs for a lot of stuff

Alex Miller (Clojure team)13:09:54

from having done a lot of perf comparison of the two approaches, chunked sequences that you get with filter/map/remove are surprisingly efficient

stephenmhopper13:09:09

So for something like this:

(->> coll
  (group-by :key)
  (map mapping-fn)
  (into {}))
The transducer equivalent would be:
(into {} (map mapping-fn) (group-by :key coll))
Assuming a sufficiently large coll and a sufficiently complex mapping-fn is there any real benefit of converting this to a transducer?

stephenmhopper13:09:29

I can certainly see the benefit of converting my initial example (the one with 4 filters, a remove, and a map step) into a transducer, but would I see benefits for relatively simple examples like this?

mpenet13:09:25

Depends on input size mostly. But in doubt, measure, they say

mpenet13:09:13

also the into version will be more gc friendly likely

Alex Miller (Clojure team)13:09:10

more accurately, it would not create g to c

Alex Miller (Clojure team)13:09:27

depends entirely on size and type of coll (whether it's something self-reducible). group-by, by it's nature, has to see all the input to do the groupings, so it's not really helped that much by being a transducer

Alex Miller (Clojure team)13:09:56

not knowing anything else, I'd probably just write that in the seq form myself

stephenmhopper13:09:45

Okay, this has been very helpful, thank you @alexmiller!

Adrian Smith14:09:22

Just wanted to draw attention to this https://github.com/codesandbox/codesandbox-client/issues/704 in case any one has the energy for it

wombawomba14:09:05

What’s an easy way to parse a file of newline-separated EDN dicts, where each individual dict may in turn contain newlines?

manutter5114:09:56

wrap it in [] and parse as an array of EDN dicts?

isak14:09:04

Or if it is too big, call https://clojuredocs.org/clojure.edn/read , then read a line, then read again

wombawomba14:09:59

Good stuff, thanks! 🙂

Matthew Cheney15:09:54

Does anyone know how to use ring.util.response/partial-content to stream a video file starting partway through (to allow seeking to unloaded parts of the video)? I've been using ring.util.response/file-response .

hiredman16:09:52

it has been a long time since I've messaged with streaming video, but I believe that partial seeking behavior is a property of the video file itself, I vaguely recall having to add hints to mp4 files

hlolli15:09:30

I need to convert a double to float, sounds easy, but I got surprised here

user> (type 3.0861591259684055E172)
java.lang.Double
user> (float 3.0861591259684055E172)
Execution error (IllegalArgumentException) at user/eval6250 (REPL:15).
Value out of range for float: 3.0861591259684055E172
is there some .toFloat or any alternatives to truncate the double to float?

Conor15:09:18

It seems to me that you would never be able to convert such a large number to a float, as it's greater than the maximum value a float can have (~3.4 x 10^38)

p-himik15:09:15

Just to make it more obvious - @U0CAUAKCG, there's E in that number, which makes it quite huge.

Matthew Cheney15:09:46

You could check if the value you're converting is greater than the float limit, and then return the float limit if true.

(defn to-float [x]
  (if (> x Float/MAX_VALUE) Float/MAX_VALUE
    (if (< x Float/MIN_VALUE) Float/MIN_VALUE
      (float x))))

hlolli15:09:08

ahhh i see, good idea. I'm processing double float audio buffers from one jna to another, so a performance is also a factor. I'll check

hlolli15:09:06

thanks @U016UGGNXB3 this works! It's a good safety net, otoh, just realised that this is a positive exponent not negative, so this is a huge number, which does not fit in the context of normalized audio signal from -1 to 1. So I'm reading wrong memory 😛

👍 3
isak16:09:23

Anyone know why Rich declined this patch? (Improvements to cond) https://clojure.atlassian.net/browse/CLJ-200 Rationale: https://github.com/Engelberg/better-cond#rationale

seancorfield16:09:19

I looked through the Google Groups archive and didn't see any responses from any of the core team on any thread where this has come up -- and it's come up in various forms half a dozen times since CLJ-200 originated (11 years ago).

seancorfield16:09:44

I wonder if @alexmiller has any insight into this? (He commented last on that ticket, after it had been closed)

Alex Miller (Clojure team)16:09:16

I think the general comment is that we don't agree those things are necessarily better or needed

Alex Miller (Clojure team)16:09:33

but if you want them, then use better-cond

Alex Miller (Clojure team)16:09:06

for me personally, I'd rather have less things in cond :)

seancorfield17:09:31

I have commented a few times on similar proposals that I just don't find better-cond to actually be better -- I find it harder to read, but then I find doseq and for harder to read when they're full of :let / :when etc 🙂

Alex Miller (Clojure team)17:09:38

less is more in this area for me

isak17:09:51

interesting, thanks 👍:skin-tone-2:

hlolli17:09:18

JNA: callback object has been garbage collected I seem to keep hitting this problem, is there any clojure trick for strong-reference? My callback object can be local scoped, local scoped and refered to by multiple sources, and ns scoped (top level) atom, always within few seconds, my callback pointer is gc'd.

hlolli18:09:15

this would be the function I use to make the callback

(defn set-process-callback
  [^JackLibrary$_jack_client jack-client callback]
  (let [cb ^JackLibrary$JackProcessCallback
        (reify JackLibrary$JackProcessCallback
          (^int invoke [_ ^int nframes] (callback nframes)))]
    (.jack_set_process_callback ^JackLibraryDirect jack-lib jack-client cb nil)))
and this could be the reference top-level
(def audio-callback (atom nil))

jsa-aerial18:09:42

You might want to look into how this sort of issue is handled in libpython-clj: https://github.com/clj-python/libpython-clj

hlolli18:09:54

thanks a lot! this is perfect, thirsty for any suggestions!

phronmophobic18:09:52

you can create a strong reference by keeping a reference to it. The easiest way might be something like:

(def callback-refs (atom #{}))
(defn set-process-callback
  [^JackLibrary$_jack_client jack-client callback]
  (let [cb ^JackLibrary$JackProcessCallback
        (reify JackLibrary$JackProcessCallback
          (^int invoke [_ ^int nframes] (callback nframes)))]
    (swap! callback-refs conj cb )
    (.jack_set_process_callback ^JackLibraryDirect jack-lib jack-client cb nil)))

hlolli18:09:03

@U7RJTCH6J that I already tried, at least with the toplevel atom. I would consider that strong too. So I'm equally confused. But afaik from reading, java considers anything not "final" a soft reference, and will gc all of them if it needs memory.

hlolli18:09:48

this is usually fine, but with native pointers, this causes big problems

hlolli18:09:09

I was reading this here, https://blog.shiftleft.io/understanding-jvm-soft-references-for-great-good-and-building-a-cache-244a4f7bb85d also looking at cpython's bindings, entirely written in java, it seems I need to get java on this one

phronmophobic18:09:21

I use jna and a top level atom has worked every time

phronmophobic18:09:13

> But afaik from reading, java considers anything not "final" a soft reference, and will gc all of them if it needs memory. I think that only applies to java code. the garbage collector will not collect objects referenced in top level atoms or else that would break a ton of clojure code

hlolli18:09:06

I'm making another test with top-level atom. I'm not an jvm expert, but could it be that the jvm pointer can be thrown around while the native ones are constant? That could explain wild gc's behavior, but this is way above my knowledge on jvm

phronmophobic18:09:36

what's JackLibrary?

hlolli18:09:36

I'm only using the lowlevel/* for now, some functions there which I need which haven't been wrapped, also don't like OOP style wrapping anyway 😛

phronmophobic18:09:53

yea, I might just use jna directly

phronmophobic18:09:08

I think libpython-clj has their own associated jna wrapper

hlolli18:09:51

so it happened again

(def callbacks (atom #{}))

;;under let
        audio-callback (csound-process
                        csnd
                        jack-ports-in
                        jack-ports-out
                        ksmps-rate)
        on-registration (fn [_] (reset! status :running))
        on-unregistration (fn [_]
                            (stop @csnd)
                            (cleanup @csnd))
        client-reg-cb (jack/make-client-registration-callback
                       {:on-registration on-registration
                        :on-unregistration on-unregistration})
       
        ]
    (apply swap! callbacks conj [audio-callback on-registration on-unregistration client-reg-cb])
just few seconds in

hlolli18:09:45

I think this is the audio-callback mainly, because it wont get called, just dies ....

phronmophobic18:09:04

the other thing that's easy to mess up is passing the wrong type. jna will automatically convert types, but if you pass 42 , that will be long when it might be expecting a float or an int

hlolli18:09:53

I'm putting my life on (set! *warn-on-reflection* true) for that 🙂 hope it's enough

hlolli18:09:12

but you mean jna may be operating differently?

phronmophobic18:09:17

that is not enough

phronmophobic18:09:32

I don't think, anyway

hlolli18:09:38

good to know, I'll try being totally explicit

phronmophobic18:09:11

I've been using jna directly, which will definitely crash when passed the wrong number type. I'm not sure how it works if you're using jna through some java stuff

hlolli18:09:42

I believe I found the problem in the end!

hlolli18:09:30

now this is local scoped

audio-callback (reify JackLibrary$JackProcessCallback
                         (^int invoke [_ ^int nframes]
                          (loop [status (perform-ksmps @csnd)
and I store audio-callback in top-level atom beofre I had a wrapper function around the reify
(defn set-process-callback
  [^JackLibrary$_jack_client jack-client callback]
  (let [cb ^JackLibrary$JackProcessCallback
        (reify JackLibrary$JackProcessCallback
          (^int invoke [_ ^int nframes] (callback nframes)))]
    (.jack_set_process_callback ^JackLibraryDirect jack-lib jack-client cb nil)))
somehow someway, it causes gc :woman-shrugging:

emccue18:09:03

@U0CAUAKCG real quick and dirty solution

emccue18:09:21

Just memoize that function

emccue18:09:49

(def set-process-callback (memoize set-process-callback))

emccue19:09:08

If there is no way for callbacks to be removed you don't have to worry about the semantics of repeated calls

hlolli19:09:52

that sounds like the kind of answer I was most looking for! thanks for chipping in!

devn23:09:27

I have a bit of an annoying transform that I would appreciate some help with. I built a nested reduce, and it mostly works, but it feels like there’s got to be a cleaner way. Input and desired output is as follows:

phronmophobic23:09:07

I've started using clojure.zip for similar treewalking examples

devn23:09:10

i would like to avoid using zippers here

phronmophobic23:09:55

what the nested reduce look like?

hiredman23:09:33

(apply merge-with into
       (for [a [{:x  {:id "123"}
                 :ys [{:type "Bar"
                       :y    "42"}
                      {:type "Foo"}
                      {:type "Bar"
                       :y    "43"}]},
                {:x  {:id "456"}
                 :ys [{:type "Bar"
                       :y    "42"}
                      {:type "Foo"}
                      {:type "Bar"
                       :y    "43"}]}
                {:x  {:id "789"}
                 :ys [{:type "Bar"
                       :y    "99"}]}]
             b (:ys a)
             :when (:y b)]
         {(:y b) [(:id (:x a))]}))

ghadi23:09:07

hiredman, Comprehender of the List Comprehension

👍 3
devn23:09:57

the nested reduce looked like shit

devn23:09:28

i was sort of kind of close before i switched to trying the nested reduce, but I missed the apply merge-with into