Fork me on GitHub
#clojure
<
2018-10-18
>
worlds-endless04:10:11

I'm trying to interop to this function: https://github.com/stanfordnlp/CoreNLP/blob/eb43d5d9150de97f8061fa06b838f1d021586789/src/edu/stanford/nlp/parser/shiftreduce/ShiftReduceParser.java#L473 ; cider sees it and autocompletes/explains the signature, but I get the following:

chrisulloa04:10:22

Have you checked to see if the arity and types of the args are correct?

worlds-endless04:10:22

As you can see from the github line, it's public static ShiftReduceParser loadModel(String path, String... extraFlags) {. That should match my attempt to call it

worlds-endless04:10:22

In the java demo code, it's called with just one arg (a string). But it's not liking me

chrisulloa04:10:46

I wonder if it's the extra flags varargs complaining

worlds-endless04:10:19

I wondered that, too, but can't find any reference to that causing problems elsewhere

seancorfield04:10:44

Java varargs become an array of items. You can't omit it.

worlds-endless04:10:55

The (java) democode omits it with lines like ShiftReduceParser model = ShiftReduceParser.loadModel(modelPath);. But I also tried giving it ["somestring"] etc without any change to the error I received

seancorfield04:10:56

You need a second argument of (into-array String [])

seancorfield04:10:17

Yes, Java can omit. Because it's a Java compiler construct.

seancorfield04:10:29

But if you're calling from other languages, you must pass the array.

worlds-endless04:10:31

that makes me sad inside...

seancorfield04:10:14

Under the hood, the signature is really public static ShiftReduceParser loadModel(String path, String[] extraFlags)

worlds-endless04:10:36

Well, that's my TIL...

👍 4
worlds-endless04:10:16

Thanks a ton, @seancorfield. That fixed the issue very quickly

seancorfield04:10:20

It's always puzzling the first time or two you encounter it...

martinklepsch08:10:30

if there's some/file.clj and some/file.cljc does clojure look at both or does one get preference?

codecitizen08:10:31

So based on https://clojuredocs.org/clojure.core/require `Consider a lib named by the symbol 'x.y.z; it has the root directory <classpath>/x/y/, and its root resource is <classpath>/x/y/z.clj, or <classpath>/x/y/z.cljc if <classpath>/x/y/z.clj does not exist.` It seems like .clj is prioritised.

martinklepsch08:10:00

Thanks! Didn't even think this would be in the docstring haha 😄

codecitizen08:10:26

Surprising how detailed it is on require^^

cpmcdaniel15:10:16

I want to load some external state once and cache it for all threads in my app (feels like a promise). However, multiple threads may be racing to initialize the value and I only want one to do the actual work (I will handle my own retries). Should I just wrap my work in a (locking my-promise ...) or is there a better way to do this?

mpenet15:10:52

@cpmcdaniel sounds like delay

cpmcdaniel15:10:20

not quite (oh, hey Max!)

cpmcdaniel15:10:02

because this thing starts out in a context map that is shared by all threads

cpmcdaniel15:10:19

and I can’t kick off the work right away

cpmcdaniel15:10:43

in this case, it’s because the external state may not be ready yet

cpmcdaniel15:10:32

also, the first thread (the one that “wins”) will parameterize the operation, and I don’t know up front which one it’s going to be - I think that’s primarily why delay would be awkward

cpmcdaniel15:10:25

so, my first thought was to use a ref and dosync, but that doesn’t guarantee that only one thread is executing the block

cpmcdaniel15:10:57

and so I think old-fashioned Java synchronized is the way to go

hoynk15:10:24

My code is using a 3rd party java library that generates lots of logs via SLF4J. I configured timbre to manage it but I don´t know how to disable log when I run the tests (lein test).

cpmcdaniel15:10:35

maybe (promise-chan) from core.async might be useful

cpmcdaniel15:10:20

combined with a single-item channel - first thread to take a non-nil value from the single-item channel gets to do the work

mpenet15:10:36

it's quite similar to a promise yes. Basically a chan that will always deliver the same value.

cpmcdaniel15:10:43

but that just feels like an excuse to use core.async

mpenet15:10:04

feels like there could be an easier solution

cpmcdaniel15:10:24

yeah, that’s why I brought it here

dpsutton15:10:26

what if each thread when it wants this value can invoke a thunk to load it and that thunk (load-resource!) will have local state of whether already invoked. and if so it's a no-op, and only the first invocation will cause it to do any work?

cpmcdaniel15:10:56

so use a fn to wrap the state? When invoked the first time it does the work and returns the state, otherwise it just returns the state.

mpenet15:10:08

feels like a simple:

mpenet15:10:16

(fn [x y]
  (let [p (promise)]
    (do-something-later-with x y
                             #(deliver p %))
    p))

dpsutton15:10:18

yeah so all of your threads can blindly call the load function and the load function is aware of whether to do anthing

Azzurite15:10:56

is there a brotli ring middleware?

dpsutton15:10:26

(let [p (promise)]
  (defn get-stuff []
    (deliver p (get-that-stuff))))

dpsutton15:10:40

looks like multiple calls to deliver are fine. so just make sure there's only one promise

mpenet15:10:01

but wouldn't get-that-stuff potentially be called multiple times with this?

cpmcdaniel15:10:14

yeah, but I don’t want other threads wasting effort

cpmcdaniel15:10:52

oooh, I have a bug

cpmcdaniel15:10:26

start-work needs to be reset to false so other threads don’t do stuff too

dpsutton15:10:33

the promise itself has a countdown latch (let [d (java.util.concurrent.CountDownLatch. 1) so it will only get invoked once

dpsutton15:10:03

clojure.lang.IFn
     (invoke
      [this x]
      (when (and (pos? (.getCount d))
                 (compare-and-set! v d x))
        (.countDown d)
        this))

mpenet15:10:33

yeah I was typing a solution involving compare and set + atom too 😛

cpmcdaniel15:10:01

is that intended behavior? Seems wasteful.

mpenet15:10:32

(let [a (atom ::empty)
      p (promise)]
  (defn f [x]
    (if (compare-and-set! a ::empty ::pending)
      (deliver p (do-stuff x))
      p)))

dpsutton15:10:39

that's strict evaluation. need a macro or a func, not a value

mpenet15:10:53

well you have to deref the ret of the fn, but that should do it

mpenet15:10:48

(let [a (atom ::empty)
      p (promise)]
  (defn f [x]
    (if (compare-and-set! a ::empty ::pending)
      @(deliver p (t x))
      @p)))

cpmcdaniel15:10:03

I’d like to turn that into a (deliver-once x expr)

dpsutton15:10:11

if you look at the source of promise, if you copy it as promise-thunk and replace the x in invoke with f and (f) in the countdown you should get what you wnt

cpmcdaniel15:10:56

This is what I had in mind originally.

dpsutton15:10:26

(let [p (promise-func)]
 (defn load! []
   (deliver p (fn [] (println "working") 3))
   p))
#'breeze.mast.repl/load!
breeze.mast.repl> @(load!)
working
3
breeze.mast.repl> @(load!)
3
breeze.mast.repl> @(load!)
3

dpsutton15:10:20

you can see the source of promise modified here:

(defn promise-func
  []
  (let [d (java.util.concurrent.CountDownLatch. 1)
        v (atom d)]
    (reify 
     clojure.lang.IDeref
       (deref [_] (.await d) @v)
     clojure.lang.IBlockingDeref
       (deref
        [_ timeout-ms timeout-val]
        (if (.await d timeout-ms java.util.concurrent.TimeUnit/MILLISECONDS)
          @v
          timeout-val))  
     clojure.lang.IPending
      (isRealized [this]
       (zero? (.getCount d)))
     clojure.lang.IFn
     (invoke
      [this f]
      (when (and (pos? (.getCount d))
                 (compare-and-set! v d (f)))
        (.countDown d)
        this)))))

dpsutton15:10:34

notice the f and (f) in the invoke section

dpsutton16:10:01

yeah. it's battle tested since that's just the code for promise. and it takes a thunk rather than a value

cpmcdaniel16:10:07

creating my own STM type feels like a right of passage

quoll16:10:24

I have a sort of philosophical question on why a function is a particular way… I like using keep where applicable. It’s effectively (remove nil? (map ...)) but it’s a little more terse. The annoying thing though is that it doesn’t support multiple arities like map does, so I sometimes end up needing the remove/map idiom anyway. I’m wondering if there is a reason why it wasn’t extended to support multiple seqs like map does, or if it’s just a matter of lack of the motivation to write it? This made me also look at the source for map, by way of comparison to the source for keep. Like keep there is an effort to make it as efficient as possible, using chunked seqs (makes sense, since maps are used a lot). However, the longer arity versions just use a lazy-seq with a simple cons. Was this just a matter of code-writing expediency, since these arities are used less frequently? Does complexity of some other operation overtake the benefits of chunking? Or is there some other reason that chunking isn’t used for these?

isak17:10:41

@quoll Not sure about most of your questions, but have you considered just using mapcat, which does support multiple collections?

quoll17:10:51

I use mapcat where appropriate, but I don’t see how it offers functionality in this context? My questions are: - Is there a reason why keep does not have arities supporting multiple seq? (If not, then I might try writing them) - Is there a reason why map only does chunking when processing a single seq? (If not, then I might try updating the arities for multiple seqs to also support chunking)

👍 4
noisesmith17:10:03

mapcat can do everything filter, keep, map etc. do if you construct your return value properly

noisesmith17:10:45

user=> (mapcat (fn [n] (when (even? n) [n])) (range 10))
(0 2 4 6 8)

👍 4
quoll18:10:09

OK, that makes sense. But my original perspective here was to make code more terse (`keep` vs. map) and optimized (chunked vs. unchunked). mapcat chunks its answers, so that does handle part of it, but the seq wrapping of intermediate values before being returned lazily makes me think it would be losing something that map offers?

hiredman18:10:35

I would strongly recommend ignoring chunking, and instead looking at transducers

quoll19:10:03

I’ve used transducers in various circumstances. I’m particularly enamored of the case where individual elements in a stream result in 0..n output elements, and being able to terminate streams early. But I never really felt like I completely understood the benefits of transducers in general. I can, and have, used them just fine. But since things like an applied map or filter are lazy, I never felt like composing transducers for these would have significantly different overhead to just threading through multiple seq processing operations. @hiredman in an effort to drink more Kool-Aid, outside of the cases where transducers offer functionality that’s not easily supported with standard seq processing functions, do you have a link which describes the why of transducers? (as opposed to the numerous pages which show how)

noisesmith19:10:29

I know you didn't ask me, but the main thing is that they move the "common ground" away from the seq abstraction to a more abstract / general space. Which means you can apply them to things that don't match the assumptions of lazy-seqs, and also use them seamlessly on lazy-seqs

noisesmith19:10:24

I don't want to be forced to turn a channel, queue, or socket into a lazy data structure shaped interface just to apply useful functional transformations to it

👍 4
roklenarcic19:10:33

Is it possible to find all namespaces in the classpath? That is, all the namespaces that can be loaded?

hiredman19:10:02

@quoll I assume you are interested in performance, which is why you care about chunking, transducers avoid the intermediate allocations of seqs, and will perform better

roklenarcic19:10:58

thanks I'll take a look

quoll19:10:02

I am. But it’s more an intellectual curiosity. Looking at the map source, there is a lot of effort in chunking… for one case, and none of the others. Is it just historical for that case? Or did the other arities miss out because there were diminishing returns in putting the effort in?

hiredman19:10:46

multi seq map is going to take a significant performance hit upfront(checking for the termination condition is more complex, and the mapped function has to be applied), so the idea of apply chunking to it, making it more complex just to claw back to maybe par, is kind of meh

hiredman19:10:56

also, chunking is gross (it is not an insignificant source of bugs introduced because people expect lazy seqs to be processed one at a time, and they were until chunking was added), so seeing it spread is kind of sad

4
andy.fingerhut19:10:18

The chunking 'boundaries' for different input seqs could be different, too.

noisesmith20:10:12

the good thing about chunking is it forces people who were using lazy-seqs for side effects to find some other abstraction

noisesmith20:10:07

sometimes that's the right replacement - it can vary