Fork me on GitHub
#clojure
<
2020-01-30
>
kenny00:01:31

I'm running into some very strange behavior with manifold. I have two functions:

(defn example1
  []
  (deferred/loop []
    (deferred/let-flow
      []
      "a")))

(defn example2
  []
  (deferred/loop []
    "a"))
If I run @(example1), the thread will be halted forever. If I call @(example2), I get a result back. Anyone familiar with manifold have any ideas here?

noisesmith00:01:12

@kenny I think this is a clue

(defmacro let-flow
  "A version of `let` where deferred values that are let-bound or closed over can be treated
   as if they are realized values.  The body will only be executed once all of the let-bound
   values, even ones only used for side effects, have been computed.
   Returns a deferred value, representing the value returned by the body.

kenny00:01:08

Oh interesting. You thinking the relevant part is "or closed over"?

noisesmith00:01:16

but intuitively with no bindings it should be a no-op

kenny00:01:29

That's what I'd think too. The deferred documentation says "the combination of `loop` and `let-flow` can allow for the specification of highly intricate asynchronous workflows". That makes me think my code is valid.

noisesmith00:01:41

this is what a dump of all stacks (via Ctrl-\ in the terminal) shows me when @(example2) is blocking

"main" #1 prio=5 os_prio=31 tid=0x00007f9bb2802000 nid=0x2603 waiting on condition [0x0000700004bb7000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000076cd6a238> (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
	at manifold.deferred.Deferred.deref(deferred.clj:441)
	at clojure.core$deref.invokeStatic(core.clj:2320)
	at clojure.core$deref.invoke(core.clj:2306)
	at user$eval1729.invokeStatic(NO_SOURCE_FILE:0)

noisesmith00:01:05

what could it be waiting on? that seems like a bug to me, or a pathological usage of let-flow

kenny00:01:08

Hmm, yeah. This does not block:

(defn example4
  []
  (deferred/loop []
    (deferred/let-flow
      [a (deferred/future "a")]
      a)))

kenny00:01:48

I actually hit this case in a more complex example than the one I posted. I just worked it down to that simple example. I could probably come up with one closer to my issue.

noisesmith00:01:17

so this is either a bug in deref-deferred, a bug in let-flow, or a pathological usage on your part that was never intended to work

csm01:01:05

I have had lots of trouble getting let-flow to work; I eventually gave up on it and wrote my own version that’s way simpler than what it tries to do

kenny01:01:33

I see. In this case I don't care about running the calls in parallel. We do have several cases where running the calls in parallel is a big perf boost.

csm01:01:26

yeah, I didn’t dig into what the problem was; I just assumed let-flow was too brittle, or I was being stupid in some non-obvious way, and moved on

kenny01:01:33

Ok, here's a boiled down case that I'm hitting. The below blocks forever

(defn- example10
  [request-map]
  (deferred/loop
    []
    (deferred/let-flow
      [resp (http2/request request-map)]
      resp)))

@(example10
   {:method            :get
    :url               ""
    :request-timeout   1000
    :throw-exceptions? false})

kenny01:01:19

Where http2 is aleph.http

kenny01:01:35

Created an issue here https://github.com/ztellman/manifold/issues/187 on this specific issue. I really enjoy working with Manifold but these issues make it tough to justify 😩 Guessing it may be best to switch this code to core.async 🙂

noisesmith01:01:42

core.async has its own issues (eg. all io should be in async/thread and blocking in go blocks can lead to unexpected stalls or very low throughput under load)

noisesmith01:01:09

not bugs per se but counterintuitive best practices

emccue02:01:42

I'm running into some pretty same-y code parsing some simple xml

emccue02:01:47

(defn simplify-declaration-xml
  "Takes the verified XML specifying the node declarations for the program
  and turns it into a more \"tame\" form"
  [parsed-declarations]
  (let [simplify-unique-field (fn [unique-field]
                                {:name (->> unique-field
                                            (:content)
                                            (filter #(= (:tag %) :name))
                                            (first)
                                            (:content)
                                            (first))
                                 :kind (->> unique-field
                                            (:content)
                                            (filter #(= (:tag %) :kind))
                                            (first)
                                            (:content)
                                            (first))})
        simplify-node (fn [node]
                        {:name (->> node
                                    (:content)
                                    (filter #(= (:tag %) :name))
                                    (first)
                                    (:content)
                                    (first))
                         :unique-fields
                               (->> node
                                    (:content)
                                    (filter #(= (:tag %) :unique-fields))
                                    (first)
                                    (:content)
                                    (mapv simplify-unique-field))})]
    (->> parsed-declarations
        (:content)
        (first)
        (:content)
        (mapv simplify-node))))

emccue02:01:03

but I can't seem to find a library that would simplify this that isnt marked as immature/experimental by the author

emccue02:01:13

I wrote an accompanying XSD if that helps any

alfredx05:01:16

Question on nrepl, The default mode seems to start a server that listens on a port number. However, network admin might not like this idea (in an cloud environment) for security reason. So, is it feasible to have the nrepl server to talk to an external host rather than opening a port? (Basically, I mean because network admins are happier if traffic is outbound rather than inbound, can we still have a remote repl without opening a port?)

sogaiu05:01:18

ssh tunneling?

hiredman05:01:21

Short answer is no

hiredman05:01:10

The long answer is nrepl (at least the last time I looked at it) is defined as an exchange of messages over any transport, so you can write your own transport to do whatever, but it can be tricky to get other tooling to work with custom transports

hiredman05:01:37

You could also look at existing alternative transports like drawbridge

alfredx05:01:18

Yah, understand that it is tricky. But isn’t this a common use case? Don’t other people have similar need to be able to repl in cloud remote environments?

vemv05:01:00

SSHing into the instance and launching a repl there (as opposed to connecting to nREPL from your computer via an exposed port) seems more secure

vemv05:01:09

Anyway I try to keep production repl interaction at a minimum. e.g. for patching data, better to create a "task" to be deployed and run. It can be QAd, code-reviewed etc

jysandy05:01:38

At my workplace nrepl doesn’t listen on 0.0.0.0, so the port isn’t exposed. We connect to it via SSH tunnelling.

👍 4
alfredx05:01:33

@U45T93RA6 I’ve done it before. Just the repl is not connected to the running Clojure application instance, right?

alfredx05:01:30

@U38004EG7 that’s cool, sounds more acceptable

vemv06:01:03

> Just the repl is not connected to the running Clojure application instance, right? I've seen that recommended, yes

seancorfield06:01:49

FWIW, we run a bare Socket REPL in a number of our production processes. We specify a JVM option when we start the process, which includes the port number, and it listens on localhost only, then we use SSH tunneling to enable connection from a local client (not nREPL).

👍 8
seancorfield06:01:01

That means no dependencies in production, no code needed in our processes -- the Socket REPL is built into Clojure itself, we control whether any given process starts a REPL via JVM options, and then we can use the exact same client setup that we use for local dev, since we use Socket REPLs locally for dev too.

alfredx08:01:45

That’s pretty cool, many thanks @U04V70XH6

lukasz14:01:27

We have a somewhat similar setup: each app runs local nreplserver, we SSH to the machine (no tunnels, requires going through a jump host) and launch an nrepl client via docker (more or less: docker run --rm -it --net host repl-client :connect 127.0.1:357381 )

sogaiu05:01:27

may be the following might address this use case at some point: https://github.com/thheller/shadow-cljs/blob/master/doc/remote.md

roklenarcic16:01:17

is there something like cond-> but where I can use the value of the expression being threaded in the test expression (conditional)?

borkdude16:01:55

@roklenarcic isn't that already the case?

user=> (cond-> {:a 0} :a (assoc :b 1) :b (assoc :c 2))
{:a 0, :b 1, :c 2}

borkdude16:01:36

oh wait... sorry, I'm using keywords as constants here maybe

roklenarcic16:01:48

they are truthy values here

Ramon Rios16:01:50

Do we have any channel to discuss about books?

Ramon Rios16:01:18

I would like to know wich clojure books would it good to be.

borkdude16:01:45

afaik there isn't a dedicated channel for that but anything goes in #off-topic

borkdude16:01:37

I guess it's a roll your own kinda thing. There's also a better-cond library which may or may not do this.

Balaji Sivaramgari17:01:56

Hello Team, Hope you are doing well. Here is the error I am getting during one of the builds in Jenkins pipeline

Balaji Sivaramgari17:01:56

Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class taoensso.timbre__init, compiling:(/tmp/form-init8075592954089703367.clj:1:73) Caused by: java.lang.NoClassDefFoundError: Could not initialize class taoensso.timbre__init 17:32:07 at clojure.lang.Compiler.load(Compiler.java:7514) 17:32:07 ... 12 more 17:32:07 Compilation failed: Subprocess failed

Balaji Sivaramgari17:01:25

Here is my project.clj file. could you please help me here?

Balaji Sivaramgari17:01:26

defproject XXXXX/XXXX "0.7.37"   :description "Transaction Registration and Tracking"   :url "https://github.com/XXXXX/RepoName"   :plugins [[lein-sub "0.3.0"]             [lein-tar "3.3.0"]             [lein-set-version "0.4.1"]             [lein-ancient "0.6.10"]             [lein-cljfmt "0.5.6"]             [test2junit "1.4.2"]             [jonase/eastwood "0.3.5"]             [lein-kibit "0.1.6"]             [lein-ancient "0.6.15"]             [lein-cloverage "1.0.10"]              [lein-nvd "1.0.0"]]   :sub ["types" "service" "client"]   :dependencies [])

borkdude17:01:12

@balaji.sivaramgari I sometimes get similar error messages when I've AOT-ed something. Running lein clean usually solves that for me

Balaji Sivaramgari17:01:54

@borkdude I will try lein clean and the lein sub do cloverage. Is that your suggestion?

borkdude17:01:26

just lein clean before whatever else you do which trips this error

Balaji Sivaramgari17:01:40

sure, I will try that and let you know my test results.

Balaji Sivaramgari17:01:28

@borkdude, still I am getting the same error

Balaji Sivaramgari17:01:04

sh "lein clean; lein sub do cloverage"

Balaji Sivaramgari17:01:05

this is the command I am running

borkdude17:01:11

sorry, don't know then

Balaji Sivaramgari17:01:35

NP, thanks for your response. 🙂

hiredman17:01:41

I have seen similar timbre errors result from using aot libraries

hiredman17:01:50

that almost certainly isn't your entire project.clj

hiredman17:01:38

I say this because I don't know for sure, but would be very surprised at any of those pulling in timbre

hiredman17:01:06

if it works locally you should check to see what is different about you local environment (often in ~/.lein/profiles.clj)

Balaji Sivaramgari17:01:31

@hiredman so, do we need to add this "[com.taoensso/timbre "4.10.0"]" to ~/.lein/profiles.clj file?

hiredman17:01:52

you should look at your code, look at the stacktrace to try and figure out what is trying to load timbre, figure out why it is trying to load timbre, and if it is a good reason then you need a dependency on timbre (in the project.clj) and if it is for a bad reason you should figure out how to stop it

hiredman17:01:42

I think what may be going on is you have a lot of plugins, and not all of them interoperate well

hiredman17:01:16

so likely one subproject depends on timbre, but one of your plugins doesn't understand subprojects

hiredman17:01:47

so it is using only the dependencies in your toplevel project.clj while trying to load that subprojects code

Balaji Sivaramgari18:01:21

@hiredman, I tried to add the [com.taoensso/timbre "4.10.0"] in dependencies section of project.clj as below. :dependencies [[com.taoensso/timbre "4.10.0"]]) but still getting the same error

seancorfield18:01:38

@balaji.sivaramgari Each of your three subprojects have a project.clj as well, yes? I think your error is coming from one of those...

seancorfield18:01:28

I confirmed that none of those plugins listed brings in Timbre (`lein deps :plugin-tree`)

Balaji Sivaramgari05:01:20

@U04V70XH6, so do we need to mention the ":dependencies [[com.taoensso/timbre "4.10.0"]])" in all project.clj files of sub projects?

Balaji Sivaramgari05:01:53

yes, we have couple of project.clj files.

seancorfield05:01:24

You need to share more information if you want anyone to help you debug this.

Ramon Rios18:01:36

Folks, i got lost on reduce sometimes

(defn add-attachments [attachments]
  (when (some? attachments)
    (reduce
     (fn [attachments {:keys [filename url]}]
       (conj {:type "attachment" :content (.File (get-template filename))}))
     []
     attachments)))
My goal here is to add maps on a vector. Can soemone help me with it?

p-himik18:01:27

You're calling conj with a single argument. It should probably be (conj attachments {...}).

Ramon Rios18:01:31

(defn add-attachments [attachments]
  (when (some? attachments)
    (reduce
     (fn [attachments {:keys [filename url]} new]
       (conj new {:type "attachment" :content (.File (get-template filename))}))
     attachments
     [])))

Ramon Rios18:01:00

[{:filename "name.pdf" :url
0m "url"}]
Still not what i wanted, but it's a advance

noisesmith18:01:39

the first arg to the reducing function (here you call it "attachments" which is confusing) is the accumulator

jumpnbrownweasel18:01:44

The fn you pass to reduce has two params, but now you have three. Take a look at the doc for reduce where it describes that function.

Ramon Rios18:01:09

(defn add-attachments [attachments]
  (when (some? attachments)
    (reduce
     (fn [{:keys [filename url]} new]
       (conj new {:type "attachment" :content (.File (get-template filename))}))
     attachments
     [])))

noisesmith18:01:39

you are still trying to get "filename" from the first arg, which is wrong - the first arg is the accumulator

noisesmith18:01:49

the second arg is the individual item

noisesmith18:01:32

in fact your code should be fixed by just switching the order of the arg list (though the name "new" is confusing here)

p-himik18:01:39

(defn add-attachments [attachments]
  (reduce (fn [acc attachment]
            (conj acc {:type    "attachment"
                       :content (-> attachment :filename get-template .File.)}))
          [] attachments))

noisesmith18:01:59

no need to apologize, just trying to help you understand

p-himik18:01:01

That's how I would do it.

p-himik18:01:19

You were also missing a point after .File since you need to call a constructor.

jumpnbrownweasel18:01:04

I admit the doc for reduce may be difficult to understand. It always helps to look at examples, see: https://clojuredocs.org/clojure.core/reduce

p-himik18:01:14

BTW you don't even need reduce:

(defn add-attachments [attachments]
  (mapv (fn [attachment]
          {:type    "attachment"
           :content (-> attachment :filename get-template .File.)})
        attachments))

Ramon Rios18:01:58

my mind is blowing in a good way

Ramon Rios18:01:48

Yeah, reduce it's a little bit tricky to understand at the beginning. I appreciate your help folks

chrisulloa22:01:00

Curious to know if anyone is using tap> for anything besides debugging?

didibus07:01:26

I used it to pretty-print, that's all 😛

hiredman23:01:07

like outside of tooling?

hiredman23:01:54

something like rebl is really kind of a repl explorer or something, not really a debugger

chrisulloa23:01:17

ah makes sense, haven’t seen much use of it i was just curious of where it’s being used

hiredman23:01:06

my impression is it isn't really suitable for use outside of tooling. it was introduced together with prepl which is a kind of special purpose repl for tools, but I don't know that I have seen specific guidance. I haven't used it much myself, but when I have my impression was it is best viewed as something similar to a logging statement

hiredman23:01:44

when you log something you want to record that some event happened usually as text for possible later viewing, with tap> you want to send some object to some tool for inspecting

chrisulloa23:01:54

yeah, when 1.10 came out I saw a few blog posts about using it to store intermediate values or log messages. putting values in it in the middle of some complicated code to inspect later for example.

hiredman23:01:09

if you haven't seen any of the rebl talks/demos they are pretty neat, but discussion of tap> to some degree takes a back seat to datafy/nav, but they really kind of work together

chrisulloa23:01:28

that makes sense

chrisulloa23:01:03

still need to checkout rebl

seancorfield23:01:00

@christian.gonzalez It's worth noting that if you tap> a lot of data, it may drop some -- it has a limit of how many items it can process at any time. I think it's 1,024.

👀 8
didibus07:01:26

I used it to pretty-print, that's all 😛