hey everone

I was following along on this: https://github.com/clojure/core.async/wiki/Pub-Sub - its neat

But, if I want to have multiple subs on one pub (same topic) - whats the best way to do that?

essentially I want many things to be able to register/listen/subscribe for an event,and when a publish goes off, they get the data and run something, but do not block each other or know about each other in anyway (similar to redis pub/sub) - is core.async the wrong tool for that?

As it is, if I register more than one sub on the pub/topic/chan, I have to do a pub for each sub and they seem to go in order

That is not correct

They do go in order, but a single published message will be sent out to any channel subscribed to whatever topic fn returns

Any/all

Internally the pub creates a mult for each topic and each subscribed channel taps the mult

What am I apparently doing wrong here? One go-loop per sub?

`

(def input-chan (chan))
         (def our-pub (pub input-chan :msg-type))
         (def output-chan (chan))

         (defn fire                                                                                                                                                       
           "Send in a message / payload that we'll publish to one or more listeners."                                                                                     
           [topic m]                                                                                                                                                      
           (>!! input-chan {:msg-type topic :data m}))

         (defn listen                                                                                                                                                     
           "Recv a message / payload that we'll run some function against."                                                                                               
           [topic f]                                                                                                                                                      
           ;; pub topic channel                                                                                                                                           
           (sub our-pub topic output-chan)                                                                                                                                
           (go-loop []                                                                                                                                                    
             (let [{:keys [data]} (<! output-chan)]                                                                                                                      
               (log/info "Received an event on topic: " {:topic topic :data data})                                                                                        
               (f data)                                                                                                                                                   
               (recur))))

         (listen :foo prn)
         (listen :foo (fn [x] (prn "Hello from the second one" x)))

         ;; Called first, it only runs code from the initial listen                                                                                                       
         (fire :foo 44)
         ;; If I call a second time, it then runs the second function 

I want the rest of my app to be able to just import "listen" or "fire" from here, and not have to think about whats going on

I could just do with futures and a col of fs I guess

You are using double bang operations in a go block, big no no

but core.async seemed to come up the most for pub/sub stuff - the behavior is still present with single, sorry, the double was a stupid guess-and-hope attempt

(edited the sample - same behavior)

If you are restarting your repl the executor used by go blocks is likely all jammed up with blocking double bang operations

Aren't

I did a fresh repl reload, same behavior. I'll do some more reading I guess

is the main use case for core.async to solve concurrency issues with code that has a lot of wait/idle time? (such as long db query or remote http request - basically similar use cases to js Promise.all ?)

no, not primarily. it's best for decoupling independent parts of your application with channels (queues) and/or building staged pipelines of work

Any idea what I'm botching in that snippet above?

hiredman is right that you should not use >!! or <!! inside a go block. however another problem is you’re creating two go blocks both pulling from output-chan. (every time you execute listen you’re starting a new go-loop)

unfortunately, I do not have time to read it atm but I can look at it tomorrow

My use case - I have a log scanner that reads lines using RandomAccessFile as lines are added - based on line regex matches, I want to fire off publishes that dont wait for anything at all. Then, I want other parts of the system to be able to sub to those particular line msg-types

how would I "listen" with multiple function callbacks without setting up a go-loop for each? (I want the body of the go-loop to come from a function sent in at sub time)

so I may fire off something like (fire :system-down-line {:date (some-date)}) - then my (potentially many independent and unaware of each other) listeners hears it and does something

if I have to store a collection of fns, I don't see what I'm gaining with channels vs just applying my list of fns via pmap or something in a future

I actually can’t see why your code is behaving the way it is, other than having 2 subs subscribing to the same topic using the same channel is weird and might do something unexpected. what happens if you make it 2 different output chans?

https://rupsshankar.tumblr.com/post/66648884392/demystifying-coreasyncs-tap-and-mult - that suggests maybe mult and tap accomplish what I'm envisioning?

oh thanks @bfabry - that was it

(defn listen                                                                                                                                                     
           "Recv a message / payload that we'll run some function against."                                                                                               
           [topic f]                                                                                                                                                      
           ;; pub topic channel                                                                                                                                           
           (let [output-chan (chan)]                                                                                                                                      
             (sub our-pub topic output-chan)                                                                                                                              
             (go-loop []                                                                                                                                                  
               (let [{:keys [data]} (<! output-chan)]                                                                                                                     
                 (log/info "Received an event on topic: " {:topic topic :data data})                                                                                      
                 (f data)                                                                                                                                                 
                 (recur)))))

yeah. from the description of what you want pub/sub should totally satisfy it

awesome thanks everyone! 🙂

the documentation is not perfectly clear, however the consequence of this https://github.com/clojure/core.async/blob/a25ae2fb44f0bc2828377866443ed638beeed9c9/src/main/clojure/clojure/core/async.clj#L869 I believe is that a channel can’t be subscribed to a topic more than once

ty

I am looking at switching to building with deps, and I wonder: my current build process uses the lein git-version plugin to produce a resources/version.edn file containing info like {:tag "5.5.0", :ref-short "4b3619b1"… } before the main build is done. That file is then slurped by a macro, so that I can use the information during the compilation. Any hints on how to produce this (e.g. run git) with clj? Or should I fall back to doing this in a top-level Makefile before running clj?

@jrychter I usually echo git rev-parse HEAD into a file, then stuff the file into the jar

I also use that: https://github.com/borkdude/sci.web/blob/b774c588a4caa7ed1b3636794ffbb78bd801cbd0/script/build#L6

So, what do people use to build AOT-compiled uberjars with tools.deps? I found https://github.com/tonsky/uberdeps, but it doesn't pre-compile.

https://github.com/clojure/tools.deps.alpha/wiki/Tools has a list of options

Hi all, I would like to compare how multiple versions of the same library perform on a set of tasks. Is there an easy way to load multiple versions of the same package into a project with lein? Is there another, better way to do this? I’m aware that I could run the tests for each version separately, with the differing versions handled with different :profiles, but this doesn’t scale well above a handful of versions.

Hmm. So after going through the build tools: uberdeps, depstar and Pack do not do global AOT, badigeon is more of a library than a tool, and cambada breaks with an error.

When I was running some tests that required :aot :all, I ended up switching to lein. None of the tools-deps options would compile everything. It was kinda unfortunate since everything else we do is in tools-deps.

why not call compile function with root ns symbol before running tests?

IIRC, I still needed to include .clj files in the final artifact.

compile just creates class files, you can then include both them and clj files in an artifact

But I think it would fail unless certain .clj files were on the classpath at runtime. We wanted to bundle without source.

I'm currently developing a (yet unreleased) library with tools deps that needs some aot, if you are intersted in example: https://github.com/cljfx/css/blob/master/release.sh

it has compile.clj script that just aot-compiles namespace I need AOTed into classes folder, and then it gets packed using depstar, since I have "classes" in :src

Well, I think I will hold off on migratoin and stick to building with leiningen for now, until things settle down a bit. It's a big app, and time is very limited.

@jrychter We build all our (uber) jars at work for production deployment with depstar but we do not AOT anything. We don't see the point of using AOT.

I have an open source library that requires AOT on one namespace, that uses deps.edn tooling https://github.com/seancorfield/cfml-interop/blob/master/deps.edn

depstar supports AOT @jrychter , you put an AOT alias with an extra directory, then call compile to do the compilation on the namespace of your choice

That cfml-interop library is packaged with depstar -- see the :build alias in that file I just linked.

Most of the packaging tools do.

Is it a code smell to use not-empty on a string?

"it depends" 😉

we use it a lot in our validations

should prob prefer clojure.string/blank?

anecdotally, I see people do it a lot

not-empty will coerce a string to a sequence which seems like it's doing work it shouldn't

We use it for situations where we want either a non-empty/non-blank string or nil in a threaded expression so clojure.string/blank? doesn't work there.

(some-> s str/trim not-empty) for example

given that I almost always use blank? with not, I kind of wish there was an optimized not-blank?

Funny. I used (complement str/blank?) yesterday

@alexmiller absolutely agree, a quick look in a couple of our codebases reveals some kind of implementation of not-blank

I wouldn't just not blank? either, the impl can exit earlier in this case

maybe that's no different, but anyhow

Hi all, I have a project that depends on https://github.com/mikera/clisk and I am having problems compiling the project with this library. When running the uberjar I get the following error: java.lang.IllegalStateException: Attempting to call unbound fn: #'clisk.functions/sigmoid It shows for any call to a clisk function. I tried to make a separate test project where I have only clisk but it results in the same error. I also tried running the test project with deps edn and then I get this error: clojure.lang.Var$Unbound cannot be cast to clojure.lang.DynamicClassLoader

^ @mikera

When using clojure.test.check, is anyone aware of a way to decorate the various clauses with explanatory text, akin to clojure.test/testing, such that it is included in the output when tests fail?

I would just use test.check inside deftest / testing / is

each is assertion can have its own explanatory text

Ah, I've been using clojure.test.check.clojure-test/defspec

Ok, I'll take a look at how to do it that way.

I wasn't familiar, but clearly there's no place for a custom message in defspec https://github.com/clojure/test.check/blob/master/src/main/clojure/clojure/test/check/clojure_test.cljc#L75

You could use this part of test.chuck https://github.com/gfredericks/test.chuck#alternate-clojuretest-integration

^ the checking macro

Hi I have a quick question on transducers, i’m using them for a little bit of ETL, and so far everything works great. However, at run time, I’m going to want a bit of logging in the mix at certain stages for better operational visibility. Is a simple log transducer at the desired points in the composed xform the best way to do this?

that's what I would use, yes - something as simple as (map #(doto % some-log-fn)) should suffice

My personal experience is that this works very well -- with the caveat that transducers were not designed for side-effecty xfns.

how so?

some most transducing contexts are eager, and side-effecting fns are fine there

Per Mr. Hickey's talk on transducers, IIRC. Using side-effects is an "at-your-own-risk" proposition

But I'm also adding that I do this frequently

(for logging specifically)

yeah I’m basically pulling stuff into datomic, but what we’re doing is a bit more involved than say the example in the docs for bulk loads,but same basic principle. set it up, then call pipeline..

I think he's saying you might not want to be doing things like augmenting from databases in the middle of an xfn chain

or doing http requests or whatnot

ironically.. that’s some of what we need to do lol

they were for data transformations

but that doesn't mean you can't it's just saying use extra awareness when doing so

I don't have the context you are referencing @goomba, but in an eager context (into, transduce, a core.async chan, eduction etc.) I don't see what the problem would be with a side effecting transducing function

in an eager context, sure. But if you're consuming from a channel or you have a stateful transducer dot dot dot

I'm saying you can do it just be aware there are extra corner cases to watch out for

a channel is eager, and that's why it's safe

all my stuff is eager thankfully

it's laziness and retries that make side-effects iffy

a channel is eager when there's something on the channel

it's eager period

no...

what do you think eager means?

if there's nothing on the channel it will block until the next thing pops in

right, as it should, and if your reducing function blocks on IO, it is still eager too

channel ops are IO

equivalently, channels are not lazy.

right

:thinking_face:

there can be gotchas of course if you are trying to use lexical scope to manage a resource and your transducer tries to use the resource and escapes scope (common gotcha eg. with with-open)

but that's not an issue with transducing functions, it's an issue with with-open

I'm not really sure "eager" can apply in an I/O context

it absolutely can

I think eager only applies when data exists

can a collection of items be consumed lazily or eagerly -- that's testable. How do you eagerly consume hypothetical data?

@goomba in clojure if you don't use a delay or a lazy function, the code is eager

You have a source for that, or is it your opinion? What happens if there's an unhandled exception? I don't think eager/lazy is defined for those situations.

from clojure core devs, I've been told the only thing that's lazy in clojure is the LazySeq type and its derivatives

and delay is an alternate, more manual, way of postponing evaluation

you mean lazily evaluated

there's no such thing as lazy that isn't lazily evaluated in clojure

I agree, but you can't evaluate data that doesn't exist

I'm sorry but this is absurd, blocking on an event is not laziness

http://clojure-doc.org/articles/language/laziness.html

this is why I'm saying transducers were designed around pure data and not intended to be used with interposing I/O -- we can dance around the semantics all we want. Everyone is welcome to use interposing side-effects but then they can get into these discussions lol

"this is why" what is why? there's nothing about blocking on IO that causes correctness issues with transducers

Are transducers somehow more provable than regular functions when it comes to IO?

are they somehow risky in a way that a normal function isn't in an IO context?

They are at least as risky as regular functions in an I/O context

OK but clojure isn't a pure language and I don't see why transducers would be called out specially here

I'm hoping there's something you are referring to that I didn't know, but so far I'm just seeing an assertion that isn't congruent with other things I know about clojure

Because if you're running a production system, and you don't want potentially masked bugs, then you should probably take special care when implementing side-effecty xfns in transducing contexts. I'm not sure why this is coming across as impractical advice.

https://groups.google.com/forum/#!topic/clojure/SVaFtQgtolc from Stuart Sierra: > Transducing/reducing is always non-lazy, so it's less risky to have side effects in a reduce compared with side effects in a lazy sequence.

Alright let me give you a concrete example

partition-all

the http://clojure.org page on transducers doesn't even mention side effects at all https://clojure.org/reference/transducers

If you're consuming from a channel, augmenting with a DB, and logging, while using partition-all for chunking batched http requests -- you could have results that would surprise you

there are a lot of different parts of the things you guys are talking about, and you haven't even agreed which parts you are talking about or even what the common terminology for those parts is

so like, you are not going to get anywhere

No, @hiredman. This is a neckbeard contest, obviously

I'm playing the "highlander" theme song in the background right now

I suspect @goomba is talking about what I think I have seen called stateful transducers, which partition-all is one, it uses a mutable array list internally, so if you use that kind of transducer in a parallel context, like can happen with core.async, you will get surprising results

but sometimes people refer to defining a collection via a reduce operation (reifing CollReduce or whatever) as a transducer as well (I think this is not correct usage but people do it) because they are concepts that are introduced together

in which case, you can have some abstract notion of collection that only exposes a reducing operation where the items in the "collection" don't exist until you do the reducing, which is different from the lazy seq model, but in certain lights you might refer to as being lazy

What are you a diplomat? 😄

OK - you can have transducers that are not safe for parallel usage, I agree

I only brought up reduce because it's something canonically eager

Better said than me, @hiredman. You're also being very charitable -- @noisesmith called me out on some imprecise language and I got roped into a "spirited" debate 😛

which is a difference between lazy seqs and the reduce model, if you are defining a lazy seq, each step in the process is lazy, if you are defining via reduce you can have a "collection" where no work has happened which seems lazy, but as soon as you start working it is all or nothing (more or less)

I don't think it's useful to conflate work that is indefinitely paused / blocked with laziness

definitely

Ok at the risk of starting this all over again, I think we're in Clojure koan territory. Meaning this is a trick question. I.e. -- lazy/eager is by definition something that can only be applied in a functional sense. It really doesn't make sense in a non-functional context.

that's false

Is code that never gets executed eager or lazy?

yes

lol

does code that never gets executed have bugs in it? it's an absurd question

or undefined

Take it to a thread please. This is a pointless (and very annoying) conversation for the whole channel to witness.

:thumbsup:

eager or lazy are properties of the abstract meaning (semantics) of the language (logic) not properties of a given execution of a program

Hello I have a problem which is I guess Unicode related, could someone help me trying to understand the issue better?

If you explain the issue, maybe someone can help...

hello sean! Ok I have a programm in java that uses jgit API to acess data from a github repository, in java it works as expected , but in clojure I get strange symbols for some unicode characters

I'm runnig my code from the clojure REPL on windows

the clojure program does the same thing: query the git repository and displays some commit data

any suggestions how to debug or fix this problem?

Some terminals or REPLs may have ways of handling non-ASCII characters that cause them to display strangely.

Looking at a sequence of Unicode code points in both environments can be a bit tedious, perhaps, but at least should always render properly on the screen.

how are you starting your clojure repl?

In Clojure, (map int some-string) will show a sequence of UTF-16 code points.

in a lot of places in java apis you can either pass in a character encoding or use a system wide default

@hiredman I type: lein repl

you should compare the value of the file.encoding property between jvms

how are you running your java program?

how do I acess this value from the clojure REPL?

my java programm is running with intelliJ

user=> (System/getProperty "file.encoding")
"UTF-8"

there are two places character encoding stuff is happening

1. on the input reading in data 2. the output printing out data

my guess now is whatever terminal you are running lein in is not setup for utf-8 output

ah!!

"Cp1252"

the REPL prints

https://en.wikipedia.org/wiki/Windows-1252 - doesn't support unicode

hmm this seems really bad, an Idea how to get a unicode shell in windows?

cygwin

I have wsl

bash

but I don't have x there and the project uses seesaw too...

try just adding “Dfile.encoding=UTF-8” to your project.clj :jvm-opts and see how it goes I reckon

https://github.com/technomancy/leiningen/blob/master/sample.project.clj#L291

Probably needs a dash before the D?

doh, yes definitely does

ok now the property prints: zettel-clj.core=> (System/getProperty "file.encoding") "UTF-8"

but the characters are still messed, which is because of the font?

so you run a java program that works correctly, does it run and output in this same terminal?

the java programm runs on intelliJ

are you able to run it from this terminal to check?

the answer might be to update windows lol https://github.com/microsoft/terminal/issues/306

I just checked it displays now correct in the seesaw swing elements, thats good enough for me

thank you very much clojure community for your help and patience 🙂

yeah I’m pretty sure it’s the terminal. so files, ui elements etc will all be fine

ALSO, if you get the cursive plugin for intellij and use that to develop your clojure, and use its repl, it will probably work fine

hm a colleague is all over it, but i'm undecided

intelliJ is quite heavy and I have to get a licence and stuff

its just a toy project is it really worth it?

if your project is open source then the license for cursive is at no cost iirc

fair enough, the repl in vim, emacs, atom and vscode probably would work as well

I usevscode actually

to edit the source, how do I get a REPL there?

https://marketplace.visualstudio.com/items?itemName=avli.clojure

https://marketplace.visualstudio.com/items?itemName=avli.clojure

? 😄

ok

i see one called betterthantomorrow.calva

no idea which one is correct

Use calva

calva is better?

I’ve never used vscode so I’d go with the judgment of someone who has

Calva is better

I'll try it thanks again everyone, cya

deps question: is it expected :local/root coordinate respects :paths in library’s deps.edn? I cannot seem to get it working

@andreas.scheinert please feel welcome to ask about Calva in #calva-dev .