Fork me on GitHub

hey everone


But, if I want to have multiple subs on one pub (same topic) - whats the best way to do that?


essentially I want many things to be able to register/listen/subscribe for an event,and when a publish goes off, they get the data and run something, but do not block each other or know about each other in anyway (similar to redis pub/sub) - is core.async the wrong tool for that?


As it is, if I register more than one sub on the pub/topic/chan, I have to do a pub for each sub and they seem to go in order


That is not correct


They do go in order, but a single published message will be sent out to any channel subscribed to whatever topic fn returns


Internally the pub creates a mult for each topic and each subscribed channel taps the mult


What am I apparently doing wrong here? One go-loop per sub?


(def input-chan (chan))
         (def our-pub (pub input-chan :msg-type))
         (def output-chan (chan))

         (defn fire                                                                                                                                                       
           "Send in a message / payload that we'll publish to one or more listeners."                                                                                     
           [topic m]                                                                                                                                                      
           (>!! input-chan {:msg-type topic :data m}))

         (defn listen                                                                                                                                                     
           "Recv a message / payload that we'll run some function against."                                                                                               
           [topic f]                                                                                                                                                      
           ;; pub topic channel                                                                                                                                           
           (sub our-pub topic output-chan)                                                                                                                                
           (go-loop []                                                                                                                                                    
             (let [{:keys [data]} (<! output-chan)]                                                                                                                      
               (log/info "Received an event on topic: " {:topic topic :data data})                                                                                        
               (f data)                                                                                                                                                   

         (listen :foo prn)
         (listen :foo (fn [x] (prn "Hello from the second one" x)))

         ;; Called first, it only runs code from the initial listen                                                                                                       
         (fire :foo 44)
         ;; If I call a second time, it then runs the second function 


I want the rest of my app to be able to just import "listen" or "fire" from here, and not have to think about whats going on


I could just do with futures and a col of fs I guess


You are using double bang operations in a go block, big no no


but core.async seemed to come up the most for pub/sub stuff - the behavior is still present with single, sorry, the double was a stupid guess-and-hope attempt


(edited the sample - same behavior)


If you are restarting your repl the executor used by go blocks is likely all jammed up with blocking double bang operations


I did a fresh repl reload, same behavior. I'll do some more reading I guess


is the main use case for core.async to solve concurrency issues with code that has a lot of wait/idle time? (such as long db query or remote http request - basically similar use cases to js Promise.all ?)


no, not primarily. it's best for decoupling independent parts of your application with channels (queues) and/or building staged pipelines of work


Any idea what I'm botching in that snippet above?


hiredman is right that you should not use >!! or <!! inside a go block. however another problem is you’re creating two go blocks both pulling from output-chan. (every time you execute listen you’re starting a new go-loop)


unfortunately, I do not have time to read it atm but I can look at it tomorrow


My use case - I have a log scanner that reads lines using RandomAccessFile as lines are added - based on line regex matches, I want to fire off publishes that dont wait for anything at all. Then, I want other parts of the system to be able to sub to those particular line msg-types


how would I "listen" with multiple function callbacks without setting up a go-loop for each? (I want the body of the go-loop to come from a function sent in at sub time)


so I may fire off something like (fire :system-down-line {:date (some-date)}) - then my (potentially many independent and unaware of each other) listeners hears it and does something


if I have to store a collection of fns, I don't see what I'm gaining with channels vs just applying my list of fns via pmap or something in a future


I actually can’t see why your code is behaving the way it is, other than having 2 subs subscribing to the same topic using the same channel is weird and might do something unexpected. what happens if you make it 2 different output chans?

ahungry03:10:29 - that suggests maybe mult and tap accomplish what I'm envisioning?


oh thanks @bfabry - that was it


(defn listen                                                                                                                                                     
           "Recv a message / payload that we'll run some function against."                                                                                               
           [topic f]                                                                                                                                                      
           ;; pub topic channel                                                                                                                                           
           (let [output-chan (chan)]                                                                                                                                      
             (sub our-pub topic output-chan)                                                                                                                              
             (go-loop []                                                                                                                                                  
               (let [{:keys [data]} (<! output-chan)]                                                                                                                     
                 (log/info "Received an event on topic: " {:topic topic :data data})                                                                                      
                 (f data)                                                                                                                                                 


yeah. from the description of what you want pub/sub should totally satisfy it


awesome thanks everyone! 🙂


the documentation is not perfectly clear, however the consequence of this I believe is that a channel can’t be subscribed to a topic more than once


I am looking at switching to building with deps, and I wonder: my current build process uses the lein git-version plugin to produce a resources/version.edn file containing info like {:tag "5.5.0", :ref-short "4b3619b1"… } before the main build is done. That file is then slurped by a macro, so that I can use the information during the compilation. Any hints on how to produce this (e.g. run git) with clj? Or should I fall back to doing this in a top-level Makefile before running clj?


Currently using metav for this purpose. Found it on:


Oooh, interesting — metav seems to do what I need (spit the info into an EDN file), but there is still the question of running it before the main build. I guess a top-level Makefile is the way to go for now.


@jrychter I usually echo git rev-parse HEAD into a file, then stuff the file into the jar


Right — so, a toplevel build script (similar to a Makefile in spirit), that calls clojure.


why call clojure?


Well, to build the main application after git info gets extracted.


So, what do people use to build AOT-compiled uberjars with tools.deps? I found, but it doesn't pre-compile.


clojure -e "(compile 'my.main)"


I run this, then with pack I pass -e classes to include the aot'd classes.


Thanks. It does make sense, but I'd need to think how to put it together with the rest of the build.


Hi all, I would like to compare how multiple versions of the same library perform on a set of tasks. Is there an easy way to load multiple versions of the same package into a project with lein? Is there another, better way to do this? I’m aware that I could run the tests for each version separately, with the differing versions handled with different :profiles, but this doesn’t scale well above a handful of versions.


One possibility is to create different versions of the library with some encoding of the version number in their name, so they have different names. That can be as small as a 1-line change, or copying a file and a one-line change, or it might be changes to many source files, depending upon the library.


and it assumes you have access to the source code of that library.


The other would be to write some kind of script that changes your :profile contents (assuming by that you mean you are using Leiningen, so creating different project.clj files for each run, but automated, not manual)


I guess the other thing I don’t like about the :profile approach is that then I have to do any comparison between the different versions as another separate step. I’ll take a look at the option of altering the name of the library. Thanks, Andy


Multiple libraries with different names approach, loaded into the same JVM process, also opens up the possibility that things will not work because the library does some kind of global variable or other resource usage that causes it to conflict with itself. Not all libraries do that, but just so you are aware of the possibility.


Hmm. So after going through the build tools: uberdeps, depstar and Pack do not do global AOT, badigeon is more of a library than a tool, and cambada breaks with an error.


Did you see my thing about using compile?


When I was running some tests that required :aot :all, I ended up switching to lein. None of the tools-deps options would compile everything. It was kinda unfortunate since everything else we do is in tools-deps.


why not call compile function with root ns symbol before running tests?


IIRC, I still needed to include .clj files in the final artifact.


compile just creates class files, you can then include both them and clj files in an artifact


But I think it would fail unless certain .clj files were on the classpath at runtime. We wanted to bundle without source.


I'm currently developing a (yet unreleased) library with tools deps that needs some aot, if you are intersted in example:


it has compile.clj script that just aot-compiles namespace I need AOTed into classes folder, and then it gets packed using depstar, since I have "classes" in :src


Well, I think I will hold off on migratoin and stick to building with leiningen for now, until things settle down a bit. It's a big app, and time is very limited.


@jrychter We build all our (uber) jars at work for production deployment with depstar but we do not AOT anything. We don't see the point of using AOT.


I have an open source library that requires AOT on one namespace, that uses deps.edn tooling


depstar supports AOT @jrychter , you put an AOT alias with an extra directory, then call compile to do the compilation on the namespace of your choice


That cfml-interop library is packaged with depstar -- see the :build alias in that file I just linked.


Most of the packaging tools do.


Is it a code smell to use not-empty on a string?


It could lead to confusion, as people may assume you mean "not-blank"


"it depends" 😉


we use it a lot in our validations


should prob prefer clojure.string/blank?

👍 3

anecdotally, I see people do it a lot


not-empty will coerce a string to a sequence which seems like it's doing work it shouldn't

👍 1

We use it for situations where we want either a non-empty/non-blank string or nil in a threaded expression so clojure.string/blank? doesn't work there.


This is the context I used it in, actually, but some found it confusing because not-empty indicated the result might be a collection


(some-> s str/trim not-empty) for example


given that I almost always use blank? with not, I kind of wish there was an optimized not-blank?

☝️ 2

Also a very common spec predicate

Derek Passen16:10:14

Funny. I used (complement str/blank?) yesterday


@alexmiller absolutely agree, a quick look in a couple of our codebases reveals some kind of implementation of not-blank


I wouldn't just not blank? either, the impl can exit earlier in this case


maybe that's no different, but anyhow


Hi all, I have a project that depends on and I am having problems compiling the project with this library. When running the uberjar I get the following error: java.lang.IllegalStateException: Attempting to call unbound fn: #'clisk.functions/sigmoid It shows for any call to a clisk function. I tried to make a separate test project where I have only clisk but it results in the same error. I also tried running the test project with deps edn and then I get this error: clojure.lang.Var$Unbound cannot be cast to clojure.lang.DynamicClassLoader


When using clojure.test.check, is anyone aware of a way to decorate the various clauses with explanatory text, akin to clojure.test/testing, such that it is included in the output when tests fail?


I would just use test.check inside deftest / testing / is


each is assertion can have its own explanatory text


Ah, I've been using clojure.test.check.clojure-test/defspec


Ok, I'll take a look at how to do it that way.

Derek Passen17:10:23

^ the checking macro

👍 2

Hi I have a quick question on transducers, i’m using them for a little bit of ETL, and so far everything works great. However, at run time, I’m going to want a bit of logging in the mix at certain stages for better operational visibility. Is a simple log transducer at the desired points in the composed xform the best way to do this?


that's what I would use, yes - something as simple as (map #(doto % some-log-fn)) should suffice


My personal experience is that this works very well -- with the caveat that transducers were not designed for side-effecty xfns.


some most transducing contexts are eager, and side-effecting fns are fine there


Per Mr. Hickey's talk on transducers, IIRC. Using side-effects is an "at-your-own-risk" proposition


But I'm also adding that I do this frequently


(for logging specifically)


yeah I’m basically pulling stuff into datomic, but what we’re doing is a bit more involved than say the example in the docs for bulk loads,but same basic principle. set it up, then call pipeline..


I think he's saying you might not want to be doing things like augmenting from databases in the middle of an xfn chain


or doing http requests or whatnot


ironically.. that’s some of what we need to do lol


they were for data transformations


but that doesn't mean you can't it's just saying use extra awareness when doing so


I don't have the context you are referencing @goomba, but in an eager context (into, transduce, a core.async chan, eduction etc.) I don't see what the problem would be with a side effecting transducing function


in an eager context, sure. But if you're consuming from a channel or you have a stateful transducer dot dot dot


I'm saying you can do it just be aware there are extra corner cases to watch out for


a channel is eager, and that's why it's safe


all my stuff is eager thankfully


it's laziness and retries that make side-effects iffy


a channel is eager when there's something on the channel


it's eager period


what do you think eager means?


if there's nothing on the channel it will block until the next thing pops in


right, as it should, and if your reducing function blocks on IO, it is still eager too


channel ops are IO


equivalently, channels are not lazy.




there can be gotchas of course if you are trying to use lexical scope to manage a resource and your transducer tries to use the resource and escapes scope (common gotcha eg. with with-open)


but that's not an issue with transducing functions, it's an issue with with-open


I'm not really sure "eager" can apply in an I/O context


it absolutely can


I think eager only applies when data exists


can a collection of items be consumed lazily or eagerly -- that's testable. How do you eagerly consume hypothetical data?


@goomba in clojure if you don't use a delay or a lazy function, the code is eager


You have a source for that, or is it your opinion? What happens if there's an unhandled exception? I don't think eager/lazy is defined for those situations.


from clojure core devs, I've been told the only thing that's lazy in clojure is the LazySeq type and its derivatives


and delay is an alternate, more manual, way of postponing evaluation


you mean lazily evaluated


there's no such thing as lazy that isn't lazily evaluated in clojure


I agree, but you can't evaluate data that doesn't exist


I'm sorry but this is absurd, blocking on an event is not laziness


this is why I'm saying transducers were designed around pure data and not intended to be used with interposing I/O -- we can dance around the semantics all we want. Everyone is welcome to use interposing side-effects but then they can get into these discussions lol


"this is why" what is why? there's nothing about blocking on IO that causes correctness issues with transducers


Are transducers somehow more provable than regular functions when it comes to IO?


are they somehow risky in a way that a normal function isn't in an IO context?


They are at least as risky as regular functions in an I/O context


OK but clojure isn't a pure language and I don't see why transducers would be called out specially here


I'm hoping there's something you are referring to that I didn't know, but so far I'm just seeing an assertion that isn't congruent with other things I know about clojure


Because if you're running a production system, and you don't want potentially masked bugs, then you should probably take special care when implementing side-effecty xfns in transducing contexts. I'm not sure why this is coming across as impractical advice.

noisesmith18:10:35!topic/clojure/SVaFtQgtolc from Stuart Sierra: > Transducing/reducing is always non-lazy, so it's less risky to have side effects in a reduce compared with side effects in a lazy sequence.


Alright let me give you a concrete example




the page on transducers doesn't even mention side effects at all


If you're consuming from a channel, augmenting with a DB, and logging, while using partition-all for chunking batched http requests -- you could have results that would surprise you


there are a lot of different parts of the things you guys are talking about, and you haven't even agreed which parts you are talking about or even what the common terminology for those parts is


so like, you are not going to get anywhere


No, @hiredman. This is a neckbeard contest, obviously


I'm playing the "highlander" theme song in the background right now


I suspect @goomba is talking about what I think I have seen called stateful transducers, which partition-all is one, it uses a mutable array list internally, so if you use that kind of transducer in a parallel context, like can happen with core.async, you will get surprising results


but sometimes people refer to defining a collection via a reduce operation (reifing CollReduce or whatever) as a transducer as well (I think this is not correct usage but people do it) because they are concepts that are introduced together


in which case, you can have some abstract notion of collection that only exposes a reducing operation where the items in the "collection" don't exist until you do the reducing, which is different from the lazy seq model, but in certain lights you might refer to as being lazy


What are you a diplomat? 😄


OK - you can have transducers that are not safe for parallel usage, I agree


I only brought up reduce because it's something canonically eager


Better said than me, @hiredman. You're also being very charitable -- @noisesmith called me out on some imprecise language and I got roped into a "spirited" debate 😛


which is a difference between lazy seqs and the reduce model, if you are defining a lazy seq, each step in the process is lazy, if you are defining via reduce you can have a "collection" where no work has happened which seems lazy, but as soon as you start working it is all or nothing (more or less)


I don't think it's useful to conflate work that is indefinitely paused / blocked with laziness


Ok at the risk of starting this all over again, I think we're in Clojure koan territory. Meaning this is a trick question. I.e. -- lazy/eager is by definition something that can only be applied in a functional sense. It really doesn't make sense in a non-functional context.


that's false


Is code that never gets executed eager or lazy?


does code that never gets executed have bugs in it? it's an absurd question


or undefined


Take it to a thread please. This is a pointless (and very annoying) conversation for the whole channel to witness.


eager or lazy are properties of the abstract meaning (semantics) of the language (logic) not properties of a given execution of a program


execuse me?


(had to start the thread somewhere)


(as in "go here to continue discussion")


and since they're logical properties, that's why they can only be defined in a logical (aka pure functional) context.


but clojure has these properties and is not purely functional


all programming languages are logics (formal systems)


anyway, I am done, I thought you and noisesmith were talking past each other, but this is just nonsense


Awww well thanks for joining. I thought it was fun. 🙂


Aren’t lazy and eager in the perspective of the consumer? I mean, just because the producer is not producing does not mean it is lazy. Whereas in lazy, the consumer has the discretion of saying when is enough… just throwing ideas here.


Yeah it's a really good question. apparently some folks think it's pointless but I think it's kind of a thought provoking question.


Because there are multiple views -- there's the syntactic view, the semantic view, and a behavioral view


the frustrating thing is that you are using terms that have specific and precise definitions in clojure in loose and incompatible ways; as far as other people learning or understanding the language is concerned you are muddying things and making them more difficult, which makes me feel a need to make a correction or objection for their sake


but this conversation is clearly pointless


this is clojure, not clojure-beginner. Where else do you expect this conversation to go?


I object to fun that adds unneeded difficulty and confusion for others


I'm not just saying it for fun -- I'm sharing practical advice from production systems that you need to treat stateful, side-effecty transducers with respect or they'll bite you.


I disagreed with your cavalier assertion that it's exactly the same as a pure function


I think saying that is equally damaging to folks who are learning


I tried to prompt for a version of what you were saying that didn't contradict usage of terminology in clojure, and when hiredman offered it I agreed, anyway I'm done here

Andreas S.20:10:17

Hello I have a problem which is I guess Unicode related, could someone help me trying to understand the issue better?


If you explain the issue, maybe someone can help...

Andreas S.20:10:26

hello sean! Ok I have a programm in java that uses jgit API to acess data from a github repository, in java it works as expected , but in clojure I get strange symbols for some unicode characters

Andreas S.20:10:57

I'm runnig my code from the clojure REPL on windows

Andreas S.20:10:55

the clojure program does the same thing: query the git repository and displays some commit data

Andreas S.20:10:51

any suggestions how to debug or fix this problem?


Some terminals or REPLs may have ways of handling non-ASCII characters that cause them to display strangely.


Looking at a sequence of Unicode code points in both environments can be a bit tedious, perhaps, but at least should always render properly on the screen.


how are you starting your clojure repl?


In Clojure, (map int some-string) will show a sequence of UTF-16 code points.


in a lot of places in java apis you can either pass in a character encoding or use a system wide default

Andreas S.20:10:26

@hiredman I type: lein repl


you should compare the value of the file.encoding property between jvms


how are you running your java program?

Andreas S.20:10:23

how do I acess this value from the clojure REPL?

Andreas S.20:10:36

my java programm is running with intelliJ


user=> (System/getProperty "file.encoding")


there are two places character encoding stuff is happening


1. on the input reading in data 2. the output printing out data


my guess now is whatever terminal you are running lein in is not setup for utf-8 output

Andreas S.20:10:57

the REPL prints


isn't that the default encoding in PDFs?


weird aside, ignore me

Andreas S.21:10:12

hmm this seems really bad, an Idea how to get a unicode shell in windows?

Andreas S.21:10:57

but I don't have x there and the project uses seesaw too...


try just adding “Dfile.encoding=UTF-8” to your project.clj :jvm-opts and see how it goes I reckon

👍 1

Probably needs a dash before the D?


doh, yes definitely does

Andreas S.21:10:21

ok now the property prints: zettel-clj.core=> (System/getProperty "file.encoding") "UTF-8"

Andreas S.21:10:37

but the characters are still messed, which is because of the font?


so you run a java program that works correctly, does it run and output in this same terminal?

Andreas S.21:10:58

the java programm runs on intelliJ


are you able to run it from this terminal to check?


the answer might be to update windows lol

Andreas S.21:10:06

I just checked it displays now correct in the seesaw swing elements, thats good enough for me

Andreas S.21:10:19

thank you very much clojure community for your help and patience 🙂


yeah I’m pretty sure it’s the terminal. so files, ui elements etc will all be fine


ALSO, if you get the cursive plugin for intellij and use that to develop your clojure, and use its repl, it will probably work fine

Andreas S.21:10:23

hm a colleague is all over it, but i'm undecided

Andreas S.21:10:37

intelliJ is quite heavy and I have to get a licence and stuff

Andreas S.21:10:46

its just a toy project is it really worth it?


if your project is open source then the license for cursive is at no cost iirc


fair enough, the repl in vim, emacs, atom and vscode probably would work as well

Andreas S.21:10:40

I usevscode actually

Andreas S.21:10:50

to edit the source, how do I get a REPL there?


i see one called betterthantomorrow.calva


no idea which one is correct

Andreas S.21:10:39

calva is better?


I’ve never used vscode so I’d go with the judgment of someone who has


Calva is better

Andreas S.21:10:05

I'll try it thanks again everyone, cya


deps question: is it expected :local/root coordinate respects :paths in library’s deps.edn? I cannot seem to get it working


It should if I understand your question


ah, it works, for some reason classpath caching prevented the change to take effect, deleting .cpcache in the main project using :local/root resolved the issue


Changes in the local project won’t invalidate the cache


So use -Sforce if you know it’s changed


ok thanks, will try to remember, just starting with deps


This is a known issue


@andreas.scheinert please feel welcome to ask about Calva in #calva-dev .