Fork me on GitHub

I wrote a couple of general functions for application of functions over a collection of collections of arguments. They constrain the number of parallel workers to a desired number. p-apply returns a channel on which results are streamed. p-apply-sync returns a collection of results, in the same order as the original inputs.


It turns out I needed a solution for this as well. 🙂


Why would I use rabbitMQ or something like that when I can use Core.Async?


I don't need to talk to other machines right now.


Or programs, (Which I could have done anyway using Http?)


persistence maybe?


some reasons to use RabbitMQ over core.async are when you have too much volume to handle with a single machine and/or when you need fault-tolerance


I have currently modelled quite a lot of stuff in core.async that will later handle volumes too big for one machine. It’s good for prototyping the logic, even if it will eventually run elsewhere on larger data sets.

Denis G12:09:14

is there a reason why if-let* (if-let which supports multiple bindigs) is not in standard lib?

Denis G12:09:01

same for when-let*

Denis G12:09:31

how can I conj only not-nil terms?

Denis G12:09:44

without always doing if-else


filter + conj


@denisgrebennicov this might be an XY problem. what are you trying to do? maybe you want (remove nil? [1 nil 2 nil 3 nil])

Denis G12:09:48

dunno. filtering AFTER insertion doesn't feel right. Can not I do it before insertion? Like (comp (partial filter not-nil?) conj) or smth


can you give us an example of input and your desired output?

Denis G12:09:57

it input is nil, return original seq, otherwise conj

Denis G12:09:04

without if-else 😅


you could write your own version of conj which would do that, but it would be highly unusual 🙂 thats why I asked for example to get a sense of what do you want to use it for, because there is most likely a more idiomatic way of doing it


so how would you use such function? I’d like to see a concrete example of data

Denis G12:09:51

just artificial actually I was thinking of a problem, having nested loops and then I realized how powerful for construct is. Especially with the :when clause which is more or less exactly what I need in my situation


what are you solving?


@denisgrebennicov Since into is based on conj, you could slip a nil filter in like this:

(into coll (remove nil?) xs)


If you want to “conditionally conj” this construct might be close to what you are after

(cond-> coll (some? x) (conj x))

Denis G13:09:47

looks sweet


Is it bad practise to have a async/go-loop running forever? I have a batching go-loop that runs all the time since messages will come in all the time and I need to batch them.


I don't think it is necessarily bad - provided it is being used throughout the life of the program then it makes sense to keep it around. I'm pretty sure there are a limited number of threads which are reserved for go blocks, so keeping one running forever would just occupy one of those threads (I think!)


An idle go block shouldn’t occupy a thread from that pool.


Is it even "normal/good practise" to have channels who are pre-defined through def -s? In most programs I've seen who are using channels and core-async, they are always in a function like (let [chan ...)


I don’t know. I have a program that is even worse in that case 🙂 It defines a go-loop at the top level because that’s the whole point of the program is to wait for stuff, process, it and recur. (And this is in ClojureScript where there is only one thread.)


The point of my program is to wait for users who are queing up, batch them together, send each user a notification, wait for accept/cancel, if too long -> send accept to the available queue again etc. So I think the go-loop would work 🙂..


@coderdanger is this hooked up to a websocket?


I bet you looked at sente already


I've looked at it but I haven't checked it out properly yet. I was thinking about using the websocket/commmunication through Sente. But the batching and stuff, not sure if it supports that.


in the sente handler you could write to another core async channel. but if you have something working already, then that’s probably not worth it


@borkdude Looks really neat. Will check it out. Still in the phase where I try different solutions 🙂


Do you guys know any really good Core.async tutorial or reading material that is up to date. I really want to grasp this stuff. I've been looking into the following: 1. (Seems a bit outdated.. but still good?) Havent looked into this one too much 2. (Not in-depth, but good introduction.) 3. Not in-depth


i'm a big fan of eric normand, and the courses i've seen have been very well-done and helpful. can't speak for the core.async one specifically though


eh the link isn’t working, but you should find a webinar by David Nolen on core async

Chris O’Donnell16:09:32

Not sure if they are exactly the same as the tim baldridge core.async videos that he had on his youtube channel, but I found those to be extremely useful for understanding how to solve problems using core.async at a high level.


Hi all, I'm trying to get started in clojurescript to make a sort of data-processing application with a simple web interface and the benefit of 'live-coding' via figwheel. I've worked through a few tutorials, but still a little confused and overwhelmed at the terminology and different templates available - is it recommended to start with the "lein new figwheel" template and build up from there? Or should I go with something more full-featured like the figwheel-main, reagent-frontend or reagent templates?


Just to note, I've never done any JS web development before so some of the terms like React/client/server/devtools in the bigger templates are quite foreign and I'm worried they might be overkill to deal with all at once... Would be really grateful for any guidance or references! Thanks 😄


hey, how well do you know JS?


or other languages


I'm pretty proficient in Python, and familiar with FP from Haskell and F#, also dabble in Emacs Lisp


No experience with JS so far unfortunately, it's the main reason I'm going with Clojurescript for a easily portable web app


@qythium You say you're not familiar with "terms like ... client/server ..." -- are there specific questions you can form to ask us about those terms?


No specific questions in particular, just unfamiliar with the web dev framework as a whole - as far as my application is concerned it doesn't have to connect to the internet or perform any complicated transactions. The browser should just be there to provide a runtime environment and UI canvas. But I start browsing through all these templates and guides and they have different clj/cljs/cljc folders and references to client/server architecture - not sure if they are relevant to my use case, before spending time trying to get my head around the concepts


What do you want to run your data processing?




some "desktop apps" are built on electron these days, where you'll end up with "client/server" language, but node is the server, so you're still doing cljs on both "sides"


I don't quite understand, sorry 😅 What 'sides' are these?


Everything is taking place on the same machine, right? Just to clarify


I just bring that up because if you look for "clojure desktop app" you may end up seeing some codebases that are electron based, which will further confuse you 🙂


lol right


It's "client/server" but on your desktop, because the web was built on that client/server architecture


yeah I was confused by the nodejs/electron stuff as well - should I be considering all that?


It's a much different path that choosing clojure with a built in webserver like jetty and serving your app that way


but the clojure version will have slower start up time


startup time isn't really a concern


then you could use any of the standard web templates, like luminus


and just package it up as a jar


ah, that's in Clojure not cljs?


You could just write a console app in Clojure for your data analysis.


Unless you specifically want to learn to build web apps and user interfaces?


will packaging as jar affect portability? We want to make it as simple as possible to just run in a client's browser without any dependencies


A JAR built with Clojure will not run in a browser...


yeah I figured as much


But a jar can have a jetty server built in


Right, you can have a full web app in a JAR file that starts it's own server -- which you can then browse in any browser (with that server running locally).


But building your analysis app as a standalone cli app, which you can build a web service over, is considered good practice too


Having a simple user interface is a major reason for doing this, I thought it would be easier to leverage the browser UI frameworks than figure out a full fledged desktop app


Are there any disadvantages to doing everything in clojurescript? Instead of splitting some bits into a jar and having to host the JVM


probably depends on the analysis being done


because of library support?


@qythium So this would be completely self-contained? Users would enter data and it would do the calculation and display it? No databases, no files on the file system?


no, it has to interact with the filesystem to write output files


Browser-based applications are not allowed to do that, in general.


but no databases


Are you more experienced with js or java?


neither, although I've done C# before..


I may be biased, but I think interacting with the filesystem and other system level things are way easier on the jvm than node (which most electron type things use)


But for folks that know all about node and the js ecosystem, they may have an easier time on that stack


Oh okay, the fs library I was looking at is nodejs specific


yeah lots of async


Either way, if you have file system interaction, you're looking at a client/server app -- a ClojureScript user interface (browser-based web app) talking to either a ClojureScript/Node.js or Clojure/JVM server-side app.


okay, so the server-side app is what does the data processing


and it somehow communicates asynchronously with a separate client app which runs in the browser?


The client app can make simple HTTP requests (GET, POST) to the server app, which responds with HTML (web pages or fragments) or JSON (data)


Does this setup with luminus and clojure jars all work with figwheel live code reloading? I tried out a cljs tutorial with it and really liked the workflow


You can have a similar workflow on the JVM with a REPL.


if you don't need a db, you prolly don't need all of luminus


a simpler template will do


(i.e., both client -- in the browser -- and server -- on the JVM can have the same live coding workflow)


luminus is good to learn though, in general


Yeah, definitely worth learning Luminus. The Web Development in Clojure book is based on that I think...?


Yup, React.js, Reagent, Luminus...


I believe so. I prefer more pure clojure stacks myself, but in the real world, working with other dev shops, you're going to need to learn how to interface with other common web stacks and skillsets. It's pragmatic in that way.


So it was really helpful having learned luminus' methods for some projects, while we'd usually run with pedestal for green field projects


At work we have Ring + Compojure (mostly, we have one Ring + Bidi app) and we generate HTML with either Selmer (mostly) or Hiccup. And we have independent front end apps based on React.js that talk to them via REST (HTTP with JSON).


Hmm, I think I'm starting to grasp it a little better - thanks for the help so far!


chestnut is a popular template, if you haven't heard of that one

Space Guy20:09:27

Is there a standard setup for figwheel-style 'reload as you save' for JVM-side code? I've found either "eval bits of it manually" or "build your own file-watcher out of hawk + tools.namespace + etc"


Ah, chestnut looks interesting, thanks! I hadn't come across it while searching


My natural workflow is evaluating code as I type forms in -- I don't even save the code in that workflow.


me too. I think cider has an experience like that, but I just use the repl directly


I type into source files (or test files), evaluate forms as I go (and and that includes running tests too). I save from time to time and re-eval whole files. But I don't actually like the load/run-on-save workflow... I prefer to have more control.


I almost never type into the REPL these days.


It makes more sense when programming reactive UIs


(the whole save, reload thing)


Yup, coming from python/jupyter notebooks using the eval-defun-at-point approach was quite natural


Have you looked at the Gorilla REPL @qythium? That's supposed to be very close to the Jupyter model.

Space Guy20:09:55

I guess that works. I've used 'gorilla-repl' (notebook-style) for a few tiny things and there's always a lot of tracking "wait, I need to re-run cell 1 then cell 3 then ..."


Yeah, that would be the reactive UI aspect of things. There's a lot less of that on the JVM.


Well, he means evaling forms at a time, rather than whole namespaces


In clojure, you want to eval whole namespaces less often


That looks interesting as well.. hmm


But back to the previous suggestion, the overall idea would be to make a pure Clojure CLI tool and interface it with a react web app which also runs on the JVM?


I've heard bearded people say that it's best to structure your app as a cli app first... get your data interface squared away first. Then, when you build the gui version, you can build on top of those apis


but then you can always test directly against the data apis, separate from the gui


seems to have been good advice generally


You can build directly over the apis, from clojure... you don't have to interface with it via a socket, but keep the web rendering stuff logically different. And a good way to enforce that discipline is to have a cli entry point for your app that does all the processing functionality.


Yeah that sounds like good advice


In that sense, it shouldn't matter what language the app providing the API was written in? Will it be ill-advised to do a F# data-processing CLI tool and try to interface it with the Reagent app?


Well, part of the exercise is to learn clojure, right? Part of the sweet spot is how clj and cljs work together. I'd recommend trying a full clojure stack.


But part of the awesomeness of cli tools that are posix compliant is that you do it however you want, yeah.


There's also OS compatibility issues which I've run into with F# - I guess jars are supposed to mostly get around that


Okay, will give your advices and the chestnut library a spin, maybe come back with more specific questions later - thanks so much for the help! 🙂


The nice thing about building REST APIs is that they're easy to write tests for in Clojure using clj-http, for example. As we've been building out our REST API over the last year or so, we've written tests first for all the error cases and happy cases we think we need per the requirements, and then writing enough logic to make those pass (even if the underlying systems are mocked to use canned test data etc). That way we can test the full HTTP authenticated request with headers etc and the various HTTP verbs and be fairly confident that there's a solid REST API that the front end folks can build and test against.


In fact, before we started using Clojure for the REST APIs themselves, we started using Clojure to write the tests for those APIs. So, yes, in answer to @qythium’s question from half an hour ago, the REST API code and the tests and the front end can all be in different languages if you want.


one quick last question, the chestnut template has several choices of React wrappers - Om/Om-Next/Reagent/Rum/Reframe


Any reason I should choose one over the other at this stage, and will I be able to switch easily later? Lots of the tutorials I saw have been using Reagent


@qythium Reagent is the most widely used and also the easiest to get started with.


Got it, thanks! I was wondering why it had Om as the default.


chestnut started as an om template back in the day


With a deps.edn project, how do I run a repl from emacs? cider-jack-in fires a repl but does not import the required libraries. Running clj from the project root works


I keep getting Could not locate ... in the classpath


@pvillegas12 Do you have the latest version of CIDER etc installed? Support for clj in CIDER is pretty new I think...?


Maybe ask in the #cider channel?


just reinstalled it, I’m on 0.18.1snapshot


Sounds recent. I'm afraid I haven't used CIDER for a while so I don't know what to suggest...


I’m trying to connect to an nrepl with clj directly


but I’m hitting other problems like the cider-nrepl problem


Hmm, I remember figuring out how to get clj to connect to an existing nREPL server instance and it was non-trivial -- but that's not really the use case it's designed for.


Once you have an nREPL server instance up and running, you can connect to it from editors, or lein / boot ... but those are designed as nREPL clients.


What I want is to send expressions from emacs to clj


Yeah, I would expect CIDER to be able to start an nREPL server instance via clj and send code to it (just as it can start nREPL server instances with lein or boot).


I guess my issue is an alias one


Hopefully someone will answer your question in #cider -- I see you asked there.


I have some dependencies in the dev alias, that’s why it can’t read the depdencies correctly


Ah, so you probably need to edit the command before it starts the REPL. Try c-u m-x cider-jack-in and it should offer the command before it runs it. So you should be able to add -A:dev to the command and then press return.


not getting the command to edit it:thinking_face:


(btw, that shows how to start an nREPL server with clj -- where you could specify the alias -- and then connect to that running server with CIDER, as a workaround)


Reading over that page, it looks like you can customize the cider-clojure-cli-parameters variable to include your -A:dev but that's going to be a global change...


If you look up dir locals you can make it override that var inside your project directory

☝️ 4