Fork me on GitHub

@lilactown why not just use core.async channels?


because I want to do something else 😛


in all serious, core.async and what I'm fiddling with are different. I used the term "stream" above because I wanted to skip over explaining my exact use case, but what I'm actually building is an incremental computation graph


examples of this in other languages are Adapton, Jane Street's Incremental, KnockoutJS/MobX


the difference between a channel and what I want is that a channel has no "current" state, while an incremental computation does. with a channel, you block (pull) until you get a message; with an incremental computation, you use the current value to compute your answer, and it will tell you (push) when it has a new value so that you may recompute


Hmm, ok 🙂 If I’m understanding correctly, it sounds like you can achieve something similar by combining core.async channels with transducers and one atom per channel which holds the latest state.


you could probably create an incremental computation graph built on top of channels and atoms, sure


that's sort of unrelated to my question


Right… 🙂 I just always try to re-invent the wheel less (if possible):

(defonce ch+state
  (let [*chan-state (atom nil)
        ch          (cljs.core.async/chan 1 (comp
                                              (map inc)
                                              (map (fn [x]
                                                     (reset! *chan-state x)
    {:chan-state *chan-state
     :chan       ch}))

  (cljs.core.async/put! (:chan ch+state) 42)
  @(:chan-state ch+state))


I can get deeper into this but just FYI there is a plethora of rich content on incremental computations out there


Are you looking into this for front-end or for back-end purposes? (or both)


what I'm working on is intended to be CLJC. there are already a few passable solutions for CLJS


Nice, I like CLJC 🙂


taking your ch+state and assuming you would use add-watch to compose them, you'll run into "glitches" where some atoms will be updated while others are not. there's also problems to solve with computing the graph in an efficient order.


I see what you mean… you could “buffer” multiple updates and update a shared atom every N ms, let’s say


Or update one shared atom naively (to start) every time there’s a change


it also implies or assumes a particular solution to the problem I posted in the channel, which is that if you deref the state before a "valid" value has been received then it returns nil. I don't know if that's a good solution


That’ll ensure consistency


Yeah… until something has been “done”, I think it’s ok-ish to be nil… I think that’s a fair thing in Clojure, nil signifies something that’s just not there yet


Or not there, in general


consider the following graph:

(def counter (source 0))

(def a (compute #(deref counter) (remove even?))

(def b (compute #(inc @a))


if I initialize a as nil until #(deref counter) returns a not even? value, then when b gets initialized it will throw a NPE


I would have to code defensively around this:

(def b (compute #(when-let [n @a] (inc n))))


and this would cascade throughout the rest of my computations


> Or update one shared atom naively (to start) every time there’s a change this still runs into the problem where between propagation of values through channels, you will see one ch+state update before another. you need a way to update the entire graph and commit it at once 🙂


So just so I understand, (compute …) returns something that is

(instance? clojure.lang.IDeref (compute ...)) => true


Same for (source… ) ok…


What I’m suggesting is having one atom shared across the whole graph.


So you’re saying that the whole graph needs to act as “one” unit…. so all computation of all nodes run together, and complete “at once” ?


yes exactly. I want it (and the way I've already implemented it) to work is that you make changes to inputs, and the graph recomputes, then updates the state of all nodes at once and then notifies external listeners


I'm considering using core.async for the scheduling of computation execution, but using them as the sort of connection between nodes feels like a red herring. it doesn't actually solve any of my problems


I understand now 🙂 Cool, I still think that it’s possible to achieve the same results with channels + one global atom shared across nodes… you’d just need to schedule/buffer all the change/update functions from all nodes on that global atom and run them all in one (swap! …) call … that would ensure a consistency (I think)


I guess I'll say, I've already implemented it with half of what you say - the state of the graph is stored in one global atom shared across all nodes - without core.async. I can send changes to inputs, and this schedules a recomputation of the graph. Each node knows how to identify itself in the graph and recompute its value based on its dependencies.


why would I want to add core.async into this mix?


For sure, sounds like the same idea.


I recently learned about something called differential dataflow


Might be of interest to you, if you’re digging deep into this.


thanks for the pointer. differential dataflow definitely falls under the umbrella of incremental computation. I've been reading the literature 🙂

👍 3

if I understand correctly, we are currently looking for a similar solution. for now, we are testing something like this.


lazy atoms that recalculate according to the viewing hierarchy. we want to get something similar to subscribe with re-frame, but stay as close as possible to pure react


yeah that looks nice


just reading the source, it looks like it may still have "glitches" occur where updates are triggered before the entire graph has completed; or if an error occurs, it will throw in the middle of updating the graph and you'll end in an inconsistent state


you will also end up doing more computations than needed if you find yourself in a diamond dependency graph:

  /   \
a       b
  \   /
when src changes, if you do a depth-first re-calculation of the graph, you'll calculate c twice, which probably isn't what you want


you can still do a lot of things with those caveats, though. reagent suffers from both of those problems as well and plenty of teams are successful building apps with it


Can't wait to see what you come up with ; )


How to convert PersistenceList to vector in clojure?


What package does PersistenceList come from?


(vec my-persistent-list)


@U2FRKM4TW I assume Aditi means the clojure.lang


oh wait, that's PersistentList

🙂 3

(vec the-list)or`(into [] the-list)`


Hello, is out there any library to visualize clojure code? Not namespaces, function/variable usages. I found some, thought about trying to build one myself, but wanted to ask around first 🙂 something like this:


nice list 🙂


"This list is incomplete. You can help by expanding it." :D


yes, thank you very much! I'll take my time reading the source codes. 😄


morpheus and are both based on the analysis output from clj-kondo, it's pretty easy to get going with that data and write your own tools on top. More info here:


I always wanted to do something like that with a code-city type thing


have the clojure bits be little floating orbs


the java around it be the city buildings


well me too, but a city would be hard for first, hope can do something in reagent with svg-s with the gathered data from the analysis.

Sam Ritchie21:03:17

Many of the suggestions don’t seem to be working anymore 😕

Sam Ritchie21:03:19

the latest error from morpheus:

[sritchie@wintermute ~/code/clj/sicmutils (sritchie/clean_up_kondo)]$ clj -A:morpheus -d graphs -f png src test
WARNING: Implicit use of clojure.main with options is deprecated, use -M
Execution error (NullPointerException) at java.util.regex.Matcher/getTextLength (
Cannot invoke "java.lang.CharSequence.length()" because "this.text" is null

Full report at:

Sam Ritchie21:03:43

worked great, finally!


Hmm, @U0508JT9N - maybe you could look into the above error with morpheus? Imo morpheus is the greatest tool in this area, provided that it works ;)


hm, sure. @U017QJZ9M7W some more context, full stack trace would be nice


can try to repro...

Sam Ritchie22:03:54

I'll get the full stack trace up these eve!

👍 1
Sam Ritchie03:03:16

Here we go. If I run the command from the Morpheus repo in sicmutils I see this

$ clj -A:morpheus -d graphs -f png src test
WARNING: Implicit use of clojure.main with options is deprecated, use -M
Errors occured while parsing command:
 Failed to validate "-d graphs"

Sam Ritchie03:03:43

I figured out that that seems to happen if graphs is not an existing directory, so I make the dir. After mkdir graphs, I see this:

$ clj -A:morpheus -d graphs -f png src test
WARNING: Implicit use of clojure.main with options is deprecated, use -M
Execution error (NullPointerException) at java.util.regex.Matcher/getTextLength (
Cannot invoke "java.lang.CharSequence.length()" because "this.text" is null
Full stacktrace:

Sam Ritchie03:03:41

$ clj --version
Clojure CLI version
aliases entry, in .clojure/deps.edn:
  {:extra-deps {thomasa/morpheus {:git/url ""
                                  :git/sha "ac982a62ef5141b556bd18e352b9e318ec129818"}}
   :main-opts ["-m" "thomasa.morpheus.main"]}

Sam Ritchie03:03:58

cc @U0508JT9N thank you for taking a look!


ack, thx. will have a look


had a look. added some changes (not commited yet) to improve error messages when params are not valid (i.e. when the directory you trying to write does not exist). the stack trace you posted seems to be coming from clj-kondo — morpheus apparently uses clj-kondo as an analyzer. as currently available morpheus on github uses a slightly older version of clj-kondo i upgraded to latest release. I don’t get that exception anymore but generating the analysis takes forever — could not wait until it finished, just stopped the process. will dig further, however would be interested if and if yes how you run clj-kondo linting against the project and if you have any performance problems


ok, found the culprit. after removing the .clj-kondo/ directory from the root of sicmutils morpheus starts working

Sam Ritchie18:03:54

Interesting, that is only there to ship extra configs. Perhaps the cache had been populated by the older kondo?


Make sure to run the tool not using :extra-deps but using :replace-deps. I've seen it before when different versions of SCI do not play well with each other

Sam Ritchie21:03:55

@U0508JT9N if you push a branch I’d be happy to test out the fix


cheers, not sure what the fix would be here on the morpheus side tho... creating a fake temp dir as config dir or something?!


I don't think you should have to fix anything, this seems like a weird bug to me.

👍 1

but as I said, invoke your tool using :replace-deps, not with :extra-deps so deps from sicmutils won't get mixed with morpheus's deps


@U017QJZ9M7W please upgrade to org.babashka/sci {:mvn/version "0.3.2"}


The issue is that clj-kondo depends on org.babashka/sci but sicmutils depends on borkdude/sci so both versions will be brought in on the classpath and they are using incompatible versions of edamame (the reader)


So the solution: 1. upgrade to org.babashka/sci 2. do not use :extra-deps but :replace-deps when invoking tools.


this should solve the problem

Sam Ritchie15:03:08

Done, thanks! I'll give Morpheus a try today

Sam Ritchie16:03:31

@U04V15CAJ okay, close,Morpheus is still using an old clj-kondo which depends on an old sci. I’ll exclude that…

Sam Ritchie16:03:43

still hitting the same error. with

   {org.babashka/sci {:mvn/version "0.3.2"}
    thomasa/morpheus {:git/url ""
                      :git/sha "ac982a62ef5141b556bd18e352b9e318ec129818"
                      :exclusions [borkdude/sci]}}
   :main-opts ["-m" "thomasa.morpheus.main"]}
I see the same errors I reported above

Sam Ritchie16:03:07

@U0508JT9N this is with

clj -A:morpheus -d graphs -f png src test

Sam Ritchie16:03:28

did that work for you in sicmutils?


I did update to clj-kondo/clj-kondo {:mvn/version "RELEASE"} locally tbh


let me push that, give me a sec


6d7d8ccc1dd58ef2d4d9c781486303cf1082b899 is the new sha

Sam Ritchie01:03:36

@U0508JT9N looks like it’s running now and creating PNGs; slow though, so maybe I am hitting the issue you hit before. I cleared the cache, running it with time now to see how long we go

Sam Ritchie03:03:10

@U0508JT9N something is odd… after 2 hours it was barely through 3 namespaces


Wtf… this should not take longer than a few seconds


that is weird…


try to use -v to know what it is working on. meanwhile i have an other look


btw svgs are more fun as they are linked together so you can click through arrows and bubbles for vars as well


i am running clj -M:morpheus -d graphs -e 'clojure.core/.*|cljs\..*|:clj-kondo/unknown-namespace/.*' -f svg -v src test after rm -rf .clj-kondo with my local morpheus which is essentially the same what you have — bit more debug


it could be faster, but it is at 3rd ns in 2 mins


let me see how long it take for the whole project


hmm, why is it taking so long?


yeah it is kinda slow. but not to that extreme what @U017QJZ9M7W sees


it is at 343 graphs generated after 17 mins


will do some profiling what makes it slow, but it is deffo not the clj-kondo analysis that is done in the beginning and time is spent with generating the individual graphs for vars


good progress!

😅 1

speed woudl be deffo a feature here in terms of a potential integration with an ide like thing


suppose graalvm compilation would be an option too if loom plays ball


I don't see anything in loom which should not play well with graal yet. But that would help startup time, not performance


fair, would not hurt either 😉


true, it could be pretty cool to have a fast CLI that would spit out an svg


still running btw. to be fair you need to generate 2 graphs for each var in the project. at 551 atm after ~30 mins


don’t think anything would stop to make this parallel as well


As a side-track, I'm now investigating compatibility of loom with babashka. It works, if I add in bb.


what is your plan with it in babashka?


No concrete plans, but it's nice to be able to generate graph svgs from scripts


and clj-kondo is available in babashka as a pod as well


so you could make a very light-weight version of something like morpheus


ah got you


would be interesting


I just checked, the added binary size is pretty small and it would enable other libraries too, so I think it's good to add support for data.priority-map anyway. A* lib is often good to have.


Anyway, back to sicmutils + morpheus!


sounds like a good plan. also a would be interesting to do a rewrite in babashka


for performance this won't help btw, babashka executes Clojure slower than the JVM


fair, need to look into performance anyways. still running btw at 1700ish graphs generated


big project this

Sam Ritchie13:03:50

@U0508JT9N is there a way to have it just process namespaces?

Sam Ritchie13:03:42

there are a few that seem to be generating nodes for many clojure.core functions; not sure if that is expected. some are doing cljs.core and clojure.core in the same png


there is a way just to gen for one var


you can exlcude clojure core stuff see my cli with the -e option


clojure core and cljs core together, maybe a cljc file?! will have a look


had a quick look only on my phone but that is a cljc file where all this core deps are possible depending on what platform you use


bit awkward that they are on one graph


but easy to filter both clj and cljs out or just one of them -- see above


and after 5 and half hours generated 6242 graphs


might look into a namespace based filter apart from profiling 😉