because I want to do something else 😛

in all serious, core.async and what I'm fiddling with are different. I used the term "stream" above because I wanted to skip over explaining my exact use case, but what I'm actually building is an incremental computation graph

examples of this in other languages are Adapton, Jane Street's Incremental, KnockoutJS/MobX

the difference between a channel and what I want is that a channel has no "current" state, while an incremental computation does. with a channel, you block (pull) until you get a message; with an incremental computation, you use the current value to compute your answer, and it will tell you (push) when it has a new value so that you may recompute

Hmm, ok 🙂 If I’m understanding correctly, it sounds like you can achieve something similar by combining core.async channels with transducers and one atom per channel which holds the latest state.

you could probably create an incremental computation graph built on top of channels and atoms, sure

that's sort of unrelated to my question

Right… 🙂 I just always try to re-invent the wheel less (if possible):

(defonce ch+state
  (let [*chan-state (atom nil)
        ch          (cljs.core.async/chan 1 (comp
                                              (map inc)
                                              (map (fn [x]
                                                     (reset! *chan-state x)
                                                     x))))]
    {:chan-state *chan-state
     :chan       ch}))

(comment
  (cljs.core.async/put! (:chan ch+state) 42)
  @(:chan-state ch+state))

I can get deeper into this but just FYI there is a plethora of rich content on incremental computations out there

Are you looking into this for front-end or for back-end purposes? (or both)

what I'm working on is intended to be CLJC. there are already a few passable solutions for CLJS

Nice, I like CLJC 🙂

taking your ch+state and assuming you would use add-watch to compose them, you'll run into "glitches" where some atoms will be updated while others are not. there's also problems to solve with computing the graph in an efficient order.

I see what you mean… you could “buffer” multiple updates and update a shared atom every N ms, let’s say

Or update one shared atom naively (to start) every time there’s a change

it also implies or assumes a particular solution to the problem I posted in the channel, which is that if you deref the state before a "valid" value has been received then it returns nil. I don't know if that's a good solution

That’ll ensure consistency

Yeah… until something has been “done”, I think it’s ok-ish to be nil… I think that’s a fair thing in Clojure, nil signifies something that’s just not there yet

Or not there, in general

consider the following graph:

(def counter (source 0))

(def a (compute #(deref counter) (remove even?))

(def b (compute #(inc @a))

if I initialize a as nil until #(deref counter) returns a not even? value, then when b gets initialized it will throw a NPE

I would have to code defensively around this:

(def b (compute #(when-let [n @a] (inc n))))

and this would cascade throughout the rest of my computations

> Or update one shared atom naively (to start) every time there’s a change this still runs into the problem where between propagation of values through channels, you will see one ch+state update before another. you need a way to update the entire graph and commit it at once 🙂

So just so I understand, (compute …) returns something that is

(instance? clojure.lang.IDeref (compute ...)) => true

yeah

Same for (source… ) ok…

What I’m suggesting is having one atom shared across the whole graph.

yes

So you’re saying that the whole graph needs to act as “one” unit…. so all computation of all nodes run together, and complete “at once” ?

yes exactly. I want it (and the way I've already implemented it) to work is that you make changes to inputs, and the graph recomputes, then updates the state of all nodes at once and then notifies external listeners

I'm considering using core.async for the scheduling of computation execution, but using them as the sort of connection between nodes feels like a red herring. it doesn't actually solve any of my problems

I understand now 🙂 Cool, I still think that it’s possible to achieve the same results with channels + one global atom shared across nodes… you’d just need to schedule/buffer all the change/update functions from all nodes on that global atom and run them all in one (swap! …) call … that would ensure a consistency (I think)

I guess I'll say, I've already implemented it with half of what you say - the state of the graph is stored in one global atom shared across all nodes - without core.async. I can send changes to inputs, and this schedules a recomputation of the graph. Each node knows how to identify itself in the graph and recompute its value based on its dependencies.

why would I want to add core.async into this mix?

For sure, sounds like the same idea.

I recently learned about something called differential dataflow

https://timelydataflow.github.io/differential-dataflow/

Might be of interest to you, if you’re digging deep into this.

thanks for the pointer. differential dataflow definitely falls under the umbrella of incremental computation. I've been reading the literature 🙂

if I understand correctly, we are currently looking for a similar solution. for now, we are testing something like this. https://github.com/ribelo/rum/blob/2a41d176f5f62fc96ca40af0d00cf7fa523d5c09/src/rum/atom.cljs

lazy atoms that recalculate according to the viewing hierarchy. we want to get something similar to subscribe with re-frame, but stay as close as possible to pure react

yeah that looks nice

just reading the source, it looks like it may still have "glitches" occur where updates are triggered before the entire graph has completed; or if an error occurs, it will throw in the middle of updating the graph and you'll end in an inconsistent state

you will also end up doing more computations than needed if you find yourself in a diamond dependency graph:

src
  /   \
a       b
  \   /
    c

you can still do a lot of things with those caveats, though. reagent suffers from both of those problems as well and plenty of teams are successful building apps with it

Can't wait to see what you come up with ; )

What package does PersistenceList come from?

(vec my-persistent-list)

@U2FRKM4TW I assume Aditi means the clojure.lang

oh wait, that's PersistentList

(vec the-list)or`(into [] the-list)`

Some things I know of that should be able to handle vars: - https://github.com/jpmonettas/clograms - https://github.com/gfredericks/clj-usage-graph - https://github.com/greglook/lein-hiera - https://github.com/testedminds/lein-topology - https://github.com/stuartsierra/class-diagram (useful for interop with Java) - https://github.com/benedekfazekas/morpheus - https://github.com/clojure/tools.analyzer (might be useful if you decide to write your own)

nice list 🙂

"This list is incomplete. You can help by expanding it." :D

yes, thank you very much! I'll take my time reading the source codes. 😄

morpheus and https://github.com/SevereOverfl0w/vizns are both based on the analysis output from clj-kondo, it's pretty easy to get going with that data and write your own tools on top. More info here: https://github.com/clj-kondo/clj-kondo/blob/master/analysis/README.md

I always wanted to do something like that with a code-city type thing

have the clojure bits be little floating orbs

the java around it be the city buildings

well me too, but a city would be hard for first, hope can do something in reagent with svg-s with the gathered data from the analysis.

Many of the suggestions don’t seem to be working anymore 😕

the latest error from morpheus:

[sritchie@wintermute ~/code/clj/sicmutils (sritchie/clean_up_kondo)]$ clj -A:morpheus -d graphs -f png src test
WARNING: Implicit use of clojure.main with options is deprecated, use -M
Execution error (NullPointerException) at java.util.regex.Matcher/getTextLength (Matcher.java:1769).
Cannot invoke "java.lang.CharSequence.length()" because "this.text" is null

Full report at:
/var/folders/xw/0lq56zhn4hb4lknppw_k086c0000gn/T/clojure-8496917355770462725.edn

https://github.com/hilverd/lein-ns-dep-graph

worked great, finally!

Hmm, @U0508JT9N - maybe you could look into the above error with morpheus? Imo morpheus is the greatest tool in this area, provided that it works ;)

hm, sure. @U017QJZ9M7W some more context, full stack trace would be nice

can try to repro...

I'll get the full stack trace up these eve!

Here we go. If I run the command from the Morpheus repo in sicmutils I see this

$ clj -A:morpheus -d graphs -f png src test
WARNING: Implicit use of clojure.main with options is deprecated, use -M
Errors occured while parsing command:
 Failed to validate "-d graphs"

I figured out that that seems to happen if graphs is not an existing directory, so I make the dir. After mkdir graphs, I see this:

$ clj -A:morpheus -d graphs -f png src test
WARNING: Implicit use of clojure.main with options is deprecated, use -M
Execution error (NullPointerException) at java.util.regex.Matcher/getTextLength (Matcher.java:1769).
Cannot invoke "java.lang.CharSequence.length()" because "this.text" is null

$ clj --version
Clojure CLI version 1.10.3.1040

:morpheus
  {:extra-deps {thomasa/morpheus {:git/url ""
                                  :git/sha "ac982a62ef5141b556bd18e352b9e318ec129818"}}
   :main-opts ["-m" "thomasa.morpheus.main"]}
  }

cc @U0508JT9N thank you for taking a look!

ack, thx. will have a look

had a look. added some changes (not commited yet) to improve error messages when params are not valid (i.e. when the directory you trying to write does not exist). the stack trace you posted seems to be coming from clj-kondo — morpheus apparently uses clj-kondo as an analyzer. as currently available morpheus on github uses a slightly older version of clj-kondo i upgraded to latest release. I don’t get that exception anymore but generating the analysis takes forever — could not wait until it finished, just stopped the process. will dig further, however would be interested if and if yes how you run clj-kondo linting against the project and if you have any performance problems

ok, found the culprit. after removing the .clj-kondo/ directory from the root of sicmutils morpheus starts working

Interesting, that is only there to ship extra configs. Perhaps the cache had been populated by the older kondo?

Make sure to run the tool not using :extra-deps but using :replace-deps. I've seen it before when different versions of SCI do not play well with each other

@U0508JT9N if you push a branch I’d be happy to test out the fix

cheers, not sure what the fix would be here on the morpheus side tho... creating a fake temp dir as config dir or something?!

I don't think you should have to fix anything, this seems like a weird bug to me.

but as I said, invoke your tool using :replace-deps, not with :extra-deps so deps from sicmutils won't get mixed with morpheus's deps

the culprit is very likely this one: https://github.com/sicmutils/sicmutils/blob/1e986a5d1bfd71a3a05e95e4a23e2707106cfa72/deps.edn#L19

@U017QJZ9M7W please upgrade to org.babashka/sci {:mvn/version "0.3.2"}

The issue is that clj-kondo depends on org.babashka/sci but sicmutils depends on borkdude/sci so both versions will be brought in on the classpath and they are using incompatible versions of edamame (the reader)

So the solution: 1. upgrade to org.babashka/sci 2. do not use :extra-deps but :replace-deps when invoking tools.

@U017QJZ9M7W plz merge :) https://github.com/sicmutils/sicmutils/pull/493/files

this should solve the problem

Another one: https://github.com/sicmutils/sicmutils/pull/494

Done, thanks! I'll give Morpheus a try today

@U04V15CAJ okay, close,Morpheus is still using an old clj-kondo which depends on an old sci. I’ll exclude that…

still hitting the same error. with

:morpheus
{:replace-deps
   {org.babashka/sci {:mvn/version "0.3.2"}
    thomasa/morpheus {:git/url ""
                      :git/sha "ac982a62ef5141b556bd18e352b9e318ec129818"
                      :exclusions [borkdude/sci]}}
   :main-opts ["-m" "thomasa.morpheus.main"]}

@U0508JT9N this is with

clj -A:morpheus -d graphs -f png src test

did that work for you in sicmutils?

I did update to clj-kondo/clj-kondo {:mvn/version "RELEASE"} locally tbh

let me push that, give me a sec

6d7d8ccc1dd58ef2d4d9c781486303cf1082b899 is the new sha

@U0508JT9N looks like it’s running now and creating PNGs; slow though, so maybe I am hitting the issue you hit before. I cleared the cache, running it with time now to see how long we go

@U0508JT9N something is odd… after 2 hours it was barely through 3 namespaces

Wtf… this should not take longer than a few seconds

that is weird…

try to use -v to know what it is working on. meanwhile i have an other look

btw svgs are more fun as they are linked together so you can click through arrows and bubbles for vars as well

i am running clj -M:morpheus -d graphs -e 'clojure.core/.*|cljs\..*|:clj-kondo/unknown-namespace/.*' -f svg -v src test after rm -rf .clj-kondo with my local morpheus which is essentially the same what you have — bit more debug

it could be faster, but it is at 3rd ns in 2 mins

let me see how long it take for the whole project

hmm, why is it taking so long?

yeah it is kinda slow. but not to that extreme what @U017QJZ9M7W sees

it is at 343 graphs generated after 17 mins

will do some profiling what makes it slow, but it is deffo not the clj-kondo analysis that is done in the beginning and time is spent with generating the individual graphs for vars

good progress!

speed woudl be deffo a feature here in terms of a potential integration with an ide like thing

suppose graalvm compilation would be an option too if loom plays ball

I don't see anything in loom which should not play well with graal yet. But that would help startup time, not performance

fair, would not hurt either 😉

true, it could be pretty cool to have a fast CLI that would spit out an svg

still running btw. to be fair you need to generate 2 graphs for each var in the project. at 551 atm after ~30 mins

don’t think anything would stop to make this parallel as well

As a side-track, I'm now investigating compatibility of loom with babashka. It works, if I add clojure.data.priority-map in bb.

what is your plan with it in babashka?

No concrete plans, but it's nice to be able to generate graph svgs from scripts

and clj-kondo is available in babashka as a pod as well

so you could make a very light-weight version of something like morpheus

ah got you

would be interesting

I just checked, the added binary size is pretty small and it would enable other libraries too, so I think it's good to add support for data.priority-map anyway. A clojure.data.* lib is often good to have.

Anyway, back to sicmutils + morpheus!

sounds like a good plan. also a would be interesting to do a rewrite in babashka

for performance this won't help btw, babashka executes Clojure slower than the JVM

fair, need to look into performance anyways. still running btw at 1700ish graphs generated

big project this

@U0508JT9N is there a way to have it just process namespaces?

there are a few that seem to be generating nodes for many clojure.core functions; not sure if that is expected. some are doing cljs.core and clojure.core in the same png

there is a way just to gen for one var

you can exlcude clojure core stuff see my cli with the -e option

clojure core and cljs core together, maybe a cljc file?! will have a look

had a quick look only on my phone but that is a cljc file where all this core deps are possible depending on what platform you use

bit awkward that they are on one graph

but easy to filter both clj and cljs out or just one of them -- see above

and after 5 and half hours generated 6242 graphs

might look into a namespace based filter apart from profiling 😉