Good Morning!

Morning

Happy Friday 🙂

I could get quite excited about Java Project Loom simplifying parallel processing, enabling us to do away with Fork/Join and reducers

but I have no idea when it is supposed to land

does anybody know?

morning

I've had to do some simple analytic stuff in node/typescript lately, and I gotta say, we're spoiled in clojureland with laziness and the various collection apis

@ben.hammond probably a few years

@ben.hammond: Yeah not sure there’s a concrete release date for it yet… IIRC it’s currently built on JDK15 which is due for release in sept 2020… so JDK16 at the earliest… but I think 17 or 18 is more realistic. I’m hoping for 17 because that’s also an LTS release.

i knew there was a good reason why I hate getting excited

Yeah…

> I can deal with the despair... > its the hope...

I’m curious why you think it’ll do away with ForkJoin/reducers though… I was thinking it’d be better for replacing the underpinnings of core.async

I would expect everying to be a plain old thread

no need for Tasks

no need for weird coroutine type structures

core.async should stay as it is, no? It's great because it works in cljs too.

cljs is a compelling reason yes

well we shouldn’t break core.async for sure… but !>> rather than !> could possibly be built ontop of virtual threads; which wouldn’t affect cljs at all.

I was looking into loom a few weeks ago. A colleague had mentioned how he was hoping it’d mean we could finally create an efficient CML implementation in clojure and on the JVM.

So I’d been looking into that and have started playing a little with CML, which is essentially core.async++++

Still trying to get my head around it though Aside from this I think the main thing I’m looking forward to in project loom is that it finally means you can do blocking I/O in a scalable way. I’ve found NIO based systems, e.g. pedestal to be a bit of a PITA when you’re dependent on dozens of parsers that were all totally reasonably written with blocking IO in mind. You then need to start pooling access to them or risk blocking all your threads.

¿CML? @rickmoynihan

Concurrent ML

https://www.acronymfinder.com/Information-Technology/CML.html

ah

ML as in 'Standard ML'?

CML was essentially an ML like language implemented to do concurrency… there was a follow on implementation called manticore that rebuilt it for true parallelism: http://manticore.cs.uchicago.edu/

but as far as I can tell the most popular implementations of it are all in scheme

Racket’s standard lib has what appears to be a faithful CML implementation… though frustratingly they don’t cite it as such.

Also guile has one

and I think there are versions of Haskell that have it

star haskell or something

There’s an attempt at an implementation of it on top of core.async too: https://github.com/polytypic/poc.cml But it seems core.async isn’t quite good enough to do it justice.

This is what is most commonly cited as the important work for understanding it: https://www.amazon.co.uk/Concurrent-Programming-ML-John-Reppy/dp/0521480892

it looks like CML doesn't have much to do with ML apart from ML having been the incubation language ?

yeah that’s my impression

it’s really just a concurrency/parallelism abstraction

well i've got some new things on my reading list now 😃

I must admit right now I don’t really understand what makes it so great… At a superficial kind of level it seems similar to core.async. What I will say is people in the know, seem to cite as being state of the art and superior to everything else… I’ve not heard anyone say this, but it feels a bit like the blub paradox for concurrency… I don’t understand what I’m missing; but some smug CML weenies might be onto something 🙂

Report back when you figure it out

its all about projecting confidence

... probably

I get the feeling that there’s more to it than that …. probably 🙂

There is this YT talk on the subject: https://youtu.be/pf4VbP5q3P0 Which attempts a comparison to other things, amongst others core.async — however I don’t think he does a particularly good job of explaining all his points.

is the CML runtime written in C? is it compatible with anything else

seems kind of hard to imagine doing real work in an isolated ecosystem nowadays

perhaps I've been plugged in to the JVM multiverse for too long

It’s an academic language as far as I can tell

So they probably only want it to be adopted for their research.

i too have difficulty imagining working in anything which didn't have a similarly rich ecosystem to the JVM @ben.hammond

Morning All!

Anyone know if there is a newer hotness for shelling out than Conch..?

"I later designed a concurrent ML implementation for an obscure language called star, which you can still find if you look for it"

reminds me of

"In 1966, I went down to Greenwich Village, New York City to a rock club called the Electric Banana. Don't look for it, it's not there anymore."

something about the way the presenter said that made me chuckle out loud

I'm about half way through that vid, and while interesting, the word 'monad' keeps appearing like an intrusive thought

AH I get it, they're synchronous, which is why they compose

yep I'm being dumb

or am I

selective communication is just (conditional) logic

monads are just computation, as are take ops in core async

so is it about non determinism then? hm

yes it's about ordering of in & out okay

ah so you can reason about it at the point that channels converge and diverge

okay I think that's the Big Idea... is it? the rendez-vous

right and that a go block is only for composing inputs and it's not that efficient and under the hood it's a continuation

ok and selective communication is just a mechanism for communicating with multiple partners and that's what alts are

Morn’ =)…

Ohh, not heard of CML, new stuff to look over when I have a moment =)…

yeah thanks to @rickmoynihan for posting the info, p interesting

@alex.lynham the rendezvous aspect exists elsewhere and I think predates CML, e.g. the CSP literature which core async / go are based around. As far as I can gather CML reifies events in a way core.async and others don’t; and importantly CML supports NAKs which lets a process decide it’s no longer available for rendezvous. Monad’s I think are only relevant in that you can transform code from do notation into CPS via a continuation monad; which is one way to implement green threads… the go macro being another. CML also has guards and a few other concepts; which I think make it easy to efficiently implement other concurrency systems in terms of CML; but the reverse is allegedly seldom true.

sounds like rendezvous might be from a lang called occam(?)

but yeah by monads, I kinda meant that interaction point, as represented by a go block - was thinking out loud really... seemed like maybe the same sort of thing, heh

either way, v interesting

@alex.lynham the m-word was coming up for me too, because it's a compositional framework so the : rv f -> rv operation used for wrap is bind , choose looks like an fmapetc... choose / select are quite interesting - it took me a moment to realise that they are roughly interrogating the backpressure state and select different execution paths based on which channel will permit a put/take (as opposed to doing starting multiple concurrent operations and selecting the first available result)

Are there any nice event driven libs that people can recommend that are cljc? The only thing coming to mind is re-frame, but I think it’s only cljc to aid in testing… The other thing I’m thinking of is hand rolling something using core.async?

@folcon depends what properties you want from your events.

For my event use cases, "hand rolling" core.async is only a few lines.

I’m thinking of writing a small proof of concept version my sim to be event driven. See if it allows me to increase comprehension of what’s going on in the code, improve performance, potentially batching events which are similar to improve cache locality. Right now it runs on a time-step and is more ECS style.

re-frame's fundamental model is very simple @folcon... reduce a stream of events onto a state object using a set of registered event handler fns. if you leave the interceptors and fx out it's almost trivial

and the wonky stuff with request animation frame

True, just wasn’t sure if it’s going to do some odd stuff within the jvm…

fwiw, I think you're right to do what you're doing

Depending on what you're doing, an agent/atom can be seen as an event handler pattern.

When you say I’m right to do what I’m doing, you mean the proof of concept? Or that my current ECS model is fine as it is? I also have a suspicion that the reactive/subscription system will let me do nice caching etc…

@folcon I don't know what your ECS model is :) I just mean that skipping re-frame might be a good choice.

Ah ok =)…

Ah, you're after a reactive programming system with caching. Got it :) I'd say building that on core.async is iffy.

I doubt you're going to get parallelism anyway, especially with re-frame.

I'd personally consider using an agent.

(Dunno your broader context though)

Hmm, is that something I’m going to have to handroll?

@folcon what paralellism are you looking for exactly?

Broader context is the simulation is the base level of the game I’m working on =)… So stuff runs on a “tick”, any new player actions are queued for next tick evaluation, so between ticks I want to evaluate the current world as well as player actions from prior ticks. It’s a game, so I expect logic to get complicated fast and I’m trying to figure out good ways to keep complexity down but keep performance evaluation up

It’s designed to run in a server with player actions coming over websockets…

So parallelism in the context of similar player actions and entities in the sim need to be computed to work out what effects they have, then apply those effects to the world…

@folcon What transaction isolation do you need for all this? 😉

Well I’m not certain what the best approach is here, I’ve looked at: 1) Writing updates as I go 2) Keeping the prior world state and collecting updates, then apply them all at the end The second approach is cleaner, but slow…

this sounds perfect for Clojure's STM to me…

Do you essentially want parallelism to allow concurrent modifications of state where those pieces of state don't affect each other?

In what way?

I guess not then :p

Oh yes.

Ah. I see.

Sorry, I said 2 things and got mixed up :)

It's getting late.

@folcon have you seen Rich's ants demo?

I have, but not really used any of those constructs =)…

https://gist.github.com/michiakig/1093917

I think they're a perfect fit for what you're describing.

I’ve seen the core.async demo and talk in clojurescript =)…

I’ve just never seen sync etc used in code in the wild?

That's because it's not useful in a lot of contexts. Most applications are (mostly) stateless. But your whole program is about modelling state.

Hmm, I’m getting the feeling that there’s not going to be much documentation to do this then?

Documentation? Plenty. There's loads on http://clojure.org about it's state constructs. Examples? Nah.

😞

I think the ants demo should be plenty though ^ State is really simple in Clojure :)

(geddit?)

😛

@folcon I'm not super experienced with refs or anything, but I'd be happy to help you debug any ? you end up with. I'd like to help improve my understanding.

The other thing I’m worried about is reproducibility, am I going to get issues that are going to be impossible to debug by opening the non-determinism door…

Well that would be good =)…

If you're doing concurrency then you give up event ordering & perfect reproducibility. It's on you to make sure your state changes fit within the rules of the world. For example if pieces can be frozen, then any "moves" from that point onward should factor that in (or not, your game, your rules!)

I’m currently writing up a blog post that covers some of the system I’ve built already… which is much simpler than what we’re discussing, but I feel like this is an area that I’ve not seen much (read any) tutorials in and I would like to see more sim-agent based games being written =)…

I think Clojure would make a good game-logic language, especially given that games take ages to start anyway.

True, but I also need to ensure that doesn’t create inconsistency =)…

@folcon Do you need to persist this state?

I will

but I don’t believe that I need to persist it while it’s running? Instead I was thinking of async persisting it while it’s running. IE: Compute a tick, save the output while broadcasting it to the clients… I did a bunch of experiments trying to work out how to do the kinds of operations I wanted to do in the db level, but it always turned out quite slow…

round-tripping to the db will be slow, I like your async idea :)

Well I was just trying to think of what needs to be loaded, ie the server reboots and you’ll need to pickup the game state =)…

Yea there’s a fun question of how much working memory some of this stuff takes, that’s the downside of my async idea :)…

Looks like I need to dig into the ant demo =)…

Btw, are there any good resources around working in transducers that are a bit more complex that just flat collections? Been trying to figure out if how I’m reducing this is efficient

(defn update-category [pantry-category portions-required]
  (-> (reduce-kv
        (fn [{:keys [portions-to-eat] :as state} food-name {:keys [portions] :as food-detail}]
          (if (<= portions-to-eat portions)
            (reduced (-> state
                       (update-in [:pantry-cat food-name :portions] - portions-to-eat)
                       (assoc :portions-to-eat 0)))
            (-> state
              (update :pantry-cat dissoc food-name)
              (update :portions-to-eat - portions))))
        {:pantry-cat pantry-category
         :portions-to-eat portions-required}
        pantry-category)
    :pantry-cat))

(defn amount-to-eat [pantry day-of-food]
  (reduce-kv (fn [pantry food-category portions-required]
               (cond-> pantry
                 (contains? pantry food-category)
                 (update food-category update-category portions-required)))
             pantry day-of-food))

(let [day-of-food {:food-base 18, :food-extra 18, :drink-base 18}
        pantry {:food-extra {"Eggs" {:name     "Eggs",
                                     :category :food-extra,
                                     :portions 168}},
                :food-base  {"Hard Bread" {:name     "Hard Bread",
                                           :category :food-base,
                                           :portions 126}},
                :drink-base {"Cheap Beer" {:name     "Cheap Beer",
                                           :category :drink-base,
                                           :portions 156}}}]
(eat-from-pantry pantry day-of-food))

This is interesting:

Loading src/ants.clj... done
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by clojure.lang.InjectedInvoker/0x0000000800b8ac40 (file:/Users/folcon/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar) to method sun.java2d.SunGraphics2D.drawImage(java.awt.Image,int,int,java.awt.image.ImageObserver)
WARNING: Please consider reporting this to the maintainers of clojure.lang.InjectedInvoker/0x0000000800b8ac40
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

Transducers are only for "flat sequences" (they're an abstraction of conj)

Reducers are generally very fast. You're usually looking at algorithmic complexity in your code.

Hmm, ok, I was asking because I saw some stuff in cgrand’s xforms, but couldn’t quite work out how to use them =)… Might have to rethink this structure then…

My shopping/eating simulation is just eating up a bit too much time >_<…

I'm lying a little bit, but I'm basically telling you the truth :)

Transducers are faster than lazy sequences. Reducers and transducers have comparable performance.

I see 😃

@dominicm So digging into this ant demo a little, were you thinking of the “agents” as a sort of threadpool which bounces back and forth between game entities? Or modelling each agent in the same way as an ant? I’m asking I’m not certain how many “ants” I can run at once? My design calls for a lot of independent agents, but I wasn’t thinking of them all as running simultaneously (mostly because I wasn’t sure that was possible =)…), instead I was thinking of them as computing their action and then moving onto the next one…

@folcon each entity in your game as a ref, like in the ants demo. My understanding is that it scales very well.

Hmm, it’s really not liking 10000 ants =)…

[258.039s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 4k, detached.

(. Thread (sleep ant-sleep-ms))