Fork me on GitHub
#off-topic
<
2020-06-29
>
borkdude17:06:03

Is this about the scalable language?

😂 6
🔥 3
Cory17:06:12

i'm willing to flip a coin on whether this is about scala or python

orestis18:06:59

The original author is involved with PyPy afaik

phronmophobic17:06:44

are there any good examples of projects that effectively utilize refs?

borkdude18:06:59

@smith.adriane I do know that clojure.test uses a ref, but I think it might as well have been an atom: https://github.com/clojure/clojure/blob/30a36cbe0ef936e57ddba238b7fa6d58ee1cbdce/src/clj/clojure/test.clj#L753

👀 3
phronmophobic18:06:08

In similar spirit to the ant sim, https://gist.github.com/michiakig/1093917 , I could imagine a todo app with the following model:

(def todo-app-state
  (ref {:todos
        (ref
         {:work-todos (ref [(ref {:complete? false
                                  :description "first"})
                            (ref {:complete? false
                                  :description "second"})
                            (ref {:complete? true
                                  :description "third"})])
          :home-todos (ref [(ref {:complete? false
                                  :description "first"})
                            (ref {:complete? false
                                  :description "second"})
                            (ref {:complete? true
                                  :description "third"})])})
        :filter-fn (ref :all)
        :next-todo-text (ref "")}))
This "feels" wrong, but I'm not convinced either way. I'm looking for other examples that might help think about the design tradeoffs here.

dpsutton18:06:46

i've never actually used refs before. but i'm assuming nesting them is no good

phronmophobic18:06:19

I can't tell if that intuition has a real basis, or it comes from the fact that nesting atoms doesn't really make sense. the key difference between atoms and refs in this circumstance is that you can deref/update nested refs within a transaction to maintain consistency.

borkdude18:06:53

I didn't know you could nest refs and get consistency.. that sounds really hard to reason about

andy.fingerhut18:06:50

I would like to write something more complete than the following belief at some point, but I am pretty sure that having atoms/refs/agents nested within the values 'contained within' other things of those kinds, can lead to incorrect behavior of updates to their contents.

phronmophobic19:06:47

@U0CMVHBL2, would love to see the more complete follow up! I assumed that nested refs should be technically possible, but would love to have more details how that would or wouldn't work either way

andy.fingerhut19:06:51

I suspect it is simpler to create cases where atom-within-atom has update bugs, and then see if that generalizes to single-ref within single-ref examples.

phronmophobic19:06:03

well, atoms within atoms definitely fails for trying to obtains a consistent snapshot

andy.fingerhut19:06:04

I'll definitely ping you if I do write up anything there, but isn't among the hottest 10 things on my plate right now.

👍 3
phronmophobic19:06:51

I realize I'm definitely off in the weeds, but it's fun to explore!

andy.fingerhut19:06:11

I wouldn't answer if they weren't weeds I would like to understand in more detail myself 🙂

bananadance 3
😁 3
phronmophobic21:06:24

@U0CMVHBL2, one last question if you ever get time: In the ants simulation, the https://gist.github.com/michiakig/1093917#file-ants-clj-L34. if the model did allow the world to shrink and expand, how would that be modeled? My assumption was that you would simply use a ref for the world rather than a var, but I would love to know if there is a better approach or if using a ref which contains other refs is fundamentally broken.

andy.fingerhut02:06:19

If you stop the simulation in some suitably clean way, then you could re-bind the var to a larger world, I would think. Are you asking about how one might model the world to change size in a concurrency-safe way?

andy.fingerhut02:06:40

I am not sure if this makes sense, but if you access every ref in the entire world in a way that would update the contents of every ref (i.e. every cell), then it seems like it might be safe to change the size of the world during such a transaction, but that is just me guessing out loud, so far.

andy.fingerhut02:06:58

Adding new cells seems like it might be easier in some sense, since you know the new cells cannot be involved in any transactions until after they have been added to the world. Removing cells/refs seems trickier to get right.

phronmophobic05:06:00

I've made a version of the original ants that allows the world to grow and shrink that seems to work :https://github.com/phronmophobic/ants/blob/master/src/ants/ants.clj Overview of changes: • world is now a ref:

(def world
  (ref
   {:places
    (apply vector 
           (map (fn [_] 
                  (apply vector (map (fn [_] (ref (struct cell 0 0))) 
                                     (range initial-dim)))) 
                (range initial-dim)))}))
behave is now fully enclosed in a dosync with (ensure world) at the top, https://github.com/phronmophobic/ants/blob/master/src/ants/ants.clj#L200behave now checks if the ant is still within the world bounds. if not, the ant "dies", https://github.com/phronmophobic/ants/blob/master/src/ants/ants.clj#L202 • added a function to resize the world:
(defn set-world-dim! [new-dim]
  (dosync
   (alter world
          update :places
          (fn [places]
            (apply vector 
                   (map (fn [i] 
                          (apply vector (map (fn [j]
                                               (get-in places [i j]
                                                       (ref (struct cell 0 0)))) 
                                             (range new-dim)))) 
                        (range new-dim))))
          ))
  nil)
resizing the world while the simulation is running seems to work just fine

hiredman18:06:57

Nah, that's gross

noisesmith18:06:24

I don't think @smith.adriane meant nesting refs led to consistency, but rather than transactions could ensure the consistency despite nesting

noisesmith18:06:39

but regardless, agreed, don't put mutable things inside refs / atoms

noisesmith18:06:51

(which includes other refs / atoms)

phronmophobic18:06:07

I understand not nesting atoms, what's the reasoning for not nesting refs?

hiredman18:06:34

because it means more mutable things, and you want immutable things

noisesmith18:06:46

the generic problem is that that consistency functionality about atoms and refs relies on retries, and is totally undermined by putting mutable objects inside them

phronmophobic18:06:31

I realize this code is ugly, but I can't see how this would cause consistency issues:

(dosync
 (let [work-todos (-> @todo-app-state
                      :todos
                      deref
                      :work-todos)
       a-todo (-> work-todos
                  deref
                  first)]
   (alter a-todo update :complete? not)
   (alter work-todos (fn [work-todos]
                       (vec
                        (drop-last work-todos))))))

phronmophobic18:06:21

I understand putting most mutable objects within a ref would have issues, but are there issues specifically with nesting refs?

hiredman18:06:59

refs are subject to write skew, so if you are going to nest them you should at least use ensure

👍 3
phronmophobic18:06:50

that's good to know!

hiredman18:06:10

but like, if your interest is in guis, you should look at elm, and thing about the state being passed around as an immutable database, with each phase (or whatever elm calls them) returning a possibly updated database

phronmophobic18:06:16

I have indeed spent a bunch of time looking at Om, Om next, elm, react, reagent, fulcro, hoplon, cljfx, re-frame, svelte, and others

hiredman18:06:04

it is a perennial thing that people keep trying to build these dataflowy mutable cell things for doing uis

hiredman18:06:05

the rock they all founder on is starting from mutable stuff, regardless of how much you put around the mutability (stm, etc) to try and make it "safe"

phronmophobic18:06:06

the references for this would be at the edges

hiredman18:06:02

instead of trying to bind a ui component to a mutable cell, instead bind a ui component to a query on an immutable database and an update message to change the state of that database

phronmophobic19:06:42

ui components aren't bound to mutable cells. they're still pure functions. the idea would be to use refs as a model and produce a single immutable value from the model before handing it to the ui component

borkdude18:06:51

@smith.adriane What if your todo app wants undo? How could you implement that using this model?

borkdude18:06:15

It becomes looooots easier with immutable values

hiredman18:06:50

if you look at the ant sim, I believe it doesn't actually nest any reference types

borkdude18:06:19

The entire Clojure APIs are optimized for working with (nested) immutable data structures, update-in, etc. Diverging from that is asking for more work and more bugs.

hiredman18:06:06

wanting multiple layers of reference types usually means you've out grown using a map as your database and need something more powerful

hiredman18:06:11

so use clojure.set/index or datascript, etc

borkdude18:06:34

@smith.adriane Are you familiar with Om.next, reagent, re-frame etc?

noisesmith18:06:52

I was going to mention datomic as something using refs, but of course that's not open source for the most part

noisesmith18:06:06

(speaking of datascript)

phronmophobic18:06:09

yes, none of the cljs libs can use refs.

borkdude18:06:12

The tendency in CLJS front-end has been more towards one single mutable atom for the entire app state and using some kind of event or query system around that

Michael J Dorian18:06:35

Just my two cents, my game simulation has at atom that serves as a collection of agents, the agents being the actual moving parts in the game, and I've found it immensely streamlined versus a single atom that contains the universe directly

borkdude18:06:36

@doby162 Why do those agents have to live in an atom though?

Michael J Dorian18:06:01

Because the number of agents is every changing

phronmophobic18:06:03

i think using refs in user interfaces is an interesting design direction. I know there are other explored design spaces, but I'm curious if there is anything to learn from trying to use refs to represent models for user interfaces

Michael J Dorian18:06:16

every client connection or NPC is a new agent

borkdude18:06:09

@doby162 I do something similar with websocket connections in an atom in a webserver.

phronmophobic18:06:31

I realize this code is ugly, but I can't see how this would cause consistency issues:

(dosync
 (let [work-todos (-> @todo-app-state
                      :todos
                      deref
                      :work-todos)
       a-todo (-> work-todos
                  deref
                  first)]
   (alter a-todo update :complete? not)
   (alter work-todos (fn [work-todos]
                       (vec
                        (drop-last work-todos))))))

borkdude18:06:34

I would be both surprised and not surprised if that worked, but either way very confused and burnt out trying to understand what happens 😉

andy.fingerhut18:06:04

I am fairly certain that one can easily write straightforward-looking examples of nesting refs/agents/atoms within other instances of the same kind of thing, or other kinds of mutable references, that lead to subtle and occasional bugs when updating their values. I would not be surprised if it is possible to write such a thing that was provably correct, but would expect it would require pretty careful reasoning about the implementation of refs/atoms/agents, and/or the user-provided update functions, to prove that.

3
andy.fingerhut18:06:05

Whereas if you limit the values contained within refs/agents/atoms to immutable values only, you are in a much-easier-to-reason-about situation.

💯 3
phronmophobic18:06:36

fwiw, the idea would be have the "hard parts" done by a library

andy.fingerhut18:06:42

Such a library seems possible, but if I were recommending that someone use such a library, I would probably want text reasoning why it was correct, with examples of when it was correct, and when it was clearly not correct (because of update function violating assumptions of the library), where that text was 3 to 4 times longer than the implementation, at least, and demonstrated in-depth knowledge of Clojure's implementation.

phronmophobic18:06:27

currently, the UI components don't really care what reference types the model uses. the components themselves are pure functions that return immutable values. I currently have a use case where I'm trying to build a user interface for a model that does use refs and I'm thinking about what that might look like

hiredman18:06:38

the big issue with nesting refs is it means the only way to read them consistently is in a transaction. which if you've seen rich talk about clojure's epochal time model, he very much wanted to avoid readers impeding writers

phronmophobic18:06:57

readers still wouldn't impede writers due to mvcc

hiredman18:06:28

they may or may not

phronmophobic18:06:36

I guess I should clarify that the the ui components are all pure functions and don't have mutable parts "inside" them. it's still : model -> pure data -> view-fn -> pure data

phronmophobic19:06:03

the question is concerned with only the model part. currently everything is setup to work with a single atom as the model. however, I'm interested in supporting: model -> pure data where the model uses refs

phronmophobic19:06:48

hence "are there any good examples of projects that effectively utilize `ref`s?"

ec19:06:26

So Im in some weird sht with Java. As in the comment, if I create static ExecutorService consumers doesn’t run, but if its not static or if I use CompletableFuture.runAsync (common pool) it runs. Also I created WorkStealingPool since I have no control on consumer functions. (probably cpu bound tho) (it makes sense?)

hiredman20:06:31

what is the value of

Runtime.getRuntime().availableProcessors()

hiredman20:06:40

I'd also suggest that is a the universe telling you not to use create threadpools using static initializers

😂 6
ec21:06:32

available=12 (ryzen power)

ec21:06:54

not using statics means drilling it down from the first Singleton parent object to make sure its created only once

ec21:06:11

Maybe I should just stick to common pool. No one should go crazy with their callbacks anyway, thx

hiredman21:06:48

I would maybe replace that while loop and whatever consumers is with a loop polling a linkedblockingqueue

hiredman21:06:15

collapsing all the weird conditions into a single x = q.poll(); if(x == null) ... else ...

hiredman21:06:54

I wouldn't be surprised it changing it to a static initializer altered the timing enough that it exposed some races/deadlocks

sova-soars-the-sora21:06:53

It's 2020 and emacs is still the only text editor with an undo tree

noisesmith21:06:58

vim / nvim has had undo-tree for ages

3
noisesmith21:06:23

(as a plugin for ui, but the underlying tree structure was always there IIRC)

noisesmith21:06:22

yeah, confirmed, the tree is built in, the UI is a plugin

andy.fingerhut23:06:54

Do you actually use the tree structure of saved-in-memory Emacs undo state, then? Even with a graphical tool for exploring such a thing, I would personally feel a lot safer having those states in separate files, or at least separate revision-controlled branches in git or similar.

noisesmith23:06:32

for the vim version, the tree introduces a new node for each recovered state from the tree, no action removes an item from that tree

noisesmith23:06:53

I don't use it for long term differences, I use it when I realize something I deleted five minutes ago might be useful to restore

noisesmith23:06:27

and yeah I git commit often, so that is my "checkpoint"

gklijs05:06:33

IntelliJ also tracks history, not sure I would want it in a tree. Even removed files are easily set back from the history of the folder.

sveri07:06:47

Yep, eclipse does the same and I cannot remember not having been able to go back to an older state because the history was linear and not tree like.

sveri07:06:15

I wonder what the advantages of a tree like undo would be?

noisesmith15:06:35

a tree like undo means that I can undo, try changes, undo again, try other changes, undo again, try more changes, and all states since opening the file are still accessible

noisesmith15:06:52

I would never want to create that many git branches

noisesmith15:06:17

and I don't always know until later "oh, the fourth thing I tried is what I want here"

noisesmith15:06:39

finally, this means I never leave weird code in a file "because I might want to try it again later"

sveri15:06:07

I get it, thank you for explaining the advantages 🙂

gklijs16:06:04

I guess a three structure would help in that case. You can still do the same with IntelliJ. It's linear, but append only, so going back creates a 'revert' change.

noisesmith16:06:00

oh, interesting - so it's like event-sourcing editor changes :D

noisesmith16:06:24

(and thus the content of the editor is a materialized view...)