Fork me on GitHub
#hoplon
<
2016-06-11
>
thedavidmeister05:06:04

@mhr maybe i misunderstand, but should those reset! calls happen inside a dosync?

thedavidmeister05:06:09

you are currently asking state to update the page 3 times per undo!

thedavidmeister05:06:40

i would think, based on my limited understanding of what you’re trying to do, that you want past, present and future to all have a new value together

thedavidmeister05:06:58

it may or may not have an impact on what you’re seeing in performance

thedavidmeister05:06:37

but currently, you are moving the system temporarily into a state where items appear in both past, present and future simlutaneously

mhr05:06:03

@thedavidmeister: I'm just learning Clojure, so I'm unfamiliar with dosync

thedavidmeister05:06:11

it’s actually part of javelin

thedavidmeister05:06:26

when you do a reset!

thedavidmeister05:06:45

then it immediately updates all the cells and the DOM

thedavidmeister05:06:05

but, say you had...

thedavidmeister05:06:30

(def left (cell :foo)) and (def right (cell nil))

thedavidmeister05:06:52

and you rendered the left and right hand sides of your site based on that

thedavidmeister05:06:58

you could move something from left to right by doing

thedavidmeister05:06:17

(do (reset! left nil) (reset! right :foo))

thedavidmeister05:06:40

after the first call to reset!, :foo is in neither left or right

thedavidmeister05:06:58

so you’ll have nil in both sides of the page propagate through all the cells, and the DOM

thedavidmeister05:06:30

which might be harmless

thedavidmeister05:06:32

or might be really bad

thedavidmeister05:06:48

but if you (dosync (reset! left nil) (reset! right :foo))

thedavidmeister05:06:14

it will do all the resets together, and then propagate the final state of the world together

mhr05:06:59

that's nice

mhr05:06:38

so I just wrap my resets in dosync and it should automatically improve performance?

mhr05:06:48

how does it work?

thedavidmeister05:06:00

oh, i have no idea, lol

thedavidmeister05:06:08

it completely comes down to what you’re doing in the DOM based on state 😛

thedavidmeister05:06:22

i mean, it would have some impact on performance because you’re doing 1/3rd of the work in theory

thedavidmeister05:06:54

but i don’t know if “the work” is the bottleneck atm 😉

mhr05:06:58

it's a part of javelin? is this a modified version of the one in the clojure core? https://clojuredocs.org/clojure.core/dosync

thedavidmeister05:06:13

because it doesn’t exist in clojurescript

thedavidmeister05:06:32

it’s very much like what’s in here under “modelling sock transfers"

thedavidmeister05:06:06

what i’m saying is more about helping you avoid subtle bugs or other issues caused by your system being in an inconsistent state from time to time

mhr05:06:16

I was having some conceptual trouble writing it. I had to be careful in my ordering to make sure I wasn't overwriting information I needed in a later reset.

thedavidmeister05:06:04

that’s fair, it’s probably good clojure practice 🙂

thedavidmeister05:06:37

you could probably do something with a let too

mhr05:06:09

also what other clojure functionality is not available in clojurescript?

thedavidmeister05:06:41

well clojurescript is based on javascript

thedavidmeister05:06:45

so it’s single threaded

thedavidmeister05:06:11

and also, the javascript primitives and java primitives are different

thedavidmeister05:06:01

so if you (reduce + my-collection) in clojurescript it will do all the lovely javascript coercion stuff that + does in javascript

mhr05:06:15

hmm. I made the dosync change and it doesn't seem to have helped performance too much, actually

thedavidmeister05:06:26

you might want to do something like (reduce + (map int my-collection)) for example 😉

thedavidmeister05:06:39

@mhr have you got this up on github or something?

thedavidmeister05:06:47

i could pull it down and have a look

thedavidmeister05:06:17

it’s just… you’re assuming the perfomance issue is in the reset! itself but it could be anywhere really

mhr05:06:56

that's true. I really appreciate you reviewing my code, I'll put it up and put a link here, should take only a minute or so

thedavidmeister05:06:16

also, the way you have “past, present, future"

thedavidmeister05:06:25

could just be a vector with a “current position” index

thedavidmeister05:06:38

and undo would then just be decrementing the index by 1

mhr05:06:45

that's much smarter than what I've been doing

mhr05:06:49

I bet that would probably fix everything

mhr05:06:53

let me try that first

thedavidmeister05:06:07

hahaha it’s really impossible to say when only looking at one part of the system

thedavidmeister05:06:10

but you should try 🙂

mhr05:06:39

I'll go ahead and make the repo for you to look at in the meantime

thedavidmeister05:06:22

making things easier to understand and more atomic are good goals in their own rights

thedavidmeister05:06:30

even if the performance bottleneck is somewhere else

mhr05:06:01

yes. I'm simulating a vector with an index through a map, it's very convoluted. I'm not sure why I went for that.

thedavidmeister05:06:32

@mhr /shrug i guess because you’re learning a bunch of things at the same time?

mhr05:06:48

I remember now. I read http://redux.js.org/docs/recipes/ImplementingUndoHistory.html to see how others implement undo/redo and saw that they were using a map, so I used it without thinking about other approaches.

thedavidmeister05:06:34

well, there’s also the datomic/datascript approach

thedavidmeister05:06:48

where you model the state as a database, and every transaction is applied as a diff

thedavidmeister05:06:03

so your undo/redo is just working through diffs

thedavidmeister05:06:22

but um, probably don’t worry about that just yet

mhr05:06:50

I'm aware of datascript and datomic

mhr05:06:58

I considered doing the diff approach

mhr05:06:13

but thought it'd be simpler to model it as a list of future and past states

thedavidmeister05:06:37

it’s definitely simpler

mhr05:06:37

is it more efficient the other way?

mhr05:06:53

I assume it must be

mhr05:06:58

otherwise why would they make it that way

thedavidmeister05:06:24

and also, i just don’t think you should optimise for efficiency until you know what is inefficient in context of a specific problem

thedavidmeister05:06:36

if your entire application state is a number

thedavidmeister05:06:46

[1 2 3] will be far more efficient than a datascript database

thedavidmeister06:06:19

if it’s a complicated structure, then having diffs of the structure will be more efficient than having a vector with a complete representation of every state in it

thedavidmeister06:06:25

also, there’s nothing stopping you (actually it’s encouraged) from building your application out of a set of different cells that relate to specific functionality, rather than shoehorning everything into a global “state"

thedavidmeister06:06:37

precisely so you can make choices like this based on what you’re doing

mhr06:06:07

good points

thedavidmeister06:06:48

ok yeah, i’m seeing the slowdown in your app around the 400 element mark on my laptop

thedavidmeister06:06:57

i’ll have a look 🙂

thedavidmeister06:06:28

it’s interesting that the performance doesn’t degrade slowly, it very suddenly chokes

thedavidmeister06:06:32

so maybe it’s gc or something

thedavidmeister06:06:55

the issue is not in the reset @mhr

mhr06:06:06

interesting

thedavidmeister06:06:26

comment out these lines, i’m well into the thousands with no issues

thedavidmeister06:06:05

it’s something to do with reflowing the DOM i would say

mhr06:06:12

if I wanted to render the past and future as list items, though, how would I make that more efficient?

mhr06:06:53

I do want to be able to display undo and redo to the user in some tangible way for the app I'm going to build

thedavidmeister06:06:06

well the adding of DOM elements is very low level, it’s just calling javascript functions that add to the DOM with appendChild

thedavidmeister06:06:19

let me keep having a play

thedavidmeister06:06:51

i think we can safely say that it’s not the updating of the cell itself causing issues

mhr06:06:57

okay cool

thedavidmeister06:06:06

so it’s either something in for-tpl that we can investigate

thedavidmeister06:06:19

or it’s just a limitation of the way browsers add to the DOM

mhr06:06:24

I had map before instead of for-tpl

mhr06:06:35

switching to for-tpl had no effect on performance

thedavidmeister06:06:58

yeah, the main thing with performance between map and the tpl is adding and removing elements

mhr06:06:16

I figure it's probably a DOM problem

thedavidmeister06:06:28

specifically when the elements you remove reference a cell that is outside the map/tpl

thedavidmeister06:06:44

because the element is never garbage collected

thedavidmeister06:06:54

because it is still watching something that exists

thedavidmeister06:06:19

so you just end up with a pile of invisible elements in memory that are basically just a memory leak

mhr06:06:31

it's a shame to me that I can't use regular clojure functions, but I see why it's necessary

thedavidmeister06:06:11

you can use map for anything you want, but if you create elements that “dial out” to a cell that is not contained to that scope

thedavidmeister06:06:28

then… well it’s just how javascript works

thedavidmeister06:06:38

it’s doesn’t clean up anything that references other scopes

thedavidmeister06:06:04

so the tpl macros just make it so you can automatically re-use those elements

thedavidmeister06:06:12

rather than constantly creating new things

thedavidmeister06:06:03

but as the issue here is the browser slowing down just from repeated appends

thedavidmeister06:06:15

i wouldn’t expect to see much difference between a tpl and a map

thedavidmeister06:06:59

anyway, enough talking, i’m going to do some playing around 🙂

mhr06:06:38

the code with a vector and cursor becomes so much more elegant, it's not even close

mhr06:06:43

I find myself wishing for python's colon slice syntax, subvec doesn't cut it 😞

thedavidmeister06:06:10

@mhr so i whipped up a “best case” scenario using straight js

thedavidmeister06:06:43

so that only really slows down from gc on my laptop

mhr06:06:17

gotcha, so it's kind of unavoidable

thedavidmeister06:06:19

so there is something between that quite raw example and what is going on in your example

thedavidmeister06:06:28

it’s definitely faster

thedavidmeister06:06:39

i wouldn’t say it’s perfectly silky

thedavidmeister06:06:51

but it doesn’t hit the wall dramatically either

thedavidmeister06:06:58

it kind of drifts in and out of being fast

thedavidmeister06:06:01

in typical js style

mhr06:06:16

but you're not sure what's making that difference

thedavidmeister06:06:33

i just ran an experiment using raw js

thedavidmeister06:06:49

to see what the absolute best case looks like

thedavidmeister06:06:53

for the type of work that you’re doing

thedavidmeister06:06:01

which is “append a list item"

thedavidmeister06:06:29

my example is completely useless from a “build an application” perspective 😛

thedavidmeister06:06:45

it’s literally just incrementing a counter and adding a list item

thedavidmeister06:06:54

the counter and the number of list items aren’t even hooked up

thedavidmeister06:06:07

i just wanted to see how fast the DOM could move

mhr06:06:37

it could be a problem with how Hoplon's HLisp works

thedavidmeister06:06:01

i think i’ll do some profiling

mhr06:06:05

I don't have any idea of the internals of HLisp, but that's my best guess

thedavidmeister06:06:11

well that’s the thing

thedavidmeister06:06:15

there aren’t really any internals

thedavidmeister06:06:47

it’s not like Hiccup where you make a data structure, then render it, then compare the render against the current state and apply the render to the DOM

mhr06:06:58

how would one do profiling in CLJS, by the way?

thedavidmeister06:06:15

as soon as you call a HLisp function it appends the child to the DOM

thedavidmeister06:06:37

if you have a cell=, that’s just an event handler watching some data

thedavidmeister06:06:10

my understanding of HLisp is that it should function very similarly to that gist i posted

mhr06:06:41

that was my understanding too, from reading the docs, but I wondered if I was mistaken

thedavidmeister06:06:55

pretty sure that’s it

thedavidmeister06:06:01

but @micha would know best

thedavidmeister07:06:27

i’m not really an expert on profiling cljs

thedavidmeister07:06:33

i’m actually pretty new too 🙂

thedavidmeister07:06:41

but i’m just going to have a crack at it with the regular profiler

mhr07:06:20

what do those percentages mean?

mhr07:06:18

as in what are they measuring? why are several items at 80%?

mhr07:06:56

I've not made any major JS apps either, so I've not used regular JS profiling tools in the browser before either

thedavidmeister07:06:00

ooh ok, well there’s total and self

thedavidmeister07:06:23

here’s another one

thedavidmeister07:06:33

total is the time for that function and all the functions it calls

thedavidmeister07:06:50

so… a function might be quite fast and have a big time because something it calls is slow

thedavidmeister07:06:00

self is just the time of the function itself

thedavidmeister07:06:16

but it’s cumulative tie

thedavidmeister07:06:33

i could call a fast function 10k times and it be slower than a slow function i call once

thedavidmeister07:06:57

i think that means that the bit that is happening in javelin is fast

thedavidmeister07:06:05

and the bit that is happening in jquery is slow

mhr07:06:02

looks like that, yeah

mhr07:06:45

perhaps the developers of javelin should consider switching to raw javascript instead of jquery...

thedavidmeister07:06:30

i don’t think javelin does anything with jquery

thedavidmeister07:06:54

if you just click on the “inc” button, you can’t really break it

thedavidmeister07:06:20

i think that at some point, the “add and update” function becomes slightly slower than the keydown repeat interval on the enter key

thedavidmeister07:06:38

so you end up with a backlog of events queued up

thedavidmeister07:06:32

i don’t know what the key repeat rate is...

thedavidmeister07:06:23

it seems to be about 40ms

thedavidmeister07:06:44

based on a very unscientific counting i just did 😛

thedavidmeister07:06:40

@mhr i definitely think that maybe someone like @micha might be able to help more with the actual cause of the slowness

thedavidmeister07:06:47

but i’ll see if i can help with a throttler

thedavidmeister07:06:06

as long as you never go past that 30-40ms threshold you won’t get the runaway queue

thedavidmeister07:06:15

so if you just limit the speed at which someone can click to like, 100ms

thedavidmeister07:06:18

you should be fine

mhr07:06:25

am I able to use a sleep function?

thedavidmeister07:06:39

sleep isn’t really a thing in javascript

mhr07:06:59

how would you limit the speed, then?

thedavidmeister07:06:23

either a throttle or a debounce

mhr07:06:31

I'm not sure what you mean by that

thedavidmeister07:06:36

in your case, you probably want a throttle

thedavidmeister07:06:57

i’m just googling to see if someone has made a convenient way to throttle in cljs 🙂

mhr07:06:17

that's a helpful idea, thanks for the ti

thedavidmeister07:06:26

it’s actually very common to run into this kind of issue in js

thedavidmeister07:06:34

when you spam many events in quick succession

thedavidmeister07:06:47

the examples in that article are resizing the window and scrolling

thedavidmeister07:06:21

but holding down a key for 10s+ and responding to every key repeat is another example

thedavidmeister07:06:17

i’m actually thinking that it could be cool to include a throttle and debounce in javelin @alandipert @micha ?

thedavidmeister07:06:34

it could nicely complement dosync

thedavidmeister08:06:03

it sort of speaks to that missile command example you told me about @micha

thedavidmeister08:06:25

you could have a dosync that is time based instead of argument based

thedavidmeister08:06:51

or maybe it’s on the cell level...

thedavidmeister08:06:57

a cell that you could update as often as you like

thedavidmeister08:06:20

but will only propagate its values at most X times per Y time span

thedavidmeister08:06:53

@mhr i think this just about hits the extent of my knowledge with javelin/hoplon for what you’re looking at

mhr08:06:32

I really appreciate you taking the time to help me though!

thedavidmeister08:06:57

@mhr i’d suggest you just keep doing what you’re doing, and assume that you can drop in a throttle at some later date if it’s really needed

thedavidmeister08:06:20

realistically, even someone typing at 100 wpm is still less than 1 letter per 100ms

mhr08:06:46

it's really late where I am, so for now, I will keep doing what I'm doing until tomorrow when I can look at my code with fresh eyes. I'm almost done converting it to vector + cursor form. it's so much better and cleaner.

mhr08:06:18

that's true, but it's not an unexpected use-case that someone would want to spam from their keyboard too

thedavidmeister08:06:44

no, but you’re not obligated to reflow the DOM in real time

mhr08:06:57

that's true

thedavidmeister08:06:01

as long as your underlying state (which we established is easily fast enough to keep up)

thedavidmeister08:06:10

has everything they typed

thedavidmeister08:06:21

you can throttle how often the DOM responds to that state

mhr08:06:30

it'll be eventually consistent, yeah

mhr08:06:34

good point

mhr08:06:38

well I'm going to bed now

mhr08:06:49

thanks for all your help with profiling and code review!

thedavidmeister09:06:04

@mhr i did this as a quick and dirty experiment

thedavidmeister09:06:07

(def state (cell {:past [] :present 0 :future []}))
(def state-t (cell nil))

(defn sync!
  []
  (reset! state-t @state)
  (with-timeout 100 (sync!)))
(sync!)

(def counter (cell= (:present state-t)))
(def past (cell= (:past state-t)))
(def future (cell= (:future state-t)))

thedavidmeister09:06:56

it definitely helped

thedavidmeister09:06:07

but again, bottomed out around 1200 elements for me

thedavidmeister09:06:20

i’ll see what other people in chat think

thedavidmeister09:06:25

i’m 90% sure it’s something in the DOM manipulation and not javelin, but i don’t know why the “raw JS” DOM updates were so much faster than those triggered by Hoplon

jjttjj16:06:43

@micha: is there a term to describe the indentation style you use? Know of any emacs tools to make it happen magically by any chance?

micha19:06:20

@jjttjj: i use vim, with the guns/vim-clojure plugins

micha19:06:15

i add some lispwords to vi's built-in lisp stuff: lispwords+=page,cell-let,this-as,add-watch,do-watch,elem,alter-var-root,vary-meta,alter-meta!,task-when,specify!,GET,POST

micha19:06:39

that configures it to indent those like macros

jjttjj19:06:56

@micha: i mean like the aligning of things in vertical columns... Do you just do that manually?

jjttjj19:06:00

(ns hoplon.test
  (:require
    [cljs.test      :as t]
    [goog.string    :as gstring]
    [clojure.string :as string]
    [hoplon.core    :as h :include-macros true]
    [javelin.core   :as j :include-macros true])
  (:import
    [goog.string StringBuffer]))

jjttjj19:06:16

like how the :as's are aligned

micha19:06:31

i do that manually, but emacs does have a pretty good library for it

micha19:06:06

i've seen my friends at work using it in spacemacs

jjttjj19:06:58

oh cool, of course i just find it now, looks like it's in clojure-mode now

jjttjj19:06:42

awesome, i had it available this whole time haha

micha19:06:52

@thedavidmeister: here is a (misnamed) thing to add throttling to a cell: https://gist.github.com/micha/fa95fc3731c29abd71ff

micha19:06:01

@jjttjj: what's it called?

jjttjj19:06:16

it's just built into clojure-mode now

jjttjj19:06:41

C-c SPC on a section of code will align things magically