This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-04-05
Channels
- # architecture (2)
- # aws (1)
- # bangalore-clj (4)
- # beginners (97)
- # boot (35)
- # cider (8)
- # cljsjs (3)
- # cljsrn (35)
- # clojure (190)
- # clojure-dusseldorf (4)
- # clojure-italy (7)
- # clojure-poland (1)
- # clojure-russia (17)
- # clojure-spec (74)
- # clojure-uk (30)
- # clojureremote (2)
- # clojurescript (298)
- # code-reviews (18)
- # component (18)
- # cursive (18)
- # datascript (3)
- # datavis (2)
- # datomic (24)
- # dirac (17)
- # emacs (3)
- # garden (7)
- # hoplon (51)
- # jobs (2)
- # jobs-rus (2)
- # leiningen (4)
- # luminus (11)
- # mount (24)
- # off-topic (1)
- # om (32)
- # onyx (25)
- # pedestal (1)
- # powderkeg (9)
- # protorepl (6)
- # re-frame (11)
- # reagent (15)
- # spacemacs (6)
- # sql (35)
- # uncomplicate (1)
- # unrepl (74)
- # untangled (130)
- # yada (6)
so it looks like some kind of binding conveyance could be hackable with something like https://github.com/binaryage/cljs-zones
I get the feeling that agents without binding conveyance could still be useful to users. You'd just have to be more careful with what you pass to the agent.
Another interesting thing about building on shared atoms and agents, rather than static calls and core async, is that we have the makings of a ref datatype. We could then implement STM coordination on top of a pool of web workers operating on the "shared" data structures. This would abstract webworks and message passing away under the atom/agent/stm concurrency abstraction. You set up your agents to bang on your refs. Order the coordination of the results in a (dosync ...
and you've just eliminated a whole class of concurrency problems: both threading and message passing.
Clojure kinda "solved" concurrent programming on the jvm. Java has good primitives, but it's still easy to shoot yourself in the foot. Clojure made that safe. Web workers are unfortunately not ideal primitives for concurrency and it's still easy to shoot yourself in the foot. So I think an agent/stm stack for cljs would help webworker concurrency in both safety and usability.
And since cljs already uses MVCC data structures, getting stm to work on top should be workable.
@john you are ignoring one fact when comparing with the JVM and STM. you do not have shared memory, so there is no point in making it transactional.
right, there's a class of coordination that is not necessary when memory is only accessed in a single thread
the state is still protected by a single thread that can read and write to it, but from the perspective of other workers, they still need coordination
well, from the perspective of a thread, it's own state is safe, because only it can update its state while it is looking at it.
@thheller I don't know if transactions apply to this, but you might eg. want to guarantee that a worker and a main process see the same changes in the same order
some coordination is needed there, and a transaction seems a sane abstraction for it
that is, you might want a coordinated and uniform view of a shared set of facts, where no two processes see different orderings of state
eg. among a group of peers, exactly one should send a message to server X, and you might want them to decide amongst themselves which one does it
(contrived example, but I often have tasks like that between vms)
the canonical STM example is moving money from one account to another. You never want there to be a state where either account has no money or double. Similarly, you may have a situation where you never want two separate pieces of information, owned by different web workers, to ever have a state inconsistent with the state of other pieces of information owned by potentially other web workers
no shared memory: everything must be serialized back and forth (might not be possible)
a good comparison for this is zookeeper I think - it's a distributed store with guaranteed ordering of events
if every step touches the same data, yes, but that's almost never strictly needed
(maybe I'm just imagining how I would use this at this point - the coordinated data is a very small subset of the data I use, though it is vital)
similarly with zookeeper - it's not meant to be a db
@noisesmith zookeeper doesn't ever execute any code for you, so its a totally different thing
@thheller but I use zookeeper to decide which code to execute
@thheller my idea here is that a group of workers could have ordered and coordinated state (and therefore differentiate confirmed state and state mid transaction)
@thheller my point is that once we start introducing the need to coordinate the states of other webworkers, isn't STM a possible solution?
> In computer science, software transactional memory (STM) is a concurrency control mechanism analogous to database transactions for controlling access to shared memory in concurrent computing.
@john no that can't work
you have neither the power, nor the penalties, of that situation
How does A ensure that B and C do not end up in an inconsistent state, relative to one another?
inter process coordination, which can also use a transaction concept, but is the exact opposite of shared memory
it's distributed state
If A wants to take 10 dollars from B and put it into C, how can it do that without either B or C not having either 0 or 20 dollars?
via a transactional system
but STM is a specific kind of mechanic for a transactional system, that assumes shared memory access
no, it emphatically is not
seeya
in a distributed system of web workers, their single threadedness is only guaranteeing local consistency, similar to an atom in clj
I don't see the atom comparison, at all
for example, where are the retries?
the point of atoms is that you can change them without locking, and you retry if concurrent changes occur
that's literally why we use them
what I'm saying is that I don't see what having a single thread has in common with using an atom
no, no protection - just the ability to retry
it's a thing that doesn't exist when you only have one thread
you can still have read-write skew if you use deref
or reset!
in a web worker system, worker A and worker B can both access an atom in worker C and, because C is single threaded, the atom is also safe from concurrent access. In that way they are similar.
but they can't access that atom - they can get a message from c, but that memory they can't touch
right, but again, it's a question of: How can you get A to take 10 dollars from B and give it to C, without B or C having 0 or 20 dollars?
the distinction I am looking to preserve here is the one between the semantics you want, and the implementation that makes it work; STM is an implementation that relies on shared memory access, but the more general concept of transactions doesn't require this
you want a transaction, STM is an implementation of transactions that is impossible on this platform, so you need another implementation
one that involves coordination and ordering, rather than shared state
if the memory isn't shared, it's not STM
if that's our only disagreement, and you can't accept that, we can stop now
but if you go read about STM - it will agree with what I just said
STM is software transactional memory. the alternative is hardware transactional memory, AFAIAA
and I'm talking about using an implementation similar to CLJ's STM, which monitors locks on state and retries occasionally
STM is an algorithm that is only possible when memory is shared http://groups.csail.mit.edu/tds/papers/Shavit/ShavitTouitou-podc95.pdf
One can use STM to provide a general highly concurrent
method for translating sequential object implementations
into non-blocking ones based on the caching approach of
[6, 26]. The approach is straightforward: use transactional
memory to to implement any collection of changes to a
shared object, performing them as an atomic k-word Compare&Swap
transaction (see Figure 2) on the desired locations.
OK - you mention locking, STM doesn't use locking!
that's the entire point of it - and a distributed version of the algo would hopefully also not require locks...
right, which is not a lock, and doesn't use a lock
anyway, it's probably not helpful to get caught up on the semantics - neither of us think we are talking about something that happens in shared memory
but my point is that you need a coordination algorithm, and the stm algorithm assumes shared memory in order to operate - what you need is a coordination algorithm
consensus
right, the point is the bank account example and the mechanism to coordinate the consistency of states between two remote states.
so with STM the approach would be to attempt the operation and then commit if there was no concurrent modification
the problem in the distributed case is that there is no check to see if there was concurrent modification
(well, you can get that, but you need to build a lot of other stuff first before you get that)
it's no longer simple
what happens if a and b make changes simultaneously that overlap?
you no longer have ordering
But to ensure B and C are consistent, A can just keep retrying the alter until it has consistency
with STM you have ordering and first one wins
consistency with what?
the point is that STM can assume simple things like "this changed or it didn't" that are no longer simple in a distributed case
you need a proper consensus agorithm, and trying to translate somthing that can use immediate and undeniable ordering and access will lead to weird failures and excess complexity
with shared memory, whether a at point 0 = a at point 1 is a simple question
with a single computer, whether a happened before or after b is simple
once you have distribution, these become trickier problems. They have solutions, but the nature of the problems mean that STM can't just be translated directly
I do have a caveat here - this theory comes from the domain of networked systems where there are a lot less guarantees, it's slightly less complex in a typical web worker scenario
OK, right - but I still think you'll save some trouble by implementing transactions, and not trying to implement the precise STM model
you keep mentioning locks - neither STM nor atoms use locks fyi
no, this is the entire point
locking uses completely different algorithms
but STM and atoms can do things faster, because of the lack of locks
it's not just that - STM can be implemented without locks
I think CLJ uses an atomic that leverages a locking compare and swap. I could be wrong though.
compare and swap isn't locking iirc... I'll have to look this up
@john control-f on this page doesn't find any locks for me, it's all using compare-and-set ops https://github.com/clojure/clojure/blob/clojure-1.9.0-alpha14/src/jvm/clojure/lang/Atom.java
there's no locking in the AtomicReference API though... but OK
anyway, if you can change a value and know another thread did not touch it in between, you don't need a lock
oh wait, right - to do a compare and set on state shared by processes that don't share memory, you need a distributed lock - which makes me think there's some option better than compare and set
BTW, I'm synchronizing state automatically between parent owner and sub-workers with access to the shared atom, if that helps your mental picture
OK - that's already coordinated without STM
it's coordinating via an actor
the actor owns the data, and it negotiates changes based on messages
the atom is a store owned by the process that is essentially an actor brokering the atom's state
then you ask one of the parents to transact with the other on your behalf
no locks
but the heavy CPU bound problem belongs in this worker. It just needs access to the state.
you could imitate STM by creating a compare-and-set based on inter-process locks, but if your work is CPU bound I don't think you'd like the result
because STM implies retries
yeah, STM wouldn't be great for a large number of long running tasks. Especially built over a message passing infrastructure.
So I had a lot to say about this topic because I've been skunkworking a replacement for my current inter process coordination. What I'm coming to now is keeping a record built by event-sourcing modifications over a kafka channel (which guarantees ordering) - that way every client can subscribe to the same steam and therefore see the same data structure. A structure that needs to be kept relatively minimal, only for things that need coordination. I'm realizing that to get the semantics I want all I actually need is guaranteed ordering of events, each event describing a compare+swap on the coordination data.
if you need transactional coordination, and different parts of the data belong to different entities, you either need multi-process data locking, or some multi-hop commit process, where if any untouched server owns any of the committed data, you only commit if that server gives the OK... but even then you would need to build some sort of proof that you don't end up in byzantine feedback behaviors I think
or maybe there's something much simpler I'm missing here
Submit lock request to lock owner. Owner sends lock request confirmation. Owner sends all subscribers lock request. Once all subscribers confirm lock request to owner, owner sends lock request success confirmation to original requester. Now original requestor can send mutation request. Owner broadcasts new state to all subscribers. All subscribers report back confirmation. Owner sends mutation complete confirmation to original requester. Original requester sends unlock request. Owner broadcasts unlock request to all subscribers. Owner sends original sender unlock request success confirmation.
Then there's all the cases where confirmations come back negative, which all need to be handled gracefully.
Once those semantics are working, I'll look into optimizing by cutting back on requests. Batching lock and mutate requests together. Leaving "soft locks" on the state, so that the last requester to have a lock doesn't have to re-lock to send a mutation request. etc. As long as the correct semantics are preserved
given that none of the threads are interruptable, couldn't you just send a compare and swap with the old data and the new data in the request, and then the recipient could accept it if the current data is still the old?
still equal that is
If the owner receives a request, where the request has the old data and the new data, if the requesters old data matches the owners current data, then we know it's atomic?
then we know it can be confirmed
right, and we can't get bad states since we can't get our thread interrupted mid action
then you tell the caller the transaction failed
the tricky part is if there was data in the transaction belonging to multiple owners
then you need the locks and such again
I'm off for dinner - I'll probably be around later
Does anybody know how to find and delete unused imports in large clojurescript codebase?
Is there a repo where the new :npm-deps argument is used? @spinningtopsofdoom
I just don't get it, how it could be used
@danielgrosse You add it to your compiler options
But you have to run clojurescript from master
It wil download your npm depenciens for you automatically when you run the cljs build
@danielgrosse https://anmonteiro.com/2017/03/requiring-node-js-modules-from-clojurescript-namespaces/
will this be compatible with lein-cljsbuild
btw?
Until now I don't had it get running with lein
as opposed to the normal html modules, which is are a mix of "html for html veterans" and "html for bored students with a graphic arts minor"
I basically want a guide on what is cheap to render and what is expensive, what are the "gotchas", and what is the general strategy for laying out a document or component to be programmatically manipulated.
@torbjornvatn Yes if you have the correct clojurescript version
@danielgrosse You need to run from the master branch of the clojurescript repository to test it out.
I want to import the net module, but got the error
events.js:141
throw er; // Unhandled 'error' event
^
Error: ENOENT: no such file or directory, open 'net'
at Error (native)
How do I use the master branch of clojurescript? With leiningen?
Okay. The master branch solved the missing net dependency. But now I have another error. The library I import depends on some other libs. One of them ws
has devdependencies which are required in catch() {} try()
blocks. Now the compiler throws an error, that this dependency isn't available.
@tjtolton Here's a guide from Google for fast html rendering https://developers.google.com/web/fundamentals/performance/rendering/
oh nice! thanks, @spinningtopsofdoom
NP always glad to help 🙂
@danielgrosse are you compiling for the browser or for nodejs?
For the browser, but it seems to be a conflict with the library or my npm install.
Which library are your trying to use?
@spinningtopsofdoom in your clojurewest talk you talked about Closure Modules and how they are not supported in :none
, this might be interesting to you then https://github.com/thheller/shadow-devtools/wiki/ClojureScript-for-the-browser
@thheller For the shadow-build modules for :none
do they do something similar to what :main
does for standard ClojureScript builds?
but I also have a "compact" mode, which just concatenates all files together into the module file
I thought the ModuleManager only needed a uri to load and a namespace with setLoaded
called with that modules name to work.
If the modules under :none
load in all namespaces they contain it should 🍀 just work.
no the module file just contains a bunch of goog.require
which want to emit script tags
I created an issue for the problem i discovered while compiling an node module. http://dev.clojure.org/jira/browse/CLJS-1999
@danielgrosse looks like a Closure problem, looking at autobahn
I can see why it would have issues
what could be the cause of this?
There’s a jquery library that I’d like to use in ClojureScript (http://jvectormap.com/) for which I did not find a compelling alternative
is there an up to date guide on how to include such a dependency in a cljs + boot project?
@joelkuiper This guide is pretty solid and if you read & follow it carefully you should have a cljsjs-style jar in ~20-30min https://github.com/cljsjs/packages/wiki/Creating-Packages
thanks!
@danielgrosse that issue is fixed in Closure Compiler master
Nothing to do with CLJS
Thanks for the information.
I'm going a little nuts trying to integrate an ES6 library with a react dependency to play nicely with clojurescript. The basic javascript build doesn't produce a single .js file that I can run an externs generator on, and my attempts to produce one with webpack winds up giving me a big file that throws errors when I import it
I see closure compiler claims to be able to just import ES6 natively now, should I try just throwing the ES6 source at closure rather than trying to transpile it to ES5 first? Anyone got general advice for this sort of thing?
FWIW, the library I'm trying to import is here: https://github.com/toxicFork/react-three-renderer
hmm so the minified js of this looks like
!function($){var apiParams={set:{colors:1,values:1,backgroundColor:1,scaleColors:1,normalizeFunction:1,focus:1},get:{selectedRegions:1,selectedMarkers:1,mapObject:1,regionName:1}};$.fn.vectorMap=function(options){var …
would it be possible to generate externs for this?
basically, it’s a classical jquery plugin that I’d like to include
anyone know if it's possible to run multiple handlers on success/failure with https://github.com/Day8/re-frame-http-fx?
@joelkuiper: I'd try plugging it into this: http://jmmk.github.io/javascript-externs-generator/
@deadghost: what I've usually done for things like that is to have the :on-success
handler itself fire one or more new events via :dispatch
or :dispatch-n
. You might want to check in the #re-frame channel too.
@timgilbert hmm, doesn’t seem to do anything
@timgilbert thanks I'll check those out
would it be really bad if I just add them as scripts to my page 😛 ?
@joelkuiper, yeah, not sure, haven't tried it with jQuery stuff before. There's also this one as an alternative: http://www.dotnetwise.com/code/externs/
Just adding it to your page is IMHO a reasonable approach versus wrestling with google compiler, although you will lose the dead code elimination and other js optimizations
foreign-libs are never dead code eliminated so you can’t lose that. You can use the script tag. What you lose is the dependency handling and bundling.
@joelkuiper let me try...
@martinklepsch it’s the first time I try to do this
so any help would be welcome!
@joelkuiper hm. generating externs is not so easy.
I figured, it seems so simple, I want an heat map between country codes 😛
but down the rabbit hole we go!
@joelkuiper guess your best bet might be trying the new extern inference stuff or writing the extern file manually along the way
remember that you only need externs for the stuff you actually use so they don’t need to be exhaustive
or just add those files to my script header 😛
@joelkuiper I forgot to mention earlier that you need externs either way if you want to advanced compile stuff
I guess I can get away with just the minified js for now
adds to page load, but meh
@joelkuiper of your entire build?
no just that plugin
you can’t advanced compile your cljs if you access properties from non-externed JS objects
I forgot about that
I guess I can just try to write my own GIS lib that does this 😛
quick and dirty solution: pass :foreign-libs
compiler option with {:file “path/to.js” :file-min “path/to/minified.js” :externs “path/to/externs.js”}
and write externs as you go.
I assume you don’t use much more than the vectorMap
function. As long as you use string keys for properties besides that you’re on the safe side (strings aren’t touched by Closure)
jQuery.fn.vectorMap = function(opts) {}
could be a start for your externs file@borkdude mentioned that cljs-oops might also be an option, since we’re already using that
seems like a good start, thanks!
@joelkuiper yeah, I guess you could do something like (def vector-map (goog.object/get js/jQuery.fn “vectorMap”))
and use that function (as far as I understand thats basically what cljs-oops does)
depending on the number of jVectorMap specific properties your accessing that might be feasible… or not. But same problem with externs so you’ll have to bite the bullet somewhere.
Also just to clarify again: > I guess I can get away with just the minified js for now foreign-libs are never part of the advanced compilation pass. They are basically prepended to the build in dependency-order after your Closure compatible code has been optimized. That’s it. No processing done.
Any tutorial series that I can follow to create a simple game in cljs? Want to learn the process of development in Clojure and use figwheel. 🙂
Any game like flappy bird that was used to demonstrate figwheel in the introductory video.
@hit023 https://github.com/bhauman/flappy-bird-demo/blob/master/src/flappy_bird_demo/core.cljs
the flappy bird demo is open source and it’s ClojureScript AFAIK
so i’m still on the hunt for a good browser testing story for full stack Clojure. ideally i’d like to be able to target http://browserstack.com’s Automate (hosted Selenium grid), so i’m wondering if perhaps a node.js + phantomjs with ClojureScript type of thing which runs on AWS lambda is feasible
our team is 100% Clojure/Script, so i don’t want to use another language
clj-webdriver is, unfortunately, abandoned
Hey, how to get path/query
string's from goog.history.Event
.
(or how to routing with path/query params?)
there's a third-party library I'd like to wrap a series of functions around to provide uniform behavior with an equivalent JVM library