Fork me on GitHub
#clojure
<
2016-12-30
>
tbaldridge02:12:38

@mingp: the key reason I can't use mount is its view of singletons everywhere. I often need many instances of the same component. And that runs against the interned nature of vars. Also look into the uses of var alteration in mount, last it looked it made use of global mutation at a fairly low level. That's something I try to stay away from.

tbaldridge02:12:50

Not saying to with Component, there's other options like Integrant (is that the name?) that I think may be better than Component as they take a data-first approach

mingp02:12:40

@tbaldridge What do you mean by "many instances of the same component"? And how does that differ from having differently named components with same/similar implementations shared through reuse?

zentrope02:12:14

I've used re-usable components, such as a subscriber that takes the name of a topic. It then gets passed to other components that need to handle those messages.

zentrope02:12:02

Actual pub/sub functions are not in the same namespace as the component, but they take the instance as a first param and operate on it.

tolitius02:12:21

@tbaldridge: I don't think you're being fair, since we already discussed (https://www.reddit.com/r/Clojure/comments/41gph5/swapping_alternate_implementations_with_mount/cz3azes/) that mount has two modes: where states are kept in vars and where states are kept in derefables.

I often need many instances of the same component
usually comes down to I have the same start and stop function (i.e. lifecycle) and I need two components that use these start and stop with different configuration. Nothing stops you to do that in mount. More of deref'ables and ClojureScript support: https://github.com/tolitius/mount/blob/master/doc/clojurescript.md#managing-state-in-clojurescript

tbaldridge02:12:48

Sorry it's been a long time

tbaldridge02:12:19

So yes you can hack around it, but namespaces don't take parameters

tbaldridge02:12:16

So imo any var based approach is going to be limited by that fact.

gfredericks02:12:42

because you can't write generic code?

tolitius02:12:49

@tbaldridge: it's a bit "too philosophical", and I don't think @ or using different components that use the same lifecycle functions (that do take parameters) is "hacking around"

tbaldridge02:12:06

(sorry was eating supper)

tbaldridge03:12:45

Not sure that it's really that philosophical, Clojure is a functional language that prefers functional purity over global mutability. In many cases a deref @ implies mutable state. Mount is no exception. So I find it very odd to propose that we mix mutability through our app. It's a out-of-band input to every function that uses @ and every function that calls them.

tbaldridge03:12:11

I guess that's the part I don't understand about Mount...why would I use so much mutability for something that can be solved by adding a new param to every function.

bbloom03:12:57

i haven’t looked at mount at all

bbloom03:12:21

but i will say that i use mutability (especially thread-bound vars) relatively frequently when they are appropriate

fabrao03:12:43

anyone that worked with midje?

tbaldridge03:12:00

@bbloom sure, but would you wrap every database connection, every server, every message queue, everything in a dynamic var?

tolitius03:12:01

@tbaldridge: both Mount and Component help managing stateful resources, that by its nature are mutable. a database connection? mutable. a network socket? mutable, etc. so, yes "`@` implies mutable state" is true, because the state that mount manages is mutable.

bbloom03:12:11

i think it is a mistake to ignore vars for philosophical reasons, or assume they are only for dev time

bbloom03:12:35

@tbaldridge no, but then again i still haven’t been satisfied with any solution thus far for managing processes & lifetimes in clj

tbaldridge03:12:39

@tolitius sure, the connection is mutable, but my reference to it doesn't need to be

bbloom03:12:45

i feel like there’s a missing abstraction, especially when it comes to failure

tbaldridge03:12:12

@bbloom I agree, but I very much disagree that the answer to all that is further overloading vars/refs

bbloom03:12:29

not going to argue there, since again, i haven’t looked at mount

bbloom03:12:45

i will note that while functional code likes the iteration strategy of re-def-ing functions, imperative code, including services, prefers a different strategy

fabrao03:12:55

I´m trying to mock a function, is there any way to use matcher for any kind of parameter in provided?

tbaldridge03:12:58

But I'm more and more in favor of Integrant: https://github.com/weavejester/integrant it's all data, and the library is 200 lines (about 1/3rd of which is comments) ...that's a major win in my book

bbloom03:12:12

in lisps of yore, you re-def a var, much as we do today, but in forth you’d make a “mark”

bbloom03:12:50

i want something like how he uses marks in there

bbloom03:12:03

let me see if i can find it to save you an hour

tbaldridge03:12:15

@bbloom to be honest a real effect system would be the best answer to all this 😉

bbloom03:12:15

(although it is worth watching)

tbaldridge03:12:30

(My kingdom for a Lisp-Eff-on-the-JVM)

tolitius03:12:30

ok, so we are ok with "`@` implies mutable state". now to why var/deref: these states should be able to be started and stopped. one difference from Component or Integrant is that Mount allows restarting parts of your "system", one of many components could be restarted while the system is running. Mount is not a full app buy in, but a library.

fabrao03:12:32

@tbaldridge like this `(provided (ssh/execute ...ip... ...porta... ...usuario... ...senha... [...anything...]) => true)`

bbloom03:12:39

about a minute or two from there

bbloom03:12:43

he compares to slime etc

fabrao03:12:54

inside the

execute
I have the ...anything... fixed as as string "get system status"

tbaldridge03:12:12

@fabrao I think you're testing at the wrong layer. I would ask...why do you care that that function is called with those arguments? I'd wager you care about some other side-effect...test that instead. If you test that a function is called with sepcific params, then you're in a bad place when you refactor someday and stop calling that function.

tbaldridge03:12:47

@tolitius all of these libs can be used as little or as much as you want. None of these demand full buyin

tbaldridge03:12:23

@tolitius and they idea of half of my system suddenly restarting underneath me scares me. How does Mount keep that thread-safe?

fabrao03:12:29

whell, I´m mocking ssh connection inside other 2 functions, I think it´s usefull for now

fabrao03:12:12

but the midje is complain that I have to fix the string in provides, so I did it with

with-redefs

tbaldridge03:12:38

@tolitius actually I'd love to see some sort of documentation on thread-safeness of Mount...how does that work (not an accusation, I'm actually interested)

tbaldridge03:12:49

@tolitius eh, I'd disagree with Stuart there I think he was being to unkind to himself. The very system we built Component for adopted the library over the course of several months. Whole subsystems existed fine without it while other modules used it.

tbaldridge03:12:52

@tolitius that's a great point you made in Reddit. I'd argue that if Function A calls B and B needs DB then A needs to hand it a DB. That's just basic functional programming. Any other pattern introduces out-of-band communication and that just leads to all sorts of pain. Just look at the implicit parameters mess in Scala.

tbaldridge03:12:33

@tolitius but still I'm wondering about thread safety...how does that work? Can I have two threads refer to two different DBs at the same time? I don't see that by looking at the mount source, but maybe I'm missing something.

tolitius03:12:56

@tbaldridge: > they idea of half of my system suddenly restarting underneath me scares me not suddenly, but manually, desirably. > I'd love to see some sort of documentation on thread-safeness of Mount this could be a great idea, what questions are you looking to answer? i.e. mount creates components that can be started and stopped. once started it is up to the start function (which is a custom function that lives anywhere) that, if needed, takes care of multiple threads accessing this component.

tbaldridge03:12:52

So that's my point, dynamic vars in Clojure are thread-bound...which is very important as it allows each thread to not mess up another

tbaldridge03:12:32

So it sounds like Mount puts it on the user to manage thread safety.

tolitius03:12:16

> two threads refer to two different DBs I think different is a key here, which means these are going to be different components, and two threads would not be racing for the same reference > it sounds like Mount puts it on the user to manage thread safety I think it is fair to say that, since mount just creates components using custom / user supplied functions, and is able to start / stop / swap (for testing)

tbaldridge03:12:54

but since components = vars in Mount doesn't that mean I now have to create code that matches my architectural state?

tbaldridge03:12:42

So now the structure of my application must now match the vars/namespaces of my runtime, which in turn means I have to tie my architectural state to the file layout of my app. If I want to connect to DB2 and DB1 at the same time, now have to make that model match the Clojure vars and my CLJ code.

tbaldridge03:12:51

This IMO is exactly what Rich was talking about when people asked him about Spec's use of a separate DB for specs...vars are so overloaded it's time we stopped the insanity.

tbaldridge03:12:18

Spec could have been designed so that specs mapped to vars...it's not and there's a reason, I'd submit that same reason applies here.

tolitius03:12:57

I think it is better to define these DBs: they are resources. If you need two resources: DB1 and DB2, these would be two different components / states that will be referenced differently, since the code needs to differ DB1 from DB2. I am not sure I follow the "fallacy" argument, maybe you can give a more concrete example.

bbloom03:12:19

i think spec made the right call by having a separate env, but the unfortunate side effect is that you suddenly have a second class citizen

bbloom03:12:29

for example, there is no spec-equiv of ns-unmap from what i can tell

bbloom03:12:43

however, ns-unmap sorta sucks anyway b/c it’s an imperative operation that you can forget to do and wind up in a weird situtation

bbloom03:12:54

again, i’d rather something like forth’s marker system

bbloom03:12:37

clojure’s namespaces are in some senses too reified and simultaneously not reified enough

tbaldridge03:12:28

@tolitius to paraphrase Rich, vars exist to give code names. Then we added doc strings, then we added inlining data, etc. Now people want tests attached to vars, now they want examples attached to vars...

tbaldridge03:12:39

Mount adds state management to vars.

tbaldridge03:12:16

All these things could be stored in external DBs. And pretty much all of them should be.

tbaldridge03:12:31

"extrernal" = not in the namespace registry.

bbloom03:12:58

@tbaldridge: there’s two elements of this - 1 is whether the data is “owned” by the var, a la metadata including doc strings, :inline, etc

bbloom03:12:05

the 2nd is whether there is a unique namespace

bbloom03:12:26

those are independent dimensions - i agree that further overloading in the first sense is clearly bad

bbloom03:12:34

the second depends largely on your needs

tbaldridge03:12:55

Absolutely...the spec problem is a lot like this discussion about Mount. Many people have asked for specs to be parameterized by a DB. That's still "under consideration" according to Stu.

bbloom03:12:22

@tbaldridge is there some discussion of that somewhere i missed?

tbaldridge03:12:27

So literally one of the first things people asked when they encountered Spec is...."how do I have this not be mutable and/or have multiple things with the same name based on context"

tbaldridge03:12:23

IIRC Rich mentioned the var overloading as well in one of his talks.

bbloom03:12:57

i think the biggest justification for having a separate logical namespace for specs is that there are LOTS of them

bbloom03:12:16

so conflicts would be plentiful

tbaldridge03:12:27

Not really...the justification I've heard from Stu and Rich on several occasions is "stop overloading vars!"

bbloom03:12:28

but for “systems” or “components", it’s less clear that’s the case

bbloom03:12:50

that plea is not a justification

bbloom03:12:01

what is the motivation for the plea?

tolitius03:12:10

> Mount adds state management to vars I like vars. They serve a "naming" purpose, yes, but they also serve "referencing" purpose. I don't think Mount adds anything to vars from the "language perspective", as do doc strings and meta data. It's a library which uses vars just for naming and reference purposes. The fact that the state behind these vars can mutate is defined by the nature of this state: i.e. a database connection is mutable and the problem mount attempts to solve: adding a lifecycle to mutable external resources.

bbloom03:12:32

again, i’m drawing a distinction between the two problems of ownership of metadata and name collisions

tbaldridge03:12:03

@tolitius I think we can probably boil down our argument to this...and correct me if I'm wrong:

tbaldridge03:12:16

I strongly believe that mutable vars are a design-time feature. Any mutation of vars or making things appear to mutate vars should be avoided. Dynamic vars get a free ride since they are thread-safe but I still don't like them

tbaldridge03:12:43

@tolitius I suspect we disagree on that statement ^^

bronsa03:12:04

there's not a 1:1 mapping between vars and specs, having unbound vars created just to hold specs would feel much weirder to me than having a separate registry

tbaldridge03:12:33

because vars are made for code (there's a meme in that statement)

bbloom03:12:35

@bronsa who says they’d be unbound? it could return the spec itself

bbloom03:12:57

although it’s much more likely that specs will be recursive, so you’d have a fair number of (declare foo bar baz) things

bronsa03:12:30

@bbloom wouldn't work for fdefs

tbaldridge03:12:47

And then some vars would contain specs, and some would contain stuff that has spec metadata (sorry, but....ick)

bbloom03:12:00

@bronsa ah yes, good point

bronsa03:12:28

well, what @tbaldridge said. fdefs are just an instance really

bbloom03:12:20

side note: i have a toy interpreter lying around somewhere in which environments are like prolog databases. clojure’s vars essentially map to a term like evaluation(sym, value) and other stuff like docstrings etc are similar docstring(sym, “blah blah”)

bbloom03:12:36

you’d just add spec(sym, …) as well

bbloom03:12:56

the result is sorta like having datomic/datascript as the implementation of namespaces 🙂

bronsa04:12:31

that's interesting, it sort of implies that the current namespace system is just one level too specialized

tbaldridge04:12:32

Are they immutable though?

bbloom04:12:07

there’s a thread-local binding of the current environment

bbloom04:12:46

the value within that binding is immutable, yes

tbaldridge04:12:22

I had a variant of Pixie running on a fully immutable effect-based namespace system once...it worked even if it did murder the JIT.

bbloom04:12:46

my thinking on first-class environments, fexprs, etc has evolved dramatically

tolitius04:12:02

@tbaldridge: I agree with both of your statements in theory. But since we are a bit more practical then IO / read write state transformer monad fans, I do accept and embrace mutation where it is needed. As we talked about earlier, and where this conversation has started: "`@` implies mutable state" => mount has a mode where all the APIs are the same, and no vars are mutated, component state lives in an atom like structure: https://github.com/tolitius/mount/blob/master/src/mount/core.cljc#L107-L116

tbaldridge04:12:01

How often is that dereffed? In other words, if I reset that atom, when does it show up in the functions that use it.

bronsa04:12:28

uh, "`@` implies mutable state"?

tbaldridge04:12:34

From my understanding of that code it's dereffed every time something uses the component, or at least that's the pattern.

tbaldridge04:12:53

@bronsa it "often" does was my original comment 😉

tbaldridge04:12:11

Promises, deffered, etc. excluded

bronsa04:12:21

fair enough :) sorry I haven't read the backlog

tbaldridge04:12:30

It's probably not worth it 🙂

tolitius04:12:18

@tbaldridge: > How often is that dereffed? whenever the "state behind the name" is needed: i.e. (store @db data) > if I reset that atom, when does it show up in the functions that use it you don't really reset! it directly, it only gets reset whenever (mount/start) / (mount/stop) are called. Since it has CAS semantics (i.e. atom), other functions will see it whenever the change is applied (it is either started or stopped)

bbloom04:12:22

so in terms of syntax and (logical) namespaces, the defstate macro doesn’t seem insane

bbloom04:12:29

the singleton and thread safety issue does tho

bbloom04:12:33

but i’m not sure what to do about it anyway

bbloom04:12:46

b/c to do much better would require a fair amount of integration with the repl or whatever

tbaldridge04:12:13

@bbloom that's it though...it doesn't need to be this way, we could just pass the DB into the function as a parameter.

bbloom04:12:32

i’ve always said that clojure lacks static implicit parameters

bbloom04:12:54

b/c the biggest reason people use dynamic vars or abuse top levels is for convenience

tbaldridge04:12:59

In one approach the code may be a bit more verbose, but I'd take that any day over any sort of implicit parameter. Implicit parameters hide information. Information I now have to keep in my head.

bbloom04:12:34

it’s more than that tho

bbloom04:12:41

it’s about non-local changes

bbloom04:12:53

in my systems, most functions do in fact take a ctx parameter

tolitius04:12:03

@bbloom: the idea behind a singleton is that an external resource is a singleton. as to thread safety, I think it depends on the usage. The intended usage is to start and stop components of an application. Usually, in 99% of time, it is a single call to (mount/start) and after everything is done (i.e. shutdown hook, end of life), a call to (mount/stop)

bbloom04:12:05

but if you need to get the ctx to someplace new, you have to make a cross-cutting change to every function in between

bbloom04:12:05

in terms of the “open systems” idea, implicit parameters give significant expressivity

bbloom04:12:17

perf sucks tho 😛

tolitius04:12:27

@tbaldridge > we could just pass the DB into the function as a parameter you still do that with mount. it's a choice that you have. if you keep all these resources at the edges of your app, they just get passed into functions as arguments

bbloom04:12:31

at least w/ first class functions w/o a type system

tbaldridge04:12:07

@tolitius then why store them in a var at all in that case?

bbloom04:12:36

in terms of clojure as it stands today, i agree with tbaldridge in terms of “just make a context parameter” and also make that the default approach

bbloom04:12:47

that said, i’ll admit to abusing top level state a fair bit during development

tbaldridge04:12:16

sure, we all do. And I use dynamic vars quite often, but more as a global accumulator when I'm using the return value of a function for something else.

fabrao04:12:30

Well, I use mount in many projects. I´ve never had to change the

defstate
, I use other things to do this, it´s read-only, and in tests I use before -> mount/start and after -> mount/stop

tbaldridge04:12:06

@fabrao that's not really what we're discussing. defstate creates state, and that state is mutated whenever you call start/stop

bbloom04:12:57

my bigger problem w/ components/subsystems in clojure is dealing with failure - missing log output or silent exceptions and the like are KILLER at dev time

tolitius04:12:37

> then why store them in a var at all in that case? (var or an "atom") because you need to get to these references somehow. In Clojure, we get to references by (require [app.foo :as foo]). With mount this stays. And there is no IoC as in Spring / Component, everything still Clojury: you need a reference in this namespace? require it.

bbloom04:12:39

i’ll also say that some things really are singletons & it’s perfectly valid to use dynamic vars in that way

bbloom04:12:49

like the current database may not be a singleton

bbloom04:12:58

but the connection pool for a particular database library might be

bbloom04:12:16

instead of passing a connection around, you may pass around the name of the db for example

bbloom04:12:33

personally, i prefer systems like that over object-oriented ones that pass around little closures

bbloom04:12:49

tbaldridge ^^ that’s another reason i frequently abuse vars w/ state in them

tolitius04:12:01

@bbloom: interesting (about failures). could you give an example of a silent failure that behaves any differently when using components/subsystems?

bbloom04:12:49

@tolitius just the other day i did something like (future (while true (handle (.take q)))) and the future failed, silently of course

bbloom04:12:02

but it had been working for hours, so i forgot i made that short little hack

bbloom04:12:35

in hindsight, i guess i should have just used async/thread or whatever

zentrope04:12:53

Would it have been noticed if you'd added an uncaughtExceptionHandler?

tolitius04:12:58

did you have:

(Thread/setDefaultUncaughtExceptionHandler
    (reify Thread$UncaughtExceptionHandler
      (uncaughtException [_ thread ex]
        (error ex (str "uncaught exception in thread: " (.getName thread))))))
in place

bbloom04:12:13

nope, b/c nrepl won’t show that output

tolitius04:12:19

and how would it behave differently in "components/subsystems" solutions?

bbloom04:12:33

well, what i want is an erlang style supervision tree

tolitius04:12:43

ah.. repl time (missed that in your description)

bbloom04:12:51

… are there other times? 😉

tolitius04:12:33

so say, you used Component, how would this silent error manifest itself differently?

tolitius04:12:52

just trying to understand this: > my bigger problem w/ components/subsystems in clojure is dealing with failure - missing log output or silent exceptions and the like are KILLER at dev time

tolitius04:12:47

you probably mean that there is no supervisor tree Erlang like solution that would keep all this components supervised in Clojure, right?

bbloom04:12:49

well w/ component or whatever i’d still have to start the thread somehow

bbloom04:12:11

i haven’t used component in anger

tolitius04:12:25

I have and wrote mount 🙂

bbloom04:12:26

i dunno what it does or doesn’t do in this regard. i just find the primitives lacking for async errors

tolitius04:12:47

I see. that's good feedback

tolitius04:12:35

although I don't think the scope of the problem (that Component or Mount attempt to solve) included async supervision, but more of a state organization and lifecycle management

tolitius04:12:46

but it is an interesting point nevertheless

bbloom04:12:05

supervision is lifecycle management

bbloom04:12:23

these are deeply intertwined concerns

tbaldridge04:12:39

Async error are really hard

bbloom04:12:47

and really painful 😛

tolitius04:12:39

yes, I can feel the connection. although this would require a lot of "different vertical" thinking time

bbloom04:12:04

what do the erlang folks do about module-level state?

tbaldridge04:12:14

The best system I've worked on in that space just killed the entire JVM when it hit a bad enough error. Worked quite well in a distributed system. But that was a rather extreme way to handle errors

tbaldridge04:12:26

All modules are immutable

bbloom04:12:43

killing the JVM wouldn’t be so bad if JVMs were cheap and nestable, but they aren't

tbaldridge04:12:20

You have to send a upgrade message to a actor and the it bootstraps itself in the new module

tbaldridge04:12:39

In Erlang all your state is in locals 😉

bbloom04:12:00

i’m a fan of this idea too: http://250bpm.com/blog:71

bbloom04:12:22

oooh right, but gen_server or whatever supports that, right?

tolitius04:12:54

the state in Erlang is also often kept in mnesia

bbloom04:12:06

cancelation and shutdown are really frickin' hard when you can’t just kill a process tree

tbaldridge04:12:12

Which is why we started with killing the JVM then added exceptions to commonly known "safe" errors

bbloom04:12:33

yup, i’ve read armstrong’s thesis 😉

bbloom04:12:13

i’ve written a fair amount of Go, and it’s quite apparent to me that CSP is totally busted for shutdown and failure

bbloom04:12:31

at least core.async will GC a deadlocked goroutine

bbloom04:12:38

@tbaldridge is there a decent default solution for printing errors in core async threads?

tolitius04:12:55

especially odbc:start(), odbc:stop() and odbc:connect(...)

drewverlee04:12:40

@bbloom > yup, i’ve read armstrong’s thesis Whats the name of the thesis?

bbloom04:12:48

@drewverlee "Making reliable distributed systems in the presence of sodware errors” - http://erlang.org/download/armstrong_thesis_2003.pdf

bbloom04:12:30

odd - the PDF renders that as “software”, but copy-pasting it as “sodware"

tbaldridge04:12:58

That's because Erlang message boxes are mutable

tbaldridge04:12:38

So stuff like the isn't driver there is storing the mutable state in an actor

tolitius04:12:15

right, just interesting to see how some of the external resources are accessed in Erlang: directly by referencing other modules (given that they are exported of course).

tbaldridge04:12:42

I'm interested in what that start fn does as the only mutable thing in Erlang is actor message boxes

tolitius04:12:18

@bbloom: I am thinking a bit more about enabling component lifecycle (i.e. start / stop) and supervising their runtime state, and I think they are in fact different concerns. "managing lifecycle" would be an overloaded phrase (used by me 30 min ago 🙂 ), so I would say mount/component enable / provide start and stop lifecycle events to components. It feels that something entirely different could then truly manage or supervise these component at runtime, and use mount/component to restart the ones that were stopped due to a failure. with mount it would be really simple since it is able to restart any part of the application i.e. (mount/start component1 component2)

bbloom04:12:43

the problem is that the “dependency graph” and the “supervision graph” can/should largely coincide completely

bbloom04:12:53

these things need to cooperate

tbaldridge04:12:24

And once you start automatically restarting things you really have to think about thread safety, so all functions have to become channels, or you deref state once, or something like that

tbaldridge04:12:53

That's a conversation I'd love to have since I've sat and thought about it for more hours than I can count, and I still hate all of my ideas.

tolitius05:12:17

@bbloom sure cooperate, yes. but it could be solved (and it feels right if it is solved) by two different libs @tbaldridge: > I'm interested in what that start fn does as the only mutable thing in Erlang is actor message boxes would not call to start just call init? http://erldocs.com/18.0/stdlib/gen_server.html#init/1

tolitius05:12:45

@tbaldridge: > once you start automatically restarting things you really have to think about thread safety could you lock on "restart by the supervisor"? performance would degrade, but if it came to this case, it would mean a certain component or a group of components were stopped (or in unsatisfactory "health") anyway

mpenet06:12:27

Erlang also has process dictionaries

mpenet06:12:42

Not to mention ets & all

mpenet06:12:38

Usually it s avoided to block in init. "Erlang in anger" has a good guide about this stuff

qqq07:12:31

(repeat (f)) is equiv to (let [x (f)] (repeat x)) // now, is there a way to generate a ( (f) (f) (f) (f) (f) (f) ... ) ? In particular, here, f = gensym -- so I want it called multiple times.

tianshu08:12:16

is there a built-in function that take a argument x return #(= % x)?

val_waeselynck08:12:27

@doglooksgood unlikely, given how easy it is to write (partial = x) 🙂

val_waeselynck08:12:19

I'm observing transient maps to be much slower for reading than regular maps, is this a known issue? Is there something wrong with my benchmark below?

val_waeselynck08:12:06

I'm trying to write a fast transient version of clojure.core/update, but at this point I'm wondering if it's worth using transients at all for this purpose..;

joshjones08:12:59

Transient data is initially essentially shared with the non-transient data it came from. So I'm not surprised the access time isn't faster. Updates, however, are the selling point for transients and should be much faster.

val_waeselynck08:12:58

@joshjones access time is worse than 'not faster', it's more than twice slower 😕

dacopare11:12:03

Does any know of a Clojure library for manipulating nested datastructures other than Specter? I heard talk of one at Clojure Exchange but the name isn't coming to me now.

pesterhazy11:12:31

clojure.walk ?

sveri12:12:45

@dacopare I remember there was a library where you could provide before / after datastructures and it figured the transformation out by itself, but unfortunately I do not remember the name

pesterhazy12:12:22

wow that sounds cool, @sveri

piotrek12:12:40

Hello, I am trying to run a Clojure script with:

java -cp clojure.jar:script-utils clojure.main script.clj
where script-utils is a directory containing util/test.clj file (with a correct ns declaration) and script.clj does (require util.test) but the execution fails with java.lang.ClassNotFoundException: util.test

piotrek12:12:34

Are uncompiled clj files on classpath not supported for scripts invocations with clojure.main?

piotrek12:12:58

(I also tried to run it with java -cp clojure.jar:script-utils clojure.main -i script.clj but the result is the same

pesterhazy13:12:35

Should be (require 'util.test), @piotrek

tbaldridge13:12:49

@dacopare my library Odin allows you to query data structures and then run updates on specific areas of the structure as defined by a query (think SQL or Datalog for Clojure data): https://github.com/halgari/odin

dacopare13:12:34

@pesterhazy @sveri @tbaldridge Thank you for your suggestions!

credulous15:12:04

Hi. Apologies if this isn’t the right place to ask this question, redirects welcome. I’m suddenly getting a class not found exception when I try to

lein run
my project.
Exception in thread "main" java.lang.ExceptionInInitializerError
	at clojure.main.<clinit>(main.java:20)
Caused by: java.lang.ClassNotFoundException: monger.core, compiling:(cavalry_admin/data.clj:1:3)
My project.clj includes the dependency
[com.novemberain/monger "3.1.0" :exclusions [com.google.guava/guava]]
and the file mentioned requires the library like this:
( (:require [monger.core :as mg] …
. I can see the library in my .m2 directory:
~/dev/cavalry-admin # ls ~/.m2/repository/com/novemberain/monger/3.1.0
total 128
drwxr-xr-x  7 rodfitzsimmonsfrey  staff    238 30 Dec 07:40 .
-rw-r--r--  1 rodfitzsimmonsfrey  staff    178 30 Dec 07:40 _maven.repositories
-rw-r--r--  1 rodfitzsimmonsfrey  staff  44450 30 Dec 07:40 monger-3.1.0.jar
-rw-r--r--  1 rodfitzsimmonsfrey  staff     40 30 Dec 07:40 monger-3.1.0.jar.sha1
-rw-r--r--  1 rodfitzsimmonsfrey  staff   7257 30 Dec 07:40 monger-3.1.0.pom
-rw-r--r--  1 rodfitzsimmonsfrey  staff     40 30 Dec 07:40 monger-3.1.0.pom.sha1
drwxr-xr-x  4 rodfitzsimmonsfrey  staff    136 30 Dec 07:40 ..
.

credulous15:12:31

I’m not sure how to figure out what I’ve misconfigured.

pesterhazy15:12:44

@credulous have you tried lein clean?

credulous15:12:20

Nope! Didn’t know about that… I ran lein deps, and also changed the lib version. I’ll try that now.

credulous15:12:51

Didn’t help, sadly

pesterhazy15:12:09

@credulous my guess is a typo somewhere

pesterhazy15:12:25

what did you change since it last worked?

pesterhazy15:12:46

also try rm -rf ~/.m2/repository/com/novemberain and re-running

pesterhazy15:12:51

and see if it fetches the jar again

rickmoynihan15:12:26

you can check that the jar you have actually contains the class/ns with jar tvf ~/.m2/repository/com/novemberain/monger/3.1.0/monger-3.1.0.jar I’d expect it does contain what you want in which case… try lein deps :tree to see whether something is overriding your dependency

rauh15:12:26

@credulous do a lein classpath and see if it includes the monger jar

qqq17:12:12

@val_waeselynck : thanks! (re repeat vs repeatedly)

qqq17:12:35

is there a way (besides rewriting the function as macro) to say: I want this function inlined instead of having it as a function call?

joshjones17:12:35

@val_waeselynck I have done some research regarding your problem. The TL;DR version is this: the slow part of the transient lookup is when you try to get a key that is not in the transient array map, when the key is a keyword and the map does not utilize keywords for keys. (with more details below). If I do: (def arraymap-1 (transient {:abc 1 :def 2 :ghi 3 :jkl 4 :mno 5 :pqr 6})) and retrieve an unknown key using (get arraymap-1 :z) then it is a 17ns operation. However, if I define the map with no keywords, but all numbers for keys and vals: (def arraymap-1 (transient {5 1 3 2 7 3 8 4 9 5 11 6})) Then a (get arraymap-1 :z) takes 270ns. Crazy. If I do a (get arraymap-1 2389752346), then it's 64ns, much better. Persistent and transient hash maps are fine, and persistent array maps are fine. The issue is the transient array map when accessed as above. There's more to this story but I don't have time today to research it any further. My curiosity took me this far but you'll have to take it from here. I suggest following the trail into doValAt and valAt in both the persistent and transient implementations in [PersistentArrayMap](https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/PersistentArrayMap.java).

roelof18:12:42

Someone who knows where I can find web.config on Windows 10

roelof18:12:03

I need that one so I can debug a 400 status code

qqq18:12:14

"deftype is for programming constructs and defrecord is for domain constructs" <-- this makes no sense to me, one is about building up the language, and the other is about domain specific? I don't get the distinction

noisesmith18:12:13

qqq: deftype lets you create something java code can access easily, defrecord makes something with sensible behaviors for usage in your own code

noisesmith18:12:58

so deftype is an implementation detail, defrecord is for modeling things in an idiomatic clojure manner (I think...)

bbloom18:12:46

i wouldn’t call deftype an implementation detail

bbloom18:12:17

what that quote means by “programming constructs” is that deftype lets you create new abstractions

bbloom18:12:31

by contrasting that with “domain constructs” it’s saying that you probably don’t need new abstractions to model data

bbloom18:12:49

you usually don’t need records to model data either, but there are some times it is useful to have a unique type for dispatch

bbloom18:12:08

the summary is this: use maps for pretty much everything, records if you need dispatch (and only if your record will coexist with dispatch against arbitrary types!) or explicitly typed fields for perf (and you measured that it actually helps!)… the only use deftype if the abstraction you have is really really not plain associative data

hugesandwich18:12:59

@qqq another way to think of it is if you are going to be doing standard hashmap-like operations on the target, then defrecord makes a lot of sense. I use defrecords for a lot of purposes. Aside from something declarative, sometimes I use it as a faster map (with some costs) or a map that I can attach a protocol/detect using a protocol. For instance, you can use a record and a protocol as a faster alternative to multi-methods in some situations like conversions or encoding/decoding for a codec or you can use it to implement a protocol that has naturally grouped functions with multiple implementations. Deftype (which also uses protocols) on the other hand as stated tends to go well with working with Java and makes a lot of sense if you're not going to be doing things like adding keys to it. I tend to use deftype a fair bit when I want some kind of resources where representing it as a map makes no sense or would be annoying (i.e. Java API client or something like that). There's no reason you can't take something you defined as a deftype and put it into a map or a record anyway or vice-versa. I would say deftype is still idiomatic and has a lot of uses. Look at clojure itself for some examples, as well as its cousin reify when you need something anonymous or with less functionality (a fair amount of use of this in core.async for example).

hugesandwich18:12:25

I would agree with pretty much everything @bbloom said.

hugesandwich18:12:36

Of course as is the case with anything done for performance reasons alone, measure first.

seancorfield18:12:23

@qqq I’ll give you two concrete examples of deftype from our (production) code base...

seancorfield18:12:01

We deftype LazyAtom as a variant of Clojure’s Atom that defers evaluation of its initialization until first use (we use it for mutable caches so we can defer the seed creation)

seancorfield18:12:58

And we also use deftype along with definterface to create functions we can add Java annotations to, for tracing function calls in New Relic.

tolitius18:12:16

@qqq: I tend to mostly use maps for data and deftypes for polymorphic dispatch. defrecords complect things (in my mind) for several reasons: * they can't be called as function i.e. ({:a 42} :a) * assoc on a record returns a record, dissoc returns a map * the way they are usually used (extending protocols) they complect data and behavior deftype is simpler, consistent and greatly named: it is a "new type". There is no expectations of deftype being a map, hence no function / assoc/dissoc inconsistencies, there is a clear expectation that it could encapsulate data, and it should not be accessed directly, etc.

josh_tackett20:12:18

Anyone know the best clojure graph db?

olslash20:12:58

i cant speak to best but i've used neo4j from another lang and it looks to have good clojure docs

olslash20:12:04

was nice to work with

olslash20:12:07

also jvm based

wilkerlucio21:12:27

hello people, can someone please explain to me why this (transduce (map #(* % %)) - 0 [2 3 5]) results in 38? before seeing that I expected a transduce like that to have the same result as (reduce - 0 (map #(* % %) [2 3 5])) which returns -38

wilkerlucio21:12:43

it seems the transduce version does a last call to - with -38 (the last accumulation), making it reverse, is that the expected behaviour?

tbaldridge21:12:09

@wilkerlucio yes, it's the expected behavior. The last call is done to do any flushing/finalization that needs to be done by the transducer pipeline. Useful if you're doing things like transients that need a final step to turn them back into persistent data.

wilkerlucio21:12:12

thanks for the clarification @tbaldridge

hugesandwich21:12:13

@josh_tackett That really depends on what you want from the DB. I wouldn't worry about Clojure so much as what the DB does to meet your needs as most graph DBs have wildly different performance and scalability characteristics and use-cases. As long as it has a Java bindings which most do, it is minimally useful from Clojure. Ideally of course it would already have Clojure bindings. It's been awhile since I picked a new graph db, but Datastax Enterprise Graph, Orient DB, and Neo4j are popular choices these days. If your needs are relatively simple or you can make compromises in performance, reliablity, built-in algorithms, etc., then you can essentially build a graph DB on top of most databases, many of them have decent bindings from Clojure - ex: Datomic, Cassandra, Elasticsearch, Redis all can host rudimentary graphs (ex: simple adjacency list-based queries) but won't meet more complex needs unless you build out what amounts to a product on top of them.

hugesandwich21:12:42

Regarding neo4j, it has a lot of support and documentation, so if anything it's a good place to learn. I haven't used it in a year or two, but my experiences with scaling it aren't good compared to others. If your datasets are small though it performs well enough and is probably the easiest of pure graph databases to get started using. A bit off your question, but I'd also ask yourself if you really need a graph database at all before you dig too deep. There are many justifications, but I've heard too many people wanting to use them when their problem wasn't really a graph problem. The issue is a graph can pretty much represent almost anything so it's easy to shoehorn things in that will do better with other solutions.

hugesandwich21:12:21

You can also use GraphX which is part of Spark. It's possible to use Flambo which has Spark bindings to do this, just requires a bit of tinkering last I checked, but maybe that changed.

bradford21:12:59

Quick Q: in Cheshire, how do I get (generate-string) to output json map keys like item: instead of strings like "item":

nikki21:12:25

@bradford: im not sure but i believe the JSON spec requires string

nikki21:12:39

Do u mean in json or clj(s)

nikki21:12:06

Like the JSON spec requires quote marks

bbloom21:12:02

nikki is right - json requires keys be quoted. js allows unquoted strings

bradford21:12:30

Ah yes.

var config = {
        mode: "fixed_servers",
        rules: {
          singleProxy: {
            scheme: "https",
            host: "host",
            port: parseInt("22225")
          },
          bypassList: [""]
        }
      };

bradford22:12:10

(I'm generating .js scripts, fun times)

qqq22:12:14

is clojure and/or guaranteed to (1) eval in order and (2) shortcircuit ?

bbloom22:12:32

yes, you can prove that to yourself easily by typing (source and) or (source or) in your repl

qqq22:12:26

@bbloom: ah yes, but unless it's guaranteed by the spec, it seems like the type of thing that could change "for the sake of parallelism"

bbloom22:12:48

@qqq: read the doc string

qqq22:12:59

"Evalutes exprs one at a time"

bradford23:12:38

What's the simplest Clojure equivalent of java's ExecutorService? I want to run a bunch of independent processes/threads (selenium web browsers) and timeout each process after a few seconds. no shared state