Fork me on GitHub
#clojure
<
2018-01-23
>
qqq02:01:34

I'm looking through all the examples here: https://github.com/jgpc42/insn/tree/master/test/insn I don't see any indication in any of the examples how it figures out HOW MANY LOCALS THIS FUNCTION USES. In a function call, the caller sets up the arguments. Then it invokes the JITTed called-function code. This called-function then needs to allocate stack space for the # of locals it uses. However, I don't see anywhere to indicate / specify how much space to allocate for locals in ANY of the examples. What m I missing:?

tbaldridge02:01:22

@qqq that's just as simple as finding the max of all the local indexes, no?

tbaldridge02:01:53

Why would you need to specify by hand how many locals are defined?

qqq02:01:06

@tbaldridge: In theory, finding max of all local indexes + checking if max is a long/double (since those take 2 cells) should work. I was under the impression that in C, the compiler generates the ASM AND omputes how much space to allocate for local vars. This intuition led me to think that for the JVM, I need to also specify it by hand.

qqq02:01:16

I don't know either way, which is why I'm asking. šŸ™‚

tbaldridge02:01:39

yeah, read up a bit on how the JVM stack machine works, this is completely different

qqq02:01:43

@tbaldridge: you're written code that emits JVM Bytecode right? Have ou ever had to specify how much space to allocate for loals?

tbaldridge02:01:41

No, that's always autocalculated by the tools

tbaldridge02:01:47

Good place to start

qqq02:01:58

section 2.6.1:

qqq02:01:59

Each frame (Ā§2.6) contains an array of variables known as its local variables. The length of the local variable array of a frame is determined at compile-time and supplied in the binary representation of a class or interface along with the code for the method associated with the frame (Ā§4.7.3).

qqq02:01:20

Unfortunatley, it's not clear here if 'compile-time' means *.java -> bytecode or bytecode->binary (I thought the latter was JIT time, and the former is compile-ime).

bronsa02:01:46

the ASM library which insn uses internally does a bunch of bookkeeping for you automatically

bronsa02:01:53

in the context of the JVM/java, compile time means -> bytecode

bronsa02:01:16

you won't find details about how the JIT works in the JVM spec nor should you care about it

qqq02:01:44

@bronsa: just to be pedantic because I'm new to all this: 1. compile time = *.java -> bytecode 2. computing # of space for locals is at compile time 3. if we were writing jvm bytecode, we WOULD have to manually specify # of space for locals 4. in insn, we are writing ASM ( http://download.forge.objectweb.org/asm/asm4-guide.pdf ) , not jvm bytecode cirectly 5. ASM computes the # of cells ofr locals for us, and we don't have to track it manually ^-- are the above statements correct ?

bronsa02:01:35

compile time in the context you just cited means -> bytecode, yes

bronsa02:01:25

if you were to write jvm bytecode by "hand" i.e. emitting bytes to a byte array, you would have to keep some bookkeeping, not exacly declaring space for locals, but other stats required by i.e. the jvm bytecode verifier

bronsa02:01:54

in insn you're writing symbolic bytecode as clojure data, internally insn emits that as JVM bytecode through the ASM library

bronsa02:01:34

don't try to apply what you know about x86 to this. the JVM byecodes are instructions for a virtual machine, not for a processor, and the virtual machine is quite significantly more abstracted

qqq02:01:58

@bronsa: got it; thanks; does insn or something above it allow me to write "three address code -> bytecode", or is this something I'd have to implement on my own

bronsa02:01:09

TAC is just an intermediate representation, nothing to do with bytecode -- you can chose to represent your intermediate language however you want as long as you have a translation to bytecode

josh_tackett03:01:12

Anyone know the best way way to set up lein on an EC2?

josh_tackett03:01:42

nvm got it:

Download the script: wget 
Move the script to $PATH, e.g.: mv lein /usr/local/bin/
Make the script executable, e.g.: chmod +x /usr/local/bin/lein
Execute lein (leiningen will install automatically)

noisesmith03:01:46

unless you are setting up a build server, consider building an uberjar and not using lein on your remote

nblumoe08:01:13

Hey, I would like to import a couple of nested classes and ideally have them available just by their own name, without having to prefix the name with the enclosing classā€™ name. For example with the following imports, I would prefer being able to use ExecCreateParam directly instead of DockerClient$ExecCreateParam. Is that possible and if yes, how? Thanks!

(:import (com.spotify.docker.client DefaultDockerClient DockerClient LogStream
                                      DockerClient$RemoveContainerParam
                                      DockerClient$ExecStartParameter
                                      DockerClient$ExecCreateParam
                                      DockerClient$ListContainersParam)
           (com.spotify.docker.client.messages ContainerConfig ContainerInfo ContainerState))

rauh09:01:02

@nblumoe Pretty certain you'd need to escape to macros

rauh09:01:17

This seems to work:

(def CS PersistentVector$ChunkedSeq)
(defmacro CS.
  ([& args]
   (list* 'new 'PersistentVector$ChunkedSeq args)))
(CS. [0] 0 0)
(instance? CS (CS. [0] 0 0))

rauh09:01:37

Though, the dot in the macro name isn't really legal IIRC.

nblumoe09:01:13

Ha, ok. I wouldnā€™t want to introduce a macro for this. If this isnā€™t possible with the existing import mechanism, I wonā€™t do it.

schmee09:01:41

why though?

nblumoe09:01:36

Why I donā€™t want to use the macro? Or why itā€™s not possible? Or why something else? šŸ™‚

schmee10:01:23

haha šŸ˜„ but yeah, why not use a macro? I mean macros are the reason you donā€™t have to limit yourself to the existing import mechanism, unless you want to for some other reason

nblumoe10:01:41

Sure, but to me it seems the additional code isnā€™t worth it. The value from doing that is very small IMHO. So, at least for me, in that case ā€œless codeā€ is more valuable than ā€œa bit of convenienceā€.

schmee10:01:35

aight, I definitely agree with that :thumbsup:

spacepluk11:01:08

I'm trying hikari-cp and I must be missing something very obvious because it doesn't seem to release be releasing the connections, this is how I'm using it https://gist.github.com/spacepluk/5f2bc9354ae297a6fc544920dabc6ea6

schmee12:01:49

what does your datasource look like?

schmee12:01:49

nvm, Iā€™m blind šŸ˜„

schmee12:01:46

with-db-connection uses with-open, so as long as the connection as a .close method it should work šŸ˜•

spacepluk11:01:32

I would expect with-db-connections to call .close on the connection it gets from the datasource but it still gets stuck after I run 10 queries which is the default pool size

fmind11:01:55

Did you guys check https://list.community/ ? The owner are creating a list to navigate through awesome lists on GitHub

qqq11:01:15

is rhino the only js interpreter on java ?

qqq11:01:35

I'm looking for a way to interpret wasm/wat code from clojure

spacepluk11:01:58

there's also nashorn

spacepluk11:01:44

and you can also wrap one the native ones

spacepluk11:01:29

@qqq but maybe something like this will be better https://github.com/cretz/asmble

qqq11:01:32

does nashorn support wasm? the only thing I can find on google is 1. a 2015 post asking the same question and 2. some presentation that mentions wasm and nashorn in the same presentation

spacepluk11:01:03

I don't know I've only used rhino on the jvm

qqq11:01:05

oooh, asmble is nice, wasm -> jvm bytecode ?

spacepluk11:01:19

that's what they say šŸ™‚

foobar14:01:32

can nrepl round trip metadata?

noisesmith15:01:22

@foobar nrepl is a protocol for sending messages from an outside process to clojure, to round trip metadata you need to adopt some convention for capturing the metadata and sending it back to clojure again

noisesmith15:01:34

for starters there's *print-meta* I guess

qqq15:01:04

Is it just me, or is the JVM one of the most dynamic yet safe languages of all time? The fact that from Clojure, I can define a new class on the fly, specify it's bytecode, and then execute it ... is mindblowing.

tbaldridge15:01:33

@qqq now you have reached enlightenment šŸ˜„

victorvoid15:01:54

Hello guys! Has anyone had to create unit tests that modify an atom that is the shared state? how do you do to do multiple tests and that each had the initial state and not modified with the previous test? Should I use 'reset! ... ' in all tests?

souenzzo15:01:04

@victorvoid I use it (works on my machine). But in general, "functions should return, not do"

(let [local (atom {})]
(with-redefs [foo.bar/my-state local]
(do-stuff!)
(is (= @local {:success? true}))))
cc http://blog.cognitect.com/blog/2016/9/15/works-on-my-machine-understanding-var-bindings-and-roots

tbaldridge15:01:38

@victorvoid most of the time I pass the atom into the functions and create a new one for each test

tbaldridge15:01:00

a reset around each test is a fallback approach.

tbaldridge15:01:05

Same for databases incidentally.

victorvoid15:01:36

I liked the idea of passing as an argument, it seems better than repeating the reset code :thinking_face:

tianshu15:01:32

I saw Clojure 1.10 has a string capture mode, what's this?

tbaldridge15:01:25

@doglooksgood is a (yet) undocumented feature needed for a unannounced project šŸ˜„

bronsa15:01:06

@doglooksgood

user=> (.captureString *in*)
nil
user=> 1 2 3 4
1
2
3
4
user=> (.getString *in*)
"1 2 3 4 \n(.getString *in*)\n"

bronsa15:01:59

as to what this is useful for, Iā€™m assuming things like what tools.readerā€™s source-logging-push-back-reader is used for ā€” reading while preserving capturing newline/comment info

tianshu16:01:06

Still no idea for this. šŸ˜¶ waiting for something come out.

Jakub HolĆ½ (HolyJak)16:01:47

I am really confused by the docs and reality of type hinting function return values. I believe the error is mine, but can you help me see it? According to https://clojure.org/reference/java_interop#typehints, "hint can be placed before the arguments vector". Yet:

(defn before-args ^String  [] "32")
(:tag (meta #'before-args))
=> nil
but if I place it before the name, it works:
(defn ^String before-name [] "32")
(:tag (meta #'before-name))
=> java.lang.String
?!

bronsa16:01:19

first example if you want to retrieve the type hint, youā€™ll have to look at :arglists

Jakub HolĆ½ (HolyJak)16:01:06

@bronsa I don't see it there either:

(defn before-args (^String  [] "32"))
(:arglists (meta #'before-args))
=> ([])

bronsa16:01:22

call meta on first of the arglist

Jakub HolĆ½ (HolyJak)16:01:27

I find it still very confusing that I can place metadata both on the function and on its arglists šŸ™‚

bronsa16:01:50

donā€™t place it on the function

Jakub HolĆ½ (HolyJak)16:01:28

never? even if it doesn't have multiple arities?

bronsa16:01:00

since 1.8 placing it on the arglist is the preferred way

bronsa16:01:48

placing it on the var has some significant edge-cases and gotchas

matan16:01:23

Isn't it quite crazy we don't have proper recursion in clojure, because "clojure uses the Java calling conventions". Can anyone detail why using the "Java calling conventions" excluded proper recursion API? šŸ™‚

bronsa16:01:46

because the jvm bytecode doesnā€™t support TCO

bronsa16:01:01

tail call optimization

noisesmith16:01:06

@matan java can't do goto outside a method , which means tail calls can't be generally optimized

matan16:01:08

Scala does

noisesmith16:01:30

@matan it can't optimize tail calls between methods

matan16:01:56

Scala does that. Doesn't Java itself have proper recursion itself? I am confused

bronsa16:01:02

scala does local TCO in exactly the same way that loop/recur work

noisesmith16:01:05

the reason we use recur for self-calls instead of it being implicitly optimized is to avoid the common error of assuming something is optimized when it can't be

noisesmith16:01:21

it could be implicit, the choice was made for it not to be

bronsa16:01:49

and like @noisesmith said, scala doesnā€™t do proper TCO, it only supports proper recursion in self recursive methods

matan16:01:57

@bronsa yep, and in scala it follows the standard API, where you just use the function's name, not a special language primitive

noisesmith16:01:00

@matan historically, in other lisps, you can assume tail calls period are optimized

noisesmith16:01:08

even if it's a call to another function entirely

noisesmith16:01:25

the choice is tediously explain this shortcoming, or making tail optimization an explicit thing (recur)

bronsa16:01:41

@matan so what, scala is not clojure and they made different choices

bronsa16:01:03

thereā€™s not a technical reason why self recursion couldnā€™t be optimized in clojure

bronsa16:01:07

itā€™s quite trivial to do so in fact

bronsa16:01:10

itā€™s a design decision not to

matan16:01:46

I admit I didn't get the trade off. I guess this is more for a blog post.

tbaldridge16:01:57

The "tradeoff" if you can call it that, is that without proper TCO only some tail calls would be optimized, and only when the function you are calling is statically known @matan

tbaldridge16:01:08

for example this won't be TCO'd in many languages without extensive constant folding: `(fn foo [x] (let [f (fn [] foo)] ((f) x)))`

tbaldridge16:01:52

To TCO that on the JVM You'd have to analyze the code to the point that you'd realize that (f) returned this and then optimize away the call into a goto

matan16:01:19

Will chew on this @tbaldridge

tbaldridge16:01:20

Or...you could just say "if you want TCO use an explicit recur" which not only is simpler to implement, it also improves code clarity

tbaldridge16:01:37

having written a fair amount of code int the TCO style, I can't tell you how many times I've written bad code because expected a tail call from a non tail call position. Clojure's compiler will throw an error on that.

matan16:01:41

Thanks guys (in Scala by the way, one may articulate their assumption of a tail call, to avoid the same trouble)

tianshu16:01:28

@holyjak just had a test, for function have one arglist, both before arg vec and before function name, type hint will work.

noisesmith16:01:44

as far as I can tell, a hint before the arg vector silently fails to do anything

bronsa16:01:03

a hint before the arg vector silently does the right thing ;)

noisesmith16:01:24

I didn't see it on the metadata

bronsa16:01:29

a hint before the var is bad and canā€™t sometimes fail

bronsa16:01:34

@noisesmith itā€™s on the arglists meta

bronsa16:01:44

different arities can have different return types

bronsa16:01:26

(not that thatā€™s a good idea)

noisesmith16:01:53

user=> (meta (first (:arglists (meta #'before-args))))
{:tag java.lang.String}
aha

noisesmith16:01:00

I would not have thought to dig that deep

tianshu16:01:13

(defn without-hint []
  (doto (ArrayList.)
    (.add "1")))

(defn with-hint ^ArrayList []
  (doto (ArrayList.)
    (.add "1")))

(defn do-without-hint []
  (let [l (without-hint)]
    (.remove l "1")))

(defn do-with-hint []
  (let [l (with-hint)]
    (.remove l "1")))

(time (dotimes [_ 1000] (do-without-hint)))
;; "Elapsed time: 3.945445 msecs"

(time (dotimes [_ 1000] (do-with-hint)))
;; "Elapsed time: 0.275406 msecs"

mfikes16:01:46

Hah. To what @tbaldridge said regarding ā€œbad code because expected a tail call,ā€ even in the ClojureScript compiler codebase (in some ClojureScript code), we have a recur which itself is not in tail position, but once we fixed the compiler bug, it pointed this out to us. My point, is that explicit recur is a good thing (and it is also good to make the compiler enforce it). If you are curious, here is the bad recur: https://github.com/clojure/clojurescript/blob/master/src/main/cljs/cljs/js.cljs#L707 which was probably really messing up the unwinding of bindings. And the related ticket. https://dev.clojure.org/jira/browse/CLJS-2476

Jakub HolĆ½ (HolyJak)16:01:37

Another metadata confusion: why the shorthand :some-flag should be before fn name but the full {:some-flag true} after it?!

(defn full {:some-flag true} [] nil)
(defn ^:some-flag shorthand [] nil)

(= (:some-flag (meta #'full)) (:some-flag (meta #'shorthand)) true)
=> true
(Notice that (defn shorthand-bad ^:some-flag [] nil) (:some-flag (meta #'shorthand-bad)) ;=> nil

noisesmith16:01:05

@holyjak

+user=> (defn shorthand-bad ^:some-flag [] nil)
#'user/shorthand-bad
+user=> (meta (first (:arglists (meta #'shorthand-bad))))
{:some-flag true}

Jakub HolĆ½ (HolyJak)16:01:17

I guess that explains why it cannot be at that place. ^:kwd and {:kwd true} are clearly handled differently here.

bronsa16:01:05

defn has a bunch of places where you can put a map and it will merged into the var meta

bronsa16:01:24

one is after the var name, the other is after the fn methods

bronsa16:01:52

user=> (defn foo ([]) {:foo true})
#'user/foo
user=> (:foo (meta #'foo))
true

bronsa16:01:55

itā€™s just convenience

noisesmith16:01:58

@holyjak you were comparing equality of nil to nil

Jakub HolĆ½ (HolyJak)16:01:37

?! I don't think so:

(= true (:some-flag (meta #'full)) (:some-flag (meta #'shorthand)) )
=> true

noisesmith16:01:30

oh sorry, I misread - it's shorthand-bad that has nil (it hides the metadata under an arglist vector)

borkdude16:01:46

cgrand.xformsā€™s str, how should I use it? this doesnā€™t work:

(transduce
   (map str/reverse)
   x/str
   ["foo" "bar"])

madstap17:01:36

The str fn from xforms.rfs is a reducing function.

(ns foo.core
  (:require [net.cgrand.xforms :as xfs]
            [net.cgrand.xforms.rfs :as rfs]))

(transduce
 (map str/reverse)
 rfs/str
 ["foo" "bar"])

madstap17:01:08

Dunno what xfs/str is...

borkdude17:01:18

Ah, I just figured out myself.

borkdude17:01:27

I misread (def str (reduce rf/str))

borkdude17:01:33

I thought it was just an alias

madstap17:01:06

Do you know what xfs/str does? I wasn't aware of it before now

borkdude17:01:25

you can use it in reduce instead of regular str

borkdude17:01:41

regular str wonā€™t use a StringBuilder

borkdude17:01:34

or, it will, but not throughout the whole reduce

cgrand18:01:25

(x/some (comp (map str/reverse) x/str) coll)

cgrand19:01:58

Well transduce rf/str is better. I should definitely add a string transducing context.

borkdude19:01:38

do you mean (x/str xform source) ?

borkdude19:01:52

thatā€™s what I expected when I first tried it, but didnā€™t look close enough

cgrand19:01:06

No I meant (transduce xform net.cgrand.xforms.rfs/str source)

cgrand19:01:04

x/str is meant to be used in aggregations.

borkdude19:01:28

(transduce xform net.cgrand.xforms.rfs/str source) already works right

borkdude19:01:08

thatā€™s how Iā€™m using it now

borkdude19:01:27

so I didnā€™t understand what you meant with "I should definitely add a string transducing context"

cgrand19:01:58

Oh I misunderstood you. Yes a x/string working like you expect.

cgrand09:01:17

xforms 0.16.0 is out, x/str with 2 arg is a string-producing transducing context. Also added x/wrap which is handy when producing strings.

borkdude09:01:44

Do you have an example of how to use the 2 arg version?

cgrand09:01:47

(x/str (x/wrap "[" "]" ", ") [1 2 3]) => ā€œ[1, 2, 3]ā€

borkdude09:01:18

ah great! thanks!

cgrand10:01:24

In many cases x/str+`interpose` seems to beat str/join

souenzzo16:01:06

How to pass a "namespace" as arguemnt? It's possible? There is alternatives?

(defn foo [d db]
  (d/q QUERY db))

bronsa16:01:55

user=> (defn foo [v & [m]] (vary-meta v merge m))
#'user/foo
user=> (meta (foo ^:foo []))
{:foo true}
user=> (meta (foo [] {:foo 1}))
{:foo 1}
@holyjak defn is doing nothing different than this

Jakub HolĆ½ (HolyJak)16:01:49

this looks as the answer I asked for, I just need more time to try to understand it šŸ™‚ I guess that - in case 1 input is [] with meta {:foo true} and no new metadata - in case 2 input is just [] without meta and an extra {:foo true} right? Because #1 is handled by the reader while #2 is handled by foo itself (or, in my example, the defn macro)

noisesmith16:01:05

@souenzzo you can use ns-resolve, but it seems like namespaces aren't the right abstraction for what you are doing - if you need a collection use a collection, if you need polymorphism use polymorphism

souenzzo17:01:26

I want to my query namespace work with datomic client or datomic peer

(ns my.query)
;; here work with peer or client
(defn do-query [..]..)
(ns my.app
(:require [datomic.api :as d] ;; using peer, but here I can change to client
 [my.query :as queries]))
...
(queries/do-query db bar)
...
There is some how to do that? how to abstract this?

noisesmith17:01:57

why not pass the function as an argument, or use a multimethod that can use either implementation?

souenzzo17:01:10

1- i will always pass d/q as first argument. Ok, possible. 2- i will reimplement datomic protocol. also possible but both looks ugly for me

noisesmith18:01:59

you don't have to reimplement the datomic protocol - it can be a multimethod, or your own protocol

noisesmith18:01:55

passing a namespace as a function argument is much uglier - it's using a construct designed for organizing code as a runtime dispatch mechanism

noisesmith18:01:49

also consider that if you are passing the query function as an argument, as long as it's the first arg you can create a partial, which can be used the same way you use the initial function

mfikes16:01:12

@holyjak Along those lines, be careful. Need (defn ^boolean bar [] true) in ClojureScript but (defn baz ^long [] 1) in Clojure, IIRC.

Jakub HolĆ½ (HolyJak)16:01:22

oops. Thanks for pointing that out!

bronsa16:01:24

@mfikes unfortunately yes

Ondřej ČermĆ”k17:01:40

Hello everyone, I know people usually don"t like to use magic libraries, but by any chance, is here someone who uses or used sqlKorma?

robert-stuttaford17:01:17

@ondrej.l.cermak perhaps better to simply ask the question you need to ask šŸ™‚

Ondřej ČermĆ”k17:01:55

alright, im trying to achieve this raw sql statement using sqlkorma

Ondřej ČermĆ”k17:01:32

and I have something like this on mind

Ondřej ČermĆ”k17:01:55

trying to figure out how to properly pass the parameter to the array_agg function through korma

Ondřej ČermĆ”k17:01:51

Another approach would be just executing the sql query as it is, but as the whole code already uses sqlKorma, I would like to somehow stick to it.

martinklepsch18:01:06

[{:title "A"
  :children [{:title "B"} {:title "C"}]}
 {:title "X"
  :children [{:title "Y"} {:title "Z"}]}]
What would be the best way turn this into a flattened collection of maps with {:title "..."}? I feel like loop/recur should make this pretty easy but Iā€™m not seeing it šŸ™ˆ

schmee19:01:48

here is a Specter solution:

(def a
  [{:title "A"
    :children [{:title "B"} {:title "C"}]}
   {:title "X"
    :children [{:title "Y"} {:title "Z"}]}])

(def all-nodes
  (recursive-path [] p
    (if-path vector?
      [ALL (multi-path
             (submap [:title])
             (path (must :children) p))]
      STAY)))

user=> (select all-nodes a)
[{:title "A"} {:title "B"} {:title "C"} {:title "X"} {:title "Y"} {:title "Z"}]

schmee19:01:17

no doubt @U173SEFUN knows a better way to accomplish this šŸ˜„

nathanmarz20:01:31

this is cleaner:

(def TREE-NODES (stay-then-continue :children ALL))
(select [ALL TREE-NODES (submap [:title])] a)

schmee20:01:37

damn, it doesnā€™t get much cleaner than that :thumbsup:

noisesmith19:01:55

you can't do branching recursion easily with loop/recur

noisesmith19:01:30

if the nesting of children is unknown in depth, the easiest thing is `(filter map? (tree-seq sequential? seq x))` (filter map? (tree-seq coll? seq x))

noisesmith19:01:16

you can do branching recursion with loop / recur but it requires extra book-keeping for pending branches since you can't follow both at once

noisesmith19:01:38

this also isn't hard with primitive (non-optimized) recursion

martinklepsch19:01:08

Yeah, I got this:

(defn flatten*
  [doc-tree]
  (mapcat (fn [{:keys [children] :as doc}]
            (concat [(dissoc doc :children)]
                    (flatten* children)))
          doc-tree))
but it didnā€™t feel quite right

noisesmith19:01:54

oh, I left the dissoc off my filter / tree-seq version

noisesmith19:01:05

(easy to throw in of course)

martinklepsch19:01:12

yeah, it doesnā€™t matter much, more for readability

martinklepsch19:01:43

but the result of your tree-seq is = to the input with the data I pasted above

martinklepsch19:01:12

That said I see how tree-seq should help with this and will investigate

noisesmith19:01:43

wait, how would it be equal, given the nested children? oh right because it is ignoring things that are not sequential

noisesmith19:01:25

oh, replace sequential? with coll? and that fixes it

bitti19:01:32

so strange all of a sudden I get a really long backtrace when I try cider-jack-in with Caused by: java.io.FileNotFoundException: Could not locate clojure/java/classpath__init.class or clojure/java/classpath.clj on classpath. as exception message

noisesmith19:01:36

and then optionally remove children

bitti19:01:48

any idea what went wrong, or how to debug this thing?

bitti19:01:25

I already removed /.m2 and /.lein and fetched all dependencies with leiningen again, to no avail

bitti19:01:41

also lein repl is working fine

mfikes19:01:48

@martinklepsch If you were looking for a loop / recur based version of flatten for performance, I bet a fast (directly reducible) one could be written by taking the implementation of flatten and re-writing it with transducers (using (into [] ...) and having it call the tree-seq' here https://gist.github.com/mfikes/cc1d1fac47e7dc112b0b9f4e3de11eae

noisesmith19:01:48

@david479 those things should never help btw

mfikes19:01:22

At the bottom is iterate, which is directly reducible

noisesmith19:01:24

@mfikes but you get the same problem my version with sequential? got - which is that flatten ignores hash-maps

mfikes19:01:37

Ahh, missed that nuance

noisesmith19:01:48

(that's easy to fix with a custom impl of course)

martinklepsch19:01:05

@mfikes thanks, perf doesnā€™t really matter, itā€™s pretty small data. Just thought I should be using recursion ā€” and my intuition with regards to recursion is pretty crap šŸ˜„

martinklepsch19:01:20

but tree-seq is nice, I like it šŸ™‚

noisesmith19:01:38

a good rule of thumb is that branching can't be optimized recursion without an extra storage arg representing unvisited branches

noisesmith19:01:11

for some problems using that is worth it, sometimes the added complexity means it's better for code clarity sake to just use regular recursion

noisesmith19:01:09

and lazy operations can move your storage out of the stack and into the heap / gc without awkward loop variables

noisesmith19:01:06

(in that sense tree-seq is actually doing the optimized recursion, but taking the recursive calls out of the call stack and putting them into continuations in lambdas in the heap)

noisesmith19:01:30

but the lambda continuation thing means it's not a perf optimization, just a stack usage one

noisesmith19:01:29

@mfikes oh that version of tree-seq is cool now that I look at it

noisesmith19:01:35

so it's an eager tree-seq thanks to eduction, right?

mfikes19:01:36

@noisesmith It was actually the basis for a directly-reducible tree-seq that I was planning on contributing to ClojureScript, but it turns out that, since the fns passed to tree-seq are not guaranteed to be side-effect free, this is a no-go

mfikes19:01:04

Otherwise, if you have side-effect free fns, it can be faster

noisesmith19:01:21

it's eager rather than lazy, right?

noisesmith19:01:12

or does the consumer decide with eduction?

mfikes19:01:15

Well, it is no more eager than iterate... but the key was that it be directly reducible

mfikes19:01:44

The actual implementation I was going to contribute didn't use eduction, but was a deftype: https://dev.clojure.org/jira/browse/CLJS-2464

mfikes19:01:18

But, I think it is probably the case that it shares a lot of characteristics with iterate

noisesmith19:01:52

wow, I can think of so many places where I could have used (eduction (take-while some?) ...) ...

mfikes19:01:56

It's unfortunate we can't pursue it (it can be up to an order of magnutude faster šŸ™‚ )

noisesmith19:01:41

you could give it a new name right? pure-tree-seq or such

mfikes19:01:51

Yeah, the (eduction (take-while some?) ...) pattern jumped out to me as well. A way to terminate a potentially infinite sequence

mfikes19:01:41

Yeah, there is no issue with using that version of tree-seq. It just can't replace the one in core. (It would break file-seq and other non-pure uses, for example.)

zsck22:01:00

Hi all. I have a really broad question. I've been thinking a lot lately about the problem of dealing with databases in software applications. In particular, it seems like any application architecture (and similar) you look at (e.g. MVC, MVP, FRP, etc.) all have similar problems with databases. What I mean by that is that one often seems to have to somehow wrap a reference to a database-like thing, whether its a closure or a type that implements some DB interface(s). I have some limited understanding that Clojure is designed specifically with database management in mind, and that Datomic is a big part of the Clojure world. Could anyone direct me to somewhere I could read about how Clojure does databases differently?

seancorfield22:01:30

@zack.mullaly Well, Datomic is certainly "databases done differently" but for other databases, it's just JDBC or whatever Java driver/SDK and the usual sort of database stuff.

seancorfield22:01:51

Where I think Clojure scores well is that it's very data-focused so you have hash maps for your data (in general) and those map very simply to rows in SQL databases or documents in MongoDB etc. So you lose all the crazy complexity that Java et al push with ORM.

mccraigmccraig22:01:18

db-record -> clojure-map has worked great for me with a variety of sql and no-sql dbs

zsck22:01:33

So is the idea basically to implement database operations abstractly, treating them as data transformation rather than thinking about something like SQL queries?

noisesmith22:01:39

yeah, I think most of the problems with databases are bizarre development paradigms that think data should belong to behaviors, once you ditch (or at least loosen) that prejudice it's much easier

noisesmith22:01:25

sql queries are fine, as long as you treat them as what they are: data transforms and selection done in database

noisesmith22:01:42

not trying to dress them up as properties of in-app objects

zsck22:01:55

I'm definitely on board with the idea of separating data and behaviour. I am trying to achieve something like that in Rust right now. I just always find myself eventually having a type that wraps a database connection, and implements interfaces (traits in Rust) that do operations accepting plain data on that database. At the end of the day though, I then ended up having to hand copies of that wrapper type around to actual do stuff with it. I'm wondering if that has to be solved at the application architecture level, or if I could actually make a library that lets me treat the database in an entirely different way.

noisesmith22:01:03

separating the connection details about the db from the data operations makes sense - otherwise you end up surrounding everything that talks to the db with a bunch of database implementation details

noisesmith22:01:16

treating an i/o as if it were a pure data operation is just as big a problem as treating data as if it were a statefull domain object

noisesmith22:01:01

but even something like clojure.java.jdbc does this for a large part, if you add a connection pool library

mccraigmccraig22:01:12

promises with error-states ftw! šŸ˜¬

noisesmith22:01:43

@mccraigmccraig yeah for me it's a question of whose code has to deal with the complexity of a potential stall, failure, retry etc.

noisesmith22:01:57

the more that can be segregated and localized the better

zsck22:01:12

I think this is where we start getting into Haskell's IO monad territory, right? As people have mentioned there, Haskell programs often end up with IO becoming somewhat infectious. I think I fundamentally have the same problem. It's hard to design an application that keeps IO in as few places as possible, but also lets you restrict the kinds of IO operations you can perform in a given function's context.

noisesmith22:01:13

@zack.mullaly this isn't perfect, but something that tends to work for me is to avoid abstracting over IO, avoid hiding it

noisesmith22:01:38

so I end up with my top level code being the primary IO actions, and that calls the things that are simple to abstract - pure functions

noisesmith22:01:49

if you do the reverse, the IO implementation details leak everywhere

seancorfield22:01:48

Depending on the sort of queries and/or persistence your app needs to do, abstracting away the data may not even be practical.

mccraigmccraig22:01:53

@noisesmith i'm a big fan of promises with a monad for composition - easy to grok code, and short-circuiting gives you an easy way to deal with a failure here, or let it propagate sensibly

seancorfield22:01:11

I tried to build a library that separated data access (of all sorts) into "pure" queries (sources) and "committables" (that described changes to the persistence layer as pure data that was then processed to update/insert against the database (or other writable / side-effect-y stuff) -- and it was possible but really too painful to be useful in anything but the smallest, simplest app.

zsck22:01:48

I've been doing something sort of similar. I've implemented a bunch of types that grant "capabilities" to perform different state-mutating functions. I'm using this Capability interface to restrict the capabilities of, e.g. a REST API's request handlers. I don't think I can really get away from having to store a reference to my DB connection in the request handlers themselves, but I suppose that if I don't take the idea too far, then I can certainly make everything the handler calls into itself pure.

noisesmith23:01:03

a manageable approach is to pass the impure function to the request handler by attaching it to the request via a middleware, the same middleware can be responsible for making sure the right connection details are used, that the connection is valid etc. since it's a context that wraps the invocation of the handler and can catch appropriate Exceptions etc.

noisesmith23:01:32

@mccraigmccraig but where things get tricky is if a problem in one context gets passed through another - if the entirety of the logic is "bail if it fails" that can be straightforward, but it still doesn't get you anything that (try ... (catch Exception _)) wouldn't - and the big problem for me is that if you try to replace Exceptions, the best case result is that you have two systems now - Exceptions and your alternative to them, because Exceptions themselves are baked into the VM, you can't hide them completely. And in reality any app of significant size has to deal with N systems, because you have to accomodate every library author's favorite alternative to Exceptions (there's multiple choices out there)

mccraigmccraig23:01:49

@noisesmith N systems would indeed be unpleasant, and Exceptions aren't going anywhere... but, promise implementation willing (manifold's is) , you can neatly treat Exceptions as promise error values and deal with just a single error transmission channel

noisesmith23:01:53

well clojure already has a thing called a promise built in, and it isn't an action chaining or error handling mechanism, and there's a few competing versions of Maybe but it has been explicitly said that clojure isn't ever going to have it in core

noisesmith23:01:48

user=> (def p (promise))
#'user/p
user=> (do (deliver p (Exception. "uh-oh")) nil)
nil
user=> (do @p, 42)
42
using do to ensure that the default print behavior for the exception isn't mistaken for something being done with the exception

mccraigmccraig23:01:46

clojure.core/promise is too weak... manifold/deferred and promesa/promise are both up to the job... gtg bed now, but happy to provide examples tmrw

noisesmith23:01:49

my point is that promise isn't even made for that - it's just a deferred value, that's it

noisesmith23:01:29

I know there are examples that work, but as you say they come from multiple places, and none will end up in core, and as an author of a sizable app I have to deal with all many of them, which is tedious

qqq23:01:10

For the shape, offsets, I need it to be int-array, and for the data field, I need it to be a float-array. Anyone know where the valid values for :type is documented ?

(def class-data
      {:name 'my.pkg.Tensor
       :fields [{:flags #{:public :static}, :name "VALUE", :type :long, :value 42}
                {:flags #{:public :static}, :name "dims", :type :long, :value 0}
                {:flags #{:public :static}, :name "shape", :type :long, :value 0}
                {:flags #{:public :static}, :name "offsets", :type :long, :value 0}
                {:flags #{:public :static}, :name "data", :type :float, :value 0}] 
       :methods [{:flags #{:public}, :name "add", :desc [:long :long]
                  :emit [[:getstatic :this "VALUE" :long]
                         [:lload 1]
                         [:ladd]
                         [:lreturn]]}]})

noisesmith23:01:22

@qqq in many contexts the string of the type as java would show it to you works for arrays

user=> (type (float-array 0))
[F
user=> (type (int-array 0))
[I
- so I would try "[F" and "[I"

qqq23:01:25

(do "** iknsn"
    (def class-data
      {:name 'my.pkg.Tensor
       :fields [{:flags #{:public :static}, :name "VALUE", :type :long, :value 42}
                {:flags #{:public :static}, :name "dims", :type "[I" , :value 0}
                {:flags #{:public :static}, :name "shape", :type "[I" , :value 0}
                {:flags #{:public :static}, :name "offsets", :type "[F" , :value 0}
                {:flags #{:public :static}, :name "data", :type :float, :value 0}]}) 

    (def result (insn/visit class-data)) 
    (def class-object (insn/define class-data)))
(comment
              java.lang.ClassFormatError: Bad string initial value in class file my/pkg/Tensor
clojure.lang.Compiler$CompilerException: java.lang.ClassFormatError: Bad string initial value in class file my/pkg/Tensor, compiling:(NO_SOURCE_FILE:46:23))


qqq23:01:56

Here is a minimal failing example:

(do "** iknsn"
    (def class-data
      {:name 'my.pkg.Tensor
       :fields [{:flags #{:public :static}, :name "data", :type "[F" :value 0}]}) 
    (def result (insn/visit class-data)) 
    (def class-object (insn/define class-data)))
(comment
              java.lang.ClassFormatError: Bad string initial value in class file my/pkg/Tensor
clojure.lang.Compiler$CompilerException: java.lang.ClassFormatError: Bad string initial value in class file my/pkg/Tensor, compiling:(NO_SOURCE_FILE:41:23))


noisesmith23:01:08

oh, this is insn stuff - sorry I don't know insn

qqq23:01:41

I should have been clearer, I'm trying to generate a tensor class with insn dynamically.

noisesmith23:01:53

@qqq there might be a clue here but I'm not really sure what it's doing https://github.com/jgpc42/insn/blob/master/src/insn/util.clj#L265

qqq23:01:53

why does (type (float-array 2)) and (class (float-array 2)) both returh [F instead of giving me a fully qualified name, like java.lang.core.array.Float or something ?

noisesmith23:01:19

because that is the name of the class, it doesn't have a package, it's an array

noisesmith23:01:41

the problem is that its name is not a valid clojure data literal

noisesmith23:01:30

@qqq check this out

user=> (Class/forName "[I")
[I
user=> (type *1)
java.lang.Class

qqq23:01:33

so this gives a different error:

(do "** iknsn"
    (def class-data
      {:name 'my.pkg.Tensor
       :fields [{:flags #{:public :static}, :name "data", :type (type (float-array 1)) :value 0}]}) 
    (def result (insn/visit class-data)) 
    (def class-object (insn/define class-data)))
(comment
              java.lang.ClassFormatError: Unable to set initial value 7 in class file my/pkg/Tensor
clojure.lang.Compiler$CompilerException: java.lang.ClassFormatError: Unable to set initial value 7 in class file my/pkg/Tensor, compiling:(NO_SOURCE_FILE:41:23))


qqq23:01:42

note I'm using (type (float-array 2)) as the type

qqq23:01:13

if I drop he value field,

(do "** iknsn"
    (def class-data
      {:name 'my.pkg.Tensor
       :fields [{:flags #{:public :static}, :name "data", :type (type (float-array 1)) }]}) 
    (def result (insn/visit class-data)) 
    (def class-object (insn/define class-data)))
appears to 'works' in that it compiles

noisesmith23:01:38

what does :value 0 mean in that context?

qqq23:01:53

it means we are trying to do float [] a = 0:

noisesmith23:01:18

well that's just bonkers

noisesmith23:01:54

or maybe java is more like C than I thought?

qqq23:01:48

alright, so

(do "** iknsn"
    (def class-data
      {:name 'my.pkg.Tensor
       :fields [{:flags #{:public :static}, :name "dims", :type :int, :value 0}
                {:flags #{:public :static}, :name "shape", :type (type (int-array 1))}
                {:flags #{:public :static}, :name "offsets", :type (type (int-array 1))}
                {:flags #{:public :static}, :name "data", :type (type (float-array 1)) }]}) 
    (def result (insn/visit class-data)) 
    (def class-object (insn/define class-data)))
also compiles, time to see if we can throw in a constructor too