Fork me on GitHub
#clojure
<
2018-01-24
>
noisesmith00:01:46

any particular reason to use (int-array 1) instead of (int-array 0)? also I'd be tempted to bind the type of that in a def at the top level and just use the def

qqq00:01:16

no, merely an irrational fear that (int-array 0) or (float-array 0) may be optimized away

noisesmith00:01:51

you don't need the array, just the type

qqq00:01:36

okay, so we have the following, which is great:

(do "** iknsn"
    
    (def class-data
      (let [ia (type (int-array 0))
            fa (type (float-array 0))]
        {:name 'my.pkg.Tensor
         :fields [{:flags #{:public :static}, :name "VALUE", :type :int, :value 4}
                  {:flags #{:public :static}, :name "dims", :type :int, :value 0}
                  {:flags #{:public :static}, :name "shape", :type ia }
                  {:flags #{:public :static}, :name "offsets", :type ia }
                  {:flags #{:public :static}, :name "data", :type ia }]

         :methods [{:flags #{:public}, :name "add", :desc [:int :int]
                    :emit [[:getstatic :this "VALUE" :int]
                           [:iload 1]
                           [:iadd]
                           [:ireturn]]}]

         })) 
    (def result (insn/visit class-data)) 
    (def class-object (insn/define class-data))
    (-> class-object .newInstance (.add 17)))
(comment
 21)


qqq00:01:00

for the next episode of #clojure does insn, let's add a constructor 🙂

qqq00:01:31

maybe I should live cast and avoid flooding the channel

qqq00:01:54

anyone interested in joining a learn-insn screencast ?

noisesmith00:01:55

@qqq also, back-referencing the convo about specifying stack usage - remember that in the jvm you can't put variable size things (even arrays) in the stack itself, so calculating stack usage is trivial

noisesmith00:01:26

you might need it if you are using a vm / architecture where you can put arbitrary data on the stack but the jvm isn't that flexible so you can let it calculate automatically

qqq00:01:42

I see; that makes sense

qqq00:01:55

A lot of the "C / x86 / mips" intuition was tripping me up.

qqq00:01:20

also, apparently the entikre operand stack is imaginary and gets compiled down to registers

noisesmith00:01:36

well surely that's architecture specific?

noisesmith00:01:37

(or I guess it could be every architecture that's worth paying attention to is register based so the distinction is academic?)

andy.fingerhut00:01:08

I haven't looked at JVM implementations in detail, but I think "operand stack is imaginary" might be overstating it. I would suspect that for bytecode that is actually interpreted (which by default in the Sun/Oracle JVM, all of it is until it decides to JIT some classes/methods into native machine instructions), it does actually have some explicit representation of the operand stack.

gonewest81800:01:25

Is currently down? I’m getting a 504 Gateway time-out just browsing the URL to look something up.

andy.fingerhut01:01:50

I see the http://clojars.org home page from where I am. I had a bit of momentary confusion and a chuckle when I saw this large text near the top of the home page, wondering whether it was down: "Clojars is a dead easy community repository ..." (My brain first register the words "Clojars is dead")

qqq01:01:54

I have a Java class (not a *.class file, but something in memory on the JVM). I want to disassemble it and see the raw jvm bytecode associated with this class. What library should I use ?

the2bears01:01:36

@qqq not sure you can, I've never seen one at least in my (too) many years of Java.

noisesmith01:01:00

no.disassemble can disassemble the bytecode for a function to symbolic instructions, if it can do that I assume there's something that can do the same for a regular class

noisesmith01:01:46

if you really mean the bytes and not symbolic instructions, surely you could just output the bytes of the class? (I assume there's some way to access that)

qqq01:01:26

disassembling a function should be good enough

noisesmith01:01:13

OK - yeah, no.disassemble would be your guy for that - and seeing what it does might be informative too https://github.com/gtrak/no.disassemble

the2bears01:01:27

Why am I still surprised by how little I know 🙂

qqq01:01:44

(no.disassemble/disassemble (fn []))
(comment
 )
this output isn't correct is it?

qqq01:01:08

I'm being serious -- there should be bytecode right

noisesmith01:01:30

are you using it as a plugin? it needs to inject stuff and replace a bunch of clojure stuff iirc

qqq01:01:33

(no.disassemble/disassemble (fn [a b] (+ a b)))
(comment
 )

is definitely wrong

qqq01:01:50

damn it, no, I just required it; I don't use lein

noisesmith01:01:11

there's probably a trick to use it without lein, not sure how that works though

qqq01:01:12

I completely ignore the docs that says:

HOWEVER, don't use it this way, let lein-nodissassemble's project middleware inject it for you.

{:plugins [[lein-nodisassemble "0.1.3"]]}

noisesmith01:01:14

yeah - it's meta, it needs to change how clojure generates bytecode in order to work properly, which means it needs to change some stuff on startup (I'm fuzzy on the details but you've discovered the failure mode)

qqq01:01:59

yeah; I've moved the question over to #boot

qqq01:01:18

if it's changing the way clojure generates bytecode -- does it mean it won'g work with insn

qqq01:01:30

because that's the case I care about ... debugging what insn is generating for me

noisesmith01:01:01

interesting question, I wonder - you might have more luck with dumping the class with an ObjectOutputStream and examining that (or feeding it to the eclipse disassembler or whatever)

noisesmith01:01:30

no.disassemble uses the eclipse disassembler API, but they also have a higher level tool inside the IDE - I assume IntelliJ IDea has something similar

qqq01:01:24

I'm using Emacs + Cider + Boot. LOL.

qqq01:01:44

I think this is where I go and read the insn source code 🙂

qqq01:01:53

cloc insn/src/
       7 text files.
       7 unique files.                              
       0 files ignored.

 v 1.72  T=0.03 s (223.7 files/s, 39525.8 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Clojure                          7            147             10           1080
-------------------------------------------------------------------------------
SUM:                             7            147             10           1080
-------------------------------------------------------------------------------
only 1080 lines, is jgpc42 here on slack ?

noisesmith01:01:52

surely you can open some bytes in a program other than your primary editor for debugging purposes?

qqq01:01:23

sure, but switching boot -> lein is going to be a pain, especially since emacs = running on laptop, boot = running on a remote machine exoosing a nrepl

tbaldridge02:01:21

emacs = running on laptop, boot = running on a remote machine exoosing a nrepl

tbaldridge02:01:25

That doesn't even make sense

qqq02:01:00

why not?

tbaldridge02:01:03

But the gist is this: in order to disassembly JVM bytecode you need a instrumentation agent, basically a JVM plugin. That requires special command line options. Not a big problem

qqq02:01:06

cider-connect than connects to the nrepl over ssh

tbaldridge02:01:28

That's pretty much the same in lein vs boot

qqq02:01:59

yes, but I'd have to port over my build.boot to whatever lein uses to make things work, and I've found it very difficult to get lein to do anything besides tweak a few configs

tbaldridge02:01:09

the command line option will be something like -agent:some-jar.jar

qqq02:01:32

@tbaldridge: I'm confused, are you now talking about getting no.disasemble to work with boot ?

tbaldridge02:01:28

Either way is a valid option: switch to lein or use the -agent stuff

tbaldridge02:01:24

That's all the no.disasemble plugin is doing: injecting the -agent command line option

seancorfield02:01:16

BOOT_JVM_OPTIONS="-javaagent:whatever.jar" boot repl (is it -agent or -javaagent?)

seancorfield02:01:55

(or put it in boot.properties -- I can't remember if you can have a project-local .properties file?)

qqq02:01:09

This is amazingly insightful; there's no way I would have guessed this on my own.

qqq02:01:04

Reasoning backward with 2020 hindsight, in theory, one could have looked up lein-nodisassemble, poked around the src files, saw https://github.com/gtrak/no.disassemble/blob/master/lein-nodisassemble/src/lein_nodisassemble/plugin.clj#L20 ... and then reailze "okay, all I have to do is add the -javaagent , and BAM")

seancorfield02:01:36

The more I work with Boot vs Leiningen, the more I feel the latter used to obfuscate a lot of things that really could have been much simpler...

seancorfield02:01:16

...and that was partly why we switched: writing plugins is a lot more work than just doing the "obvious" thing in Clojure code.

seancorfield02:01:06

The downside to Boot is the lack of declarative support for tooling (particularly Cursive), but maybe adopting deps.edn will help resolve that (my boot-tools-deps is a work in progress since tools.deps keeps changing!).

qqq02:01:34

I'm trying to define a constructor "Tensor". First attempt shows I define a public member function Tensor (but not a constructor). Second example shows error when I try to name it my.dyn.Tensor Question: how do I begin debugging this? I'm not sure where to look.

;; we try to define constructor Tensor, and get outpt
(do "** insn" 
    (def class-data
      (let [fa (type (float-array 0))]
        {:name 'my.dyn.Tensor
         :fields [{:flags #{:public :static}, :name "VALUE", :type :int, :value 4}
                  {:flags #{:public :static}, :name "data", :type fa }] 
         :methods [{:flags #{:public}, :name "Tensor", :desc [:int :int]
                    :emit [[:iload 1] [:ireturn]] }]}))
    (def result (insn/visit class-data)) 
    (def class-object (insn/define class-data))
    (cr/reflect class-object))
(comment
 {:bases #{java.lang.Object},
 :flags #{:public :final},
 :members
 #{{:name my.dyn.Tensor,
    :declaring-class my.dyn.Tensor,
    :parameter-types [],
    :exception-types [],
    :flags #{:public}}
   {:name VALUE,
    :type int,
    :declaring-class my.dyn.Tensor,
    :flags #{:public :static}}
   {:name Tensor,
    :return-type int,
    :declaring-class my.dyn.Tensor,
    :parameter-types [int],
    :exception-types [],
    :flags #{:public}}
   {:name data,
    :type float<>,
    :declaring-class my.dyn.Tensor,
    :flags #{:public :static}}}})


;; okay fine, let us try to rename it my.dyn.Tensor
(do "** insn" 
    (def class-data
      (let [fa (type (float-array 0))]
        {:name 'my.dyn.Tensor
         :fields [{:flags #{:public :static}, :name "VALUE", :type :int, :value 4}
                  {:flags #{:public :static}, :name "data", :type fa }] 
         :methods [{:flags #{:public}, :name "my.dyn.Tensor", :desc [:int :int]
                    :emit [[:iload 1] [:ireturn]] }]}))
    (def result (insn/visit class-data)) 
    (def class-object (insn/define class-data))
    (cr/reflect class-object))
(comment
              java.lang.ClassFormatError: Illegal method name "my.dyn.Tensor" in class my/dyn/Tensor
clojure.lang.Compiler$CompilerException: java.lang.ClassFormatError: Illegal method name "my.dyn.Tensor" in class my/dyn/Tensor, compiling:(NO_SOURCE_FILE:70:23))

qqq02:01:00

insn is insn, cr is clojure.reflect

seancorfield02:01:25

@qqq Remind me, where do I find the insn project?

qqq02:01:06

I'm grepping through the source for the word "constr" right now, there's a file src/insn/core.clj which seems to contai nall the important bits

seancorfield02:01:20

For the exception when you try to name the method my.dyn.Tensor -- I'm not surprised since that's an illegal method name. I'll go read about insn and see if I can help...

qqq02:01:22

Great! This looks like progress: It appears the correct name is ":init" and I needc to call super or init or something:

;; we try to define constructor Tensor, and get outpt
(do "** insn" 
    (def class-data
      (let [fa (type (float-array 0))]
        {:name 'my.dyn.Tensor
         :fields [{:flags #{:public :static}, :name "VALUE", :type :int, :value 4}
                  {:flags #{:public :static}, :name "data", :type fa }] 
         :methods [{:flags #{:public},
                    :name :init
                    :desc [:int :int],
                    :emit [[:return]] }]}))
    (def result (insn/visit class-data)) 
    (def class-object (insn/define class-data))
    (cr/reflect class-object))
(comment
 java.lang.VerifyError: Constructor must call super() or this() before return
                       Exception Details:
                         Location:
                           my/dyn/Tensor.<init>(II)V @0: return
                         Reason:
                           Error exists in the bytecode
                         Bytecode:
                           0x0000000: b1                                     
                       )

seancorfield02:01:13

Yup, was just about to say :init is the constructor and :clinit is the class initializer (`static` stuff I assume)

seancorfield02:01:16

[:invokespecial :super :init [:void]]

qqq02:01:21

It works now; thanks!

seancorfield02:01:24

That test file provides a lot of insight into the stuff you can do ... it looks really cool!

qqq02:01:50

I looked through all the test files ... twice ... I just didn't draw the mental connection of :init == constructor 🙂

qqq02:01:06

and after realizing :init == constructor, I didn't think to recheck the test files 🙂

seancorfield02:01:26

Ironically, I anticipated it because in CFML, when you construct a Java object, you call init() to invoke the constructor...

qqq02:01:28

Thanks for your help; this sabved me hours of debugging.

qqq02:01:55

ColdFusionMarkupLanguaged?

seancorfield02:01:40

Yes, and the components -- classes -- in CFML have a method called init() as their constructor:

component {
  function init( arg ) {
    variables.arg = arg;
    return this;
  }
}

seancorfield02:01:03

and then var obj = new MyComponent( 42 );

qqq02:01:37

while you'r ehere: clinit = stuff for initializing STATIS MEMBERS of class? init = stuff for initializing non-static members of object/class when created? so clinit = called once per class; init = called once per new object created ?

seancorfield02:01:08

Yup, sounds right.

seancorfield02:01:40

Doesn't Java have a static { ... } code block these days for class-level initialization?

Alex Miller (Clojure team)03:01:29

It also has { ... } for instance-level initialization, although that’s rarely used

seancorfield03:01:16

"The static initializer block is a very interesting item in Java that unknown by most of the Java novice community; it is glossed over in Java books but none of the books really go into any sort of dept." [sic] -- http://www.engfers.com/code/static-initializer-block/

seancorfield02:01:53

(it's been so long since I wrote Java)

qqq02:01:27

dunno, you're talking to someone that would rather use clojure to generate jvm bytecode than write java 🙂

seancorfield03:01:16

"The static initializer block is a very interesting item in Java that unknown by most of the Java novice community; it is glossed over in Java books but none of the books really go into any sort of dept." [sic] -- http://www.engfers.com/code/static-initializer-block/

seancorfield03:01:14

I've only started to see that in code recently so I assumed it was relatively new -- TIL!

seancorfield03:01:42

I can't recall ever seeing instance initialization blocks (although they've been there since Java 1.1 it seems).

Alex Miller (Clojure team)03:01:00

Most people just use field initializers or constructors but there are some concurrency cases where they come in handy

bitti04:01:24

you probably saw it without noticing, since it's sometimes disguised as "double brace initialization"

Alex Miller (Clojure team)04:01:42

Well that’s an abomination, esp if you know Clojure

bitti03:01:40

regarding my problem from yesterday, it seems that cider-nrepl 0.17.0-SNAPSHOT is broken, I had to go back to 0.16.0

bitti03:01:55

I wonder if nobody else had problems?

qqq04:01:18

Quoting wikipedia:

The JVM operates on primitive values (integers and floating-point numbers) and references. The JVM is fundamentally a 32-bit machine. long and double types,
How is this possible? If refs are only 32-bits, how can JVM address so much memory?

justinlee04:01:14

presumably because there is no pointer arithmetic?

tbaldridge06:01:24

Pointer compression

tbaldridge06:01:17

And refs can be 64 bits in some jvm modes

justinlee06:01:01

but i thought the real point is that pointers are essentially opaque from the jvm

justinlee06:01:25

in other words: pointer compression is nifty but not actually needed to reconcile the statement @qqq’s quote from wikipedia. you could do 64bit pointers in the jvm.

qqq06:01:49

I asked the wrong question.

qqq06:01:14

For local varaibles, int, float take up 1 cell, long, double take up 2 cells; object references, must somewhere in spec, be stated how many cells they take up

qqq06:01:38

Anyone know which section this is stated in?

justinlee06:01:18

it’s been a while since i dug into the jvm spec, but what my hazy memory from 15 years ago is that it is not defined because you can’t actually look at a pointer in the jvm or observe its properties (except when you interface with native code). but maybe i’m misremembering.

justinlee06:01:47

the whole point of the jvm was statically verifiable type safety (which is also why it is stack based instead of register based). that’s what i was half remembering. but i could be wrong.

cfleming07:01:30

@qqq I’m not sure that’s specified. On 64-bit arch, if you have compressed OOPS on an object reference is 32 bits, otherwise it’s 64.

qqq07:01:55

the JVM needs to figure out how much space for the 'locals' of a class/function, so surely there must be a mapping of "a pointer/reference is X bytes" right? if not, I'm misunderstanding something fundamental

qqq07:01:40

int a;; // iload/istore 0
Object b;; //load/store 1
int c; // load/store ???

qqq07:01:55

how do we figure out the ??? unless we know the "size" of "Object b" ?

cfleming07:01:06

Well, I guess the size is fixed at startup based on JVM args, but it’s not constant across architectures or JVM flag combos.

qqq07:01:46

iload/istore are BYTECODE, so it must be determined at the ".java -&gt; .class" compilation process right?

cfleming07:01:18

My memory on this is hazy, but aren’t they slot numbers, where the slot size can vary?

cfleming07:01:50

i.e. if a is a long, is b then loaded at 1 or 2?

qqq07:01:15

if they're just 'slot numbers', why does this happen

int a; // slot 0
long b; // spans 1 & 2, but iload/istore 1
int c; // iload/istore 3

qqq07:01:22

if they're just "slots", why does long/double take 2 slots ?

cfleming07:01:39

Ok, in that case your question makes total sense, and I don’t know the answer.

qqq07:01:27

sorry if I seem too agressive, attacking the mental model, not attacking the person 🙂 // been trying to figure this out for a while now

qqq07:01:05

my mental model is: "if int/float are 1 slolt, and long/double are 2 slots", then 1 slot has to be 32 bits

cfleming07:01:24

Yeah, which is how primitives are definitely defined (I do remember that much).

qqq07:01:30

and if all of this is determined at ".java -&gt; .class" compile time, then this has to be somewhere in the specs

cfleming07:01:33

But I’m not sure about references.

cfleming07:01:33

> At any point in time, an operand stack has an associated depth, where a value of type long or double contributes two units to the depth and a value of any other type contributes one unit.

cfleming07:01:12

According to that, an object reference is one unit, but I don’t know how that works.

cfleming07:01:25

> The number of method parameters is limited to 255 by the definition of a method descriptor (§4.3.3), where the limit includes one unit for this in the case of instance or interface method invocations.

cfleming07:01:34

Looks like references are one unit.

qqq07:01:51

but surely, there are JVMs that have more than 2^32 = 4,000,000,000 objects? 🙂 there are azul system machines with TBs of RAM right?

cfleming07:01:08

I don’t understand it, but it’s specced here:

cfleming07:01:23

> A single local variable can hold a value of type boolean, byte, char, short, int, float, reference, or returnAddress. A pair of local variables can hold a value of type long or double.

cfleming07:01:40

I understand in the case of compressed OOPs, but not otherwise.

qqq07:01:41

A single local variable can hold a value of type boolean, byte, char, short, int, float, reference, or returnAddress.

qqq07:01:54

1. Thank you for digging this up for me. I really appreciate it.

qqq07:01:03

2. I missed the "referenced or returnAddress" part in the past.

qqq07:01:27

3. This raises more questions, i.e. what if > 2^32 objects, but that can be resolved another day.

jakob07:01:47

I have this code

(defmulti address-type (fn [a] (:address/use-delivery-address a)))
(defmethod address-type true [_]
  "true!")
(defmethod address-type false [_]
  "false!")

(address-type {:address/use-delivery-address false}) ;returns "false!" as expected
which works as expected. If I want to change how my “defmulti” works for this type by evaluating it again, it doesn’t seem to “overwrite” it.
(defmulti address-type (fn [a] (:address/something-else a)))
(address-type {:address/something-else false}) ; expect to still retunr "false!" but instead it throws an exception
WHY? 🙂

robert-stuttaford07:01:46

you need to recompile all the forms that make address-type, @karl.jakob.lind; you’re defmulti and defmethod are actually mutative calls to an in-memory dispatch database

robert-stuttaford07:01:02

e.g. call (methods address-type) to query that database

jakob07:01:49

but it doesnt work even when I recompile all defmethods. hm..

jakob07:01:13

get this exception No method in multimethod 'address-type' for dispatch value: null

robert-stuttaford07:01:57

a handy trick to resolve this sort of thing is to provide a default defmethod impl that logs its args; then you can inspect dispatch failures by seeing the actual data it’s inspecting

noisesmith18:01:11

the specific problem here is that defmulti has defonce semantics - you need to explicitly destroy the old multimethod before the new definition will compile

jakob07:01:20

thanks, will do some debugging

noisesmith18:01:45

I don’t know if you resolved this, or if you know this already, but in order to redefine a defmethod it doesn’t suffice to reload the code, you have to destroy the old defmethod. This can be done by destroying the var via ns-unmap or by turning it into something that isn’t a multimethod eg. (def my-multi nil) - after that a reload actually works

jakob10:01:34

I have wasted a few hours because I didn't knew about this. 😂 Thanks!

jakob08:01:45

Ok I am stuck again. the args of the default method didnt help me. it just shows what I expected: the data I send in

tianshu11:01:23

Hi, How can I listFiles in a folder in my resources in a uberjar?

noisesmith18:01:24

I think it’s helpful to remember that as far as Java is concerned, a Jar does not contain file objects. A file is strictly and specifically something you access via OS filesystem APIs, and is a specific instance of a thing you can read from given a path or name (URI). A URI can be a thing inside a jar, a URL indicating something you access via HTTP, an object served via FTP, or a File on disk.

tianshu11:01:23

thanks, I'll have a try.

tianshu11:01:39

@joelsanchez that works, thanks again.

joshkh13:01:01

has anyone come across a clojure library to handle barfing / slurping of s-expressions?

joshkh13:01:35

or a similar library to group infix boolean logic like so: (A and B or C and D) => (or (and A B) (and C D))

vikeri13:01:28

How do I define reader conditionals inside a macro? Say I have a macro that I use from both clj and cljs and want the macro to compile to js/Error in cljs and Exception in clj.

moxaj13:01:58

@vikeri (boolean (:ns &env)) will be true in cljs, false in clj

vikeri13:01:12

@moxaj I get an error: Unable to resolve symbol &env in this context

moxaj13:01:42

you can only access &env directly inside the macro

vikeri13:01:26

That’s what I tried to do

vikeri13:01:06

Or wait, I put it in a def out side the macro. Does that matter?

moxaj13:01:24

yeah, that won't work

moxaj13:01:29

&env is a hidden argument

vikeri13:01:41

Ah, I see. Now I got it working, thanks

qqq14:01:31

(do "** breaks" 
        (def class-data
          (let [ia (type (int-array 0))]
            {:name    'my.dyn.Tensor
             :fields  [{:flags #{:public :static}, :name "shape", :type ia } ]
             :methods [{:flags #{:public}, :name :init, :desc [ia],
                        :emit  [[:aload 0]
                                [:invokespecial :super :init [:void]] 
                                [:return]]}]}))

        (def class-object (insn/define class-data)) 
        (.newInstance class-object (int-array [1 2 3])))
(comment
 java.lang.IllegalArgumentException: No matching method found: newInstance for class java.lang.Class)

    
    (do "** works" 
        (def class-data
          (let [ia (type (int-array 0))]
            {:name    'my.dyn.Tensor
             :fields  [{:flags #{:public :static}, :name "shape", :type ia } ]
             :methods [{:flags #{:public}, :name :init, :desc [],
                        :emit  [[:aload 0]
                                [:invokespecial :super :init [:void]] 
                                [:return]]}]}))

        (def class-object (insn/define class-data)) 
        (.newInstance class-object ))
How do I debug this? The main difference is: :desc [ia] vs :desc []

grzm16:01:08

Is there an official channel for zprint help requests? I'd rather avoid creating noise issue tickets in github for what's likely clarification rather than actual issues.

qqq16:01:19

I'd just ask here: if there's a more appropriate channel, someone will redirect you.

grzm16:01:50

Cool. Well, there are a couple of things I'd like to format differently, in particular ns declarations (which I like to do in the style Stuart Sierra outlined here: https://stuartsierra.com/2016/clojure-how-to-ns.html), and protocols. I haven't figured out a way to get these to format as I'd like without stomping on the other function formats. I think if there were more :list options available I could figure out a way to do it.

grzm16:01:54

I try not to get into bikeshedding territory, but code format is pretty much by definition bikeshedding.

grzm16:01:53

And hacking into the internals of zprint might be yakshedding, a term a friend of mine recently introduced me to.

dpsutton17:01:38

issues will appear in google results though. it may actually benefit more people to be on github

hlship18:01:58

Looking for some opinions. We have an internal framework that we use to build our Component system; it's an EDN file that establishes what components to build (as fully qualified function names, typically for map->Record constructor functions) plus configuration and references. We hope to open source it in the near future. This approach has a lot of benefits, but one drawback; we end up with a lot of long, namespaced symbols. But because data is data, we have the option to use shorter symbols in the EDN file, but expand them to fully qualified symbols before passing them to the library to be instantiated. I came up with an approach based on EDN readers:

:my-service {:sc/create-fn #g auth/AuthService}
is expanded by the reader into the desired output:
:my-service {:sc/create-fn com.walmarts.example.auth/map->AuthService}
A co-worker has another option, where we use an alternative key instead:
:my-service {:ex/create-fn auth/AuthService}
... and we scan for :ex/create-fn and replace with the :sc/create-fn key, and the fully qualified function name. I'm wavering between the two solutions. The EDN reader solution involves settting up the readers before reading the EDN, but then the rest of the pipeline is the same. The co-worker's solution involves reading normally, then finding and converting the keys before continuing with the rest of the pipeline. Thoughts?

tbaldridge18:01:37

@hlship another option would be to somehow tie into the reader. What you're talking about here is a lot like what happens with keywords and the reader.

tbaldridge18:01:55

clojure
(require '[clojure.core :as c])

::c/+

hlship18:01:12

Well, I kind of like the #g marker that says "this value is treated specially".

tbaldridge18:01:15

the c/ gets expanded at read-time into clojure.core

tbaldridge18:01:42

I'd be tempted to turn the symbols into keywords, and add a hook for referring new namespaces

tbaldridge18:01:53

One thing that makes me hesitate with both of your suggestions is that they require custom logic for interpreting symbols. Logic that already exists in Clojure.

hlship18:01:29

To be clear, this is an .edn file, that's read in as data, then used to build a Component system.

hlship18:01:41

It would not be as big a deal if it was Clojure code.

tbaldridge18:01:42

actually a backtick would work for symbols as well

{:sc/create-fn `auth/AuthService}

hlship18:01:48

(clojure.edn/read-string "foo")
=> foo
(clojure.edn/read-string "`foo")
java.lang.RuntimeException: Invalid leading character: `

hlship18:01:35

Typically, some or all of the EDN files (which are deep merged together) will have runtime data from a container, or just be specific to the deployment environment (qa vs prod, etc.).

tbaldridge18:01:39

So that's the next question, does it "have" to be EDN? Something I haven't rectified in my own mind yet is why/when .edn is prefered over .clj files. Clojure code being data and all that.

tbaldridge18:01:12

As time progresses these EDN based data DSLs take on more and more logic, until they (poorly) implement a subset of Clojure 🙂

grzm18:01:08

recapitulating ant?

hlship18:01:22

One of the things I like about this approach is that initial REPL startup is faster: we're loading a lot less code; in our main namespace; most code gets loaded by the library when it resolves those fully qualified symbols.

tbaldridge18:01:26

But at any rate, I like the #g approach the best of your two solutions because that cleanly abstracts all the symbol interpretation logic into one place in the code.

tbaldridge18:01:58

@hlship sure, but nothing stops you from calling (load-file "my-config.clj") from within your app.

tbaldridge18:01:06

Or some eval based solution.

hlship18:01:34

Nothing requires the use of Component either, until you try to build an effective development work cycle without it. 🙂

tbaldridge18:01:47

Right, but what I'm saying is what's the use-case for EDN here? If you're looking for external configuration of the system, you can do that with runtime loading a .clj file, and you won't artificially restrict the power of your configurations.

potetm18:01:23

In case you were unaware

tbaldridge18:01:25

I've worked on several systems with EDN based configs like this and there's always confusion about what works and what doesn't and how you programmatically create the configs. All this goes away if you have Clojure available in the configuration language.

andy.fingerhut19:01:45

One thing to be cautious about there, of course, is that as soon as you use the Clojure reader rather than the EDN reader, you may open yourself up to arbitrary code execution. Sharp-equals club, I think it has been called? See notes on last example here (more for benefit of those new to this than for you @tbaldridge) http://clojuredocs.org/clojure.core/read

tbaldridge20:01:03

True, but how often do you read arbitrary configs from the outside world?

tbaldridge20:01:30

This is no different than saying that the JVM's ability to load .jar files is exposure arbitrary code execution.

hlship18:01:48

Our library takes ideas of integrant, and elsewhere, but layers on top of Component, something we're committed to at a large scale.

hlship18:01:42

But the thing I'm working on it standalone, and though I will continue to use Component, the other library could be excised in favor of the code-as-configuration approach.

hiredman19:01:14

I think I see this in a similar way to @tbaldridge, all the apps I've ever worked on use some kind of edn map for configuration, and at some point gain some kind of system for merging maps, so you have overlays or inheritance or something else, all of which are ways to move what could be explicit conditionals in code to implicit conditionals in the configuration "runtime"

hiredman19:01:37

so after spending some time poking at various configuration libraries and thinking about bootstrapping component, my crazy opinion is use a rules engine for configuration and bootstrapping

johnj19:01:14

@tbaldridge I hear a lot of mantra about data > code, maybe that's why people try to default to that

ghadi19:01:33

there's even more mantra around keep things simple

ghadi19:01:52

rules engines, inheritance, lots of complexity there

ghadi19:01:41

regarding the original question -- I prefer a symbol for the value, not a tagged value. Once you validate config it's easy to walk all the :sc/create-fn 's and resolve them

hiredman19:01:48

configuration for large software is a complex thing

ghadi19:01:03

maybe it shouldn't be

ghadi19:01:13

When I have complex config needs (that I can't avoid), I tend to make a thing very much like debian conf files except EDN or JSON 20-defaults.conf

ghadi19:01:26

I find a set of files, sort them, merge them

ghadi19:01:55

deep-merge vs plain merge is a knob you can control

ghadi19:01:14

whatever I choose, I like doing a validation pass before the app boots up

hiredman19:01:20

this is what I mean, your "simple" configuration system is actually hiding a complex set of conditionals

ghadi19:01:18

I didn't say it was simple, but to avoid complexity if you can

ghadi19:01:38

I see lots of effort wasted into config frameworks

ghadi19:01:24

(no offsense @hlship) but it seems like every organization has their riff on this sort of configuration framework

hlship19:01:40

That is so true.

ghadi19:01:44

both of my recent clojure jobs have had their own spin

hiredman19:01:14

and pretty much all of them are merging maps, with their own particular set of rules for the merging

hiredman19:01:00

and it is the most likely thing to be opensourced

ghadi19:01:32

almost all of them fail to consider validating input before the app boots up, often leading to a zombie state

seancorfield19:01:52

I'm guilty of creating the one we use at World Singles. I reviewed a whole bunch of them and none of them did everything we needed.

ghadi19:01:05

Heh. Who else is 'guilty'?

ghadi19:01:18

Immuconf is an interesting approach to overrides at least, but it's still just a map merge.

seancorfield19:01:23

(Well, I'm guilty of creating two different ones in fact -- the new one is a big improvement over the old one 🙂 )

hiredman19:01:28

databases of facts (maps) + rules (strategy for merging maps) = rules engine (configuration system)

seancorfield19:01:17

(ours has the ability to provide custom value merging functions -- although we don't use it at the moment)

hiredman19:01:40

once you start trying to deal with things like "read config value x from the node data on aws or from the environment variable X if not on aws" the map merge stuff is just kind of a joke

ghadi19:01:41

I consciously avoid having the config files be sentient about where to get config data. I usually have something (a "driver") that grabs from S3, or a Kubernetes ConfigMap, filesystem path, etc. and gives fully formed data into the "merge" process

hiredman19:01:26

funny you should say that, the last config system prototype I wrote looked just like that

ghadi19:01:33

but it doesn't pull from everywhere for a given app, usually one or two different places max

ghadi19:01:21

sort of a 12-factor or dependency injection lesson - don't give me the recipe for the thing, just gimme the thing

hiredman19:01:46

the issue with that is bootstrapping, how do you configure the S3 driver?

hiredman19:01:36

the general answer to that is some kind of bootstrap configuration

hiredman19:01:50

so now you have two configuration systems

potetm20:01:04

We went fully dynamic. Put it all in Datomic w/ a single edn file that provided the Datomic URI.

potetm20:01:39

(There were particular needs that pushed us that way. Probably not a good starting approach.)

hiredman20:01:44

that is the other thing, and some point someone will say "gee, I wish configuration was a dynamic thing we could update on the fly"

potetm20:01:40

Upside of datomic: history for all changes, caching is built-in, flexible schema

potetm20:01:59

very high read/very low write, kind of an ideal use case

sggdfgf21:01:59

how to take subset of map? only 2 particular keys? 😳

bellis21:01:55

(fn [m k1 k2] {k1 (m k1) k2 (m k2)}) 

sundarj21:01:19

(select-keys {:a 2, :b 3, :c 7} [:a :c])
{:a 2, :c 7}

sggdfgf21:01:34

thanks guys!

Victor Ferreira23:01:56

Hey guys, whats is the best and most simple webframework to make http apis in clojure?

hlship23:01:15

Use GraphQL and Lacinia

johnj23:01:03

ring with compojure

johnj23:01:45

or pedestal

johnj23:01:16

I hear it lacks docs/community

greglook23:01:27

Ring is about as simple as it gets; I like bidi better as a router though, since it doesn’t involve singleton route declarations.

kenrestivo23:01:08

bidi is very nice, used it a bit

aria4223:01:24

Anyone got hot tips on a good Clojure plotting library? Hard to find something that appears maintained, looking for something I can adapt and use with Jupyter kernel