Fork me on GitHub
#clojure
<
2018-12-05
>
Riles00:12:18

Could someone point me to a high level overview of the clojure internals?

hiredman00:12:08

what do you mean by internals? what happens at the repl when you hit enter? how the compiler works? how the datastructures work?

Riles00:12:13

Hm, I've been especially curious about the runtime and the datastructures, but I'm also curious how the compiler is structured, how it differs from the cljs compiler.

hiredman00:12:17

"runtime" is tricky to define the boundaries of

hiredman00:12:55

there is a clojure.lang.RT which is largely a bunch of static methods

hiredman00:12:22

it is written in java, and those methods are called by clojure code to do various things

hiredman00:12:38

clojure differs substantially internally from clojurescript

hiredman00:12:47

to some degree because large parts of clojure are written in java (including the compiler and the datastructures), and because clojurescript is to some degree dependent on clojure (the macro language for clojurescript is actually clojure not clojurescript)

Riles00:12:56

I've been reading through the clojure.lang java files so far. Is there a good entrypoint to start reading the compiler?

hiredman00:12:57

I might start with the eval method in Compiler.java and just start kind of walking through how it would execute for different forms

bronsa00:12:02

the compiler exists entirely in clojure.lang.Compiler

Riles00:12:07

thank you!

bronsa00:12:08

the reader is clojure.lang.LispReader

bronsa00:12:37

you can also read clojure.lang.DynamicClassLoader and clojure.lang.Reflector

bronsa00:12:50

those 4 classes will give you a good overview of the compilation process

❤️ 1
bronsa00:12:06

they're pretty heavy to get into though, especially the compiler code

bronsa00:12:07

Gary did a very good talk at this year's conj talking through how code is loaded/compiled, which is possibly higher level than what you want to learn but I would start there to learn about the high-level view

bronsa00:12:31

they all go in different level of details

Riles00:12:03

Thanks so much!!

hiredman00:12:26

clojurescript tends to lean pretty hard on macro's ability to emit (js* ...) calls which are just blobs of js that the compiler sticks in verbatim, which isn't something the clojure compiler can do (it is compiling to jvm bytecode not java source), so those kinds of things tend to end up as static methods on RT

hiredman00:12:36

the clojure compiler does however have some facilities for recognizing certain static method calls and replacing them directly with inlined bytecode

wei00:12:24

I'm getting a NullPointerException [trace missing] resetting an atom to a large map. any suggestions on next steps for troubleshooting? I'm able to spit the map to a file, so it doesn't look like the NPS is related to realizing the data.

hiredman00:12:34

there is a jvm option you need to set to stop it from omitting stacktraces

hiredman00:12:53

-XX:-OmitStackTraceInFastThrow

hiredman00:12:52

my guess is you don't actually have an atom, but nil

hiredman00:12:05

user=> (reset! nil 1)
NullPointerException   clojure.core/reset! (core.clj:2373)
user=> 

wei00:12:11

I see. it's a production environment so I can't easily change the jvm options

wei00:12:44

looks like it is an atom though-

core.data=> (type cells)
clojure.lang.Atom

hiredman00:12:25

but what makes you think cells is what is being reset?

hiredman00:12:34

the only other way I can think of to get an npe from a call to reset! is with a watcher, but then the exception will point to the watcher function, not the call to reset!

wei00:12:48

because

(reset! cells -x)
throws the npe for certain values of -x

wei01:12:23

I just tried some simple values and it doesn't seem to break

hiredman01:12:09

there is no value you can pass in as value to reset! there that will cause an npe

hiredman01:12:54

a commonish error is derefing the atom before calling reset!

hiredman01:12:39

for example

hiredman01:12:41

(let [some-atom (atom nil)
      cells @some-atom]
  (reset! cells 1))

wei01:12:23

hmm, that's good to know, thanks. I'll dig further and see if I can find a culprit

wei02:12:38

just to circle back, it was indeed a problem with realizing the data structure and not an issue with reset!. thanks @U0NCTKEV8 for the insights!

sgerguri01:12:49

I have been looking into duct and its module system. There are a few official modules that can be used with it, such as the web and the logging one. That said, I am struggling to understand the value add here - it saves the user some typing by providing some defaults but if I'm reading it correctly it is impossible to override said defaults in the configuration (unless something is flagged as ^:displace in the defaults) whilst at the same time hiding what those defaults are from the user; one needs to dig into the source code or simply prep the config to get its actual value. Is there an obvious use case or is it just flat out better to not use the automagic defaults and hand-roll with base duct or, better yet, base integrant?

jumar11:12:53

Btw. there's also #duct Maybe the author himself can provide some guidance 🙂.

👍 1
1
urzds10:12:26

I repeatedly have the problem that when I try to print the value of something, I get just a lazyseq. (str value) or (doall value) do not change that. What am I doing wrong?

noisesmith16:12:13

To go into more detail about this - the toString of a LazySeq doesn't show its contents. Doall doesn't change the type, it merely ensures all the elements are realized. If you use pr-str instead of str, or prn instead of println, it will actually reveal the contents

👍 1
schmee10:12:15

try vec

👍 1
schmee10:12:31

beware if the sequence is large or infinite though

Audrius12:12:07

did you guys have any Clojure step debugger? Like In Java there is very good which can also evaluate expressions in breakpoints. I would like something like that

potetm12:12:13

#cursive has one @masta

jumar12:12:26

You can do similar thing with #cider too. It's not as good as the Cursive's one (especially for debugging Java code 🙂 ) but you can get pretty far with it.

Saikyun13:12:49

if I want to create a "standalone" development environment, currently using leiningen, how do I get the dependencies to be used from the folder I'm working in?

Saikyun13:12:54

basically I have a normal leiningen project, then I used lein-libdir (https://github.com/djpowell/lein-libdir) to pull my dependencies into lib/. but when I try to use e.g. lein run it still tries to download the dependencies from maven

Saikyun13:12:49

is there a better way to do all this? basically I just want a folder with my code and all dependencies. it doesn't have to be with leiningen, but I want nREPL and whatever build tool I'm using needs to be run with pure java

Saikyun13:12:07

or clojure obv, but not any precompiled exes or whatever (I'm on windows)

pesterhazy14:12:31

Everytime I encounter it, I get taken aback by the fact that lein uberjar seems to evaluate namespaces and actually run the code that runs at ns init time

pesterhazy14:12:03

Does this happen generally or only when you use gen-class or AOT?

Saikyun14:12:47

I got it to work by adding the following in my project.clj

:plugins [[lein-libdir "0.1.1"]]
:local-repo "local-m2"  
:repositories {"local"  "./lib"}
and running lein libdir before moving the folder to my server

Saikyun14:12:31

(also have to put lein.bat and the standalone jar in the same folder and change the directories of lein.bat, but that was quite straightforward)

Saikyun14:12:36

lein repl doesn't work though, since it seems to download more dependencies...

tomaas14:12:45

hi, using the clj-http library how can I post with this type of this structure (example in curl)? I can't figure out how to pass items array correctly

manutter5114:12:10

@tomaas I think you probably need square braces around {"place" "a-place"}. The endpoint is probably expecting a list of places, but you’re only passing a place that’s not part of a list.

manutter5114:12:27

In other words, you need [{"place" "a-place"}]

tomaas14:12:06

just figured it out

tomaas14:12:15

:form-params {"items[][place]" "a-place" "user" "a-user"}

manutter5114:12:37

Ok, that looks plausible — just a direct mapping.

manutter5114:12:53

I assume that’s a non-clj endpoint you’re posting to?

tomaas14:12:09

@manutter51 correct

Matt Butler15:12:09

Is there a good resource for understanding the execution of clojure on a low/implementation level? If thats too vague lmk.

bronsa15:12:38

if you have particular questions I'd be happy to answer them

Matt Butler15:12:00

Thanks for these, I probably will in time have plenty of questions.

bronsa15:12:40

heh, I remembered you had one too but couldn't find had my life depended on it

Matt Butler15:12:55

Thanks Renzo, it was actually your talk and a discussion about if doing

(let [long-lazy-seq (range 1000)]
  (doseq [x long-lazy-seq]))
Would hold onto the head of the long-lazy-seq that sparked this question 🙂

Matt Butler15:12:47

Using range for an example here (was reading in a giant csv in reality). But once again, I don't know if that changes the properties, which is sort of the point 😄 i just don't know.

reborg15:12:42

I see… then maybe you don’t need to go down to the lower details, it seems more a lazy seq problem?

Matt Butler15:12:29

I don't actually have a problem that I need solving. Was merely an entry point into realising how little I knew about how it all worked 🙂

bronsa15:12:18

well understanding locals clearing requires going down to bytecode level to be fair

markbastian16:12:00

Hi all, I'm trying to get a grip on datafy and nav. Suppose I wanted to create a navigable file system. I could create something simple like this to datafy a file:

;Note: Just a strawman, not what you'd do in real life...
(extend-protocol clojure.core.protocols/Datafiable 
  java.io.File 
  (datafy [file]
    (if (.isDirectory file)
      {:files (into [] (.listFiles file))}
      {:name (.getName file)})))
Now, where I am scratching my head is figuring out how to navigate it. I am assuming I am starting with a datafied file, e.g.
(->> "." io/file datafy)
Let's say I want to navigate to the first file in the directory. Would the invocation be something like one of these?
;Seems mostly right, but idk what goes in the last position.
(-> "." io/file datafy (nav [:files 0] ?what-goes-here))
;Based on @searcorfield's blog, maybe the original file? This seems not right.
(let [f (io/file ".")]
  (-> f datafy (nav [:files 0] f)))
I am guessing if someone can steer me in the right direction on how to invoke nav the implementation of the Navigable protocol will be pretty clear. Thanks in advance!

alexmiller16:12:45

Sean had a good writeup on this stuff yesterday

ghadi16:12:33

^ quite good

alexmiller16:12:37

I think the answer to your question is, the file at position 0 (which you likely need to look up)

alexmiller16:12:50

that doesn’t need to be datafied, because datafy will be called on it

alexmiller16:12:03

at which point you’re back to the beginning

markbastian16:12:36

Ok, let me expand on @seancorfield's example. Suppose I have the following tables and data:

;... = more rows
;Students Table
[{:name "Mark" :id 1} ...]

;Classes Table
[{:name "Advanced REPL Usage" :duration "4 weeks"}
 {:name "Basic REBL Usage" :duration "1 week"} ...]

;Enrollment Table
[{:enrollee "Mark" :course-name "Advanced REPL Usage"}
 {:enrollee "Mark" :course-name "Basic REBL Usage"} ...]
In his example, he states you might do (nav row column value) so would you do?
(nav {:name "Mark" :id 1} :name "Mark")
This seems insufficient. Or, should the value be the destination. I could do this:
(nav {:name "Mark" :id 1} :name enrollment-table)
I could see this being used to produce a sequence of courses Mark is enrolled in. Final option:
(nav students-table [:name "Mark"] enrollment-table)
This doesn't take a column in the first position, but seems reasonable. Is one of these the "correct" way to expect to use nav? Thanks again!

thegeez16:12:34

I think you need (nav students-table 0 {:name "Mark" :id 1})

seancorfield16:12:21

@markbastian Yes, this is correct

(nav {:name "Mark" :id 1} :name "Mark")
-- you do (get row column) first to get the value, and then you call nav with that value.

markbastian16:12:53

How do I know where to nav to?

seancorfield17:12:06

The behavior for nav depends entirely on the implementation of Navigable you provided on the row (somehow -- metadata or a protocol extension).

markbastian17:12:36

So, would I only be able to nav to one location with that implementation?

NoahTheDuke17:12:52

Does that mean if you don't know what's in a given "cell", you'd have to write (nav m :name (:name m))?

seancorfield17:12:07

If you look at the java.jdbc.datafy namespace, it assumes by convention that a column called fooid is a foreign key into the table foo with the PK (`id`) value of fooid in that original row.

markbastian17:12:11

Or is the idea that there is an implict destination for a given row?

seancorfield17:12:35

Navigation is defined by whoever provides the protocol implementation.

seancorfield17:12:54

The implementation could provide multiple results if you wanted -- but for a given datafication of something, there will be one navigation since they are sort of inverses of each other.

markbastian17:12:55

Yeah, that's the part I'm thinking of right now. I realize you could implement it many ways and I'm trying to figure out if there's a "right" or "preferred" way.

seancorfield17:12:56

Does this ASCII graphic help?

Obj-> datafy -> Data
                  /
                /
              /
            nav
            /
          /
        /
      /
  NewObj -> datafy -> NewData

seancorfield17:12:33

When you nav, you are given (part of) the data representation of a thing and it navigates you to the corresponding part of the thing...

markbastian17:12:13

Yes, I think so. It's just the details that I think are not entirely clear ATM. I thought you navigated from a thing to a thing.

seancorfield17:12:15

The "right" way is "whatever semantics you need for datafication/navigation in your domain".

markbastian17:12:57

Suppose you are at a node in your graph (e.g. a row in a db) and you want to navigate to another node (e.g. you have your name and you could link out to a table with your enrollments or another with your children). Seems like you'd want some way to specify which place you want to go to.

seancorfield17:12:21

Yes, that specification would be encoded in the how nav works.

alexmiller17:12:22

I think I would call all of those Obj -> Data, NewObj -> NewData to perhaps clarify their roles

seancorfield17:12:00

@markbastian Have you looked at the java.jdbc example source code?

markbastian17:12:41

Is that in the clojure jdbc project?

alexmiller17:12:06

and you have to keep in mind that datafy has the opportunity to close over Obj and embed a nav impl in the metadata for Data

alexmiller17:12:13

so datafy can influence subsequent nav (and Obj itself can have metadata that influences how datafy works)

markbastian17:12:26

So, would it be safe to say that coll (nav coll k v) is the "source" you want to navigate from (The docs say "context of coll" but not what coll is), k is the index you want to go to, and v is your partial data describing "to"?

markbastian17:12:36

Well, I think maybe what I just said is incorrect

markbastian17:12:43

at least regarding "to"

thegeez17:12:44

Here's how I understand nav thus far: (nav coll k v) -> v is sufficient for 'plain' clojure data, where you navigate to further data that is already in the data you're seeing. Stu's talk contains two examples where where coll and k are needed. For files the datafy'ed representation is {:name "filename", :modified 123timestamp, :contents "<elided>"}. Navigating to "<elided>" does not show the actual string "<elided>", it will load the contents of the file. This is because the calling nav with the key :contents will open the file, as the code shows in this slide: http://www.youtube.com/watch?v=c52QhiXsmyI?t=16m08s. An example that depends on coll is the Datomic example. (nav datomic-db k long-entity-id-as-v) will pull the entity from the database as shown here: https://www.youtube.com/watch?v=c52QhiXsmyI?t=21m23

seancorfield17:12:52

Yes, it's linked to and explained in my blog post.

markbastian17:12:36

I'll need to re-read it (as well as read the source). I think that and looking at the source will help it sink in a bit.

seancorfield17:12:13

It was just a rough proof of concept, knocked out over lunch on Friday at the Conj. It doesn't leverage the additional metadata that datafy adds, for example, but it shows how nav's behavior can be driven by data that is closed over.

markbastian17:12:52

Yeah, I think the concept is awesome and this could be revolutionary. I'm just trying to soak in the implementation bits right now.

seancorfield17:12:29

As I say in that blog post, it's tough to really grok this until you've actually built something and seen it work in REBL...

seancorfield17:12:15

I spent most of that lunch break just "doing it wrong" 😐

markbastian17:12:29

I am at that stage right now.

markbastian17:12:46

But I appreciate all your help.

didibus18:12:08

It's trying to solve the n+1 problem I think

didibus18:12:45

Where you want to access a bunch of data. But for technical reason, that data is represented as more complex types and is potentially spread in many places, encrypted, etc.

didibus18:12:44

So you grab n things, datafy it to get rid of the technicalities of the data, and then you can drill down and up with nav, which under the hood handles the technicalities of the drill down/up

didibus18:12:28

Another way to think of it is that Clojure data can now have hyperlinks.

didibus18:12:38

So one piece of data can link to another, and nav is the function to navigate the hyperlink

didibus18:12:17

The piece of data with the link has metadata on it that tells nav how to do the navigation

didibus18:12:12

Actually, I'm patiently waiting for someone to implement a web crawler using Datafiable and Navigable

alexmiller19:12:56

wasnt that Stu’s opening demo?

didibus18:12:41

@markbastian

noisesmith19:12:05

that explanation makes me wonder if someone has implemented nav for adjacency-list - that form of normalized graph data would be a natural "lowest-common-denominator" target

noisesmith19:12:16

because it sounds like datafy would be able to do proper graphs rather than being limited to trees

alexmiller19:12:21

yes, and it already does for classes, etc

noisesmith19:12:41

:thumbsup: cool

alexmiller19:12:33

the key is that graphs and trees are not different, if you have laziness

hiredman19:12:14

nav is very interesting in a sort of clojure-core-ology(see kremlinology) sense. the way you pass in both the key and the value seems odd at first, but it really ties in with the model for spec, and for clojure in general. in spec most of the time you don't specify, uh, for lack of a better word, values on their own, you specify attributes (keys in maps). and in clojure of course there is an emphasis on reusing the same generic data structures and not putting data in specific types. nav, in a more type centered approach, might be an operation on a new Link type, or a Link protocol that gets extended to bits of data, but instead says the number 1 is just the number one, a bit of generic data, not any kind of link, but in the context of a :customer-id key of a particular map it is a link

👍 1
cfleming20:12:56

One question I have about nav - it doesn’t seem possible, given an object, to test if it’s navigatable, without actually invoking nav. This would be useful in a UI, for example, to know which data elements to render as links. Is there a good way to handle this?

richhickey21:12:00

it will either satisfy the protocol or have the metadata

richhickey21:12:29

remember that nav is on the container, not the item

richhickey21:12:39

maybe remember is the wrong word 🙂

cfleming21:12:54

But doesn’t everything satisfy the protocol because the protocol is extended to Object?

1
cfleming21:12:33

(satisfies? clojure.core.protocols/Navigable (Object.))
=> true

Lennart Buit21:12:04

you seem to be able to extend the protocol with metadata only, so you can check whether there is an impl in metadata maybe?

Lennart Buit21:12:18

but yeah, thats not the entire story

cfleming21:12:06

@lennart.buit I can implement my own types which implement the protocol directly, right?

Lennart Buit21:12:16

yeah, hence the “not the entire story”. You can only find some of the impls that way

dpsutton21:12:50

user=> (deftype A [])
user.A
user=> (satisfies? clojure.core.protocols/Navigable (A.))
true

cfleming21:12:04

Yeah, the type will satisfy the protocol whether it implements it directly or not.

cfleming21:12:49

I guess I could check for metadata or the object implementing the protocol’s interface, but that seems a bit janky.

hiredman21:12:33

that also won't catch if the protocol is extended afterwards

cfleming21:12:08

Very true. Is it possible to check if the protocol implementation is the Object one?

cfleming21:12:20

I’m assuming not from userland.

richhickey21:12:50

and I think it doesn’t go to the initial question, which will still come into play for containers/collections heterogeneous in link/non-link items - need to answer per path/key

richhickey21:12:55

asking a container if satisfies nav is only asking if it’s a browser, not that this tag is a link

cfleming21:12:33

Right, I really want to know “which elements within this thing can be browsed”

richhickey21:12:50

understood, would need a wider protocol

cfleming21:12:35

Or rather, “is this item inside this browser also browsable”, I guess.

richhickey21:12:30

(can-nav? same args as nav)

cfleming21:12:33

right, or navigable?. Although that might create confusion where people would expect the semantics of (satisfies? Navigable)

sashton22:12:50

Here’s some datafy/nav code I created to wrap my head around the concept: https://gist.github.com/sashton/56a14d275c71d7ce3e224c11c9a16bd4 And if I’m doing anything wrong there, I’m happy to hear feedback.

seancorfield23:12:15

That's pretty much what clojure.java.jdbc.datafy does by convention (assume <foo>id is a foreign key into table <foo>, assuming a PK of id -- can all be overridden with :schema option) -- which is to say, yeah, that looks pretty spot on.