Fork me on GitHub
#clojure
<
2018-11-30
>
rickmoynihan00:11:33

After initially being blown away by the implications of datafy/nav and the 1.10.0 metadata extensions wrt REBL I thought there was a snag applying it to some usecases I have resolving non edn data objects but no - there’s no snag at all… it should just work. I’m actually stunned at the simplicity and beauty of the design. Amazing work!

rickmoynihan00:11:29

I’ve had a nagging feeling that a sidecar REPL viewer like this was the missing piece of clojure for quite a while; but I didn’t think it would be this good! it feels like a game changer… especially also with the predicate driven viewers… will they be extensible at all?

alexmiller00:11:33

For now, no. But that could change.

rickmoynihan00:11:27

Looks like you can start it from nrepl with (cognitect.rebl/ui) and send it values with (cognitect.rebl/inspect [1 2 3 4 5 6]) — though I suspect there are much better ways to do this

rickmoynihan00:11:56

Something that would be really cute is a version of scope-capture that works with REBL

rickmoynihan00:11:44

Looks like you can also tap> it values

jjttjj01:11:53

I'm trying to parse about 1000 seperate csv files containing 60million rows total (all of the shape, with 10 columns) into one big in-memory datastructure, a map of columns. Currently the code I have to parse each file looks like this:

(defn ptx [a f]
  (let [csv    (.read csvr f StandardCharsets/UTF_8)
        cols   (map keyword (.getHeader csv))
        rows   (.getRows csv)
        rows   (for [row rows] (zipmap header (.getFields row)))]
    (doseq [col cols]
      (let [col-vals (map col rows)]
        (swap! a update col (fnil concat []) col-vals)))))
with a being an atom to accumulate the results across files I think what's killing me is GC. Are there any general strategies I should use here?

rickmoynihan01:11:54

transducers on a transient

jjttjj01:11:59

it'll get through about 900 of the CSV files quickly but the later ones are drastically larger the csvs start at about 30kb but by the last 200 files they're about 30MB each and this is where things start to slow to a crawl. But when I run this on a single 30MB csv file it's considerably faster than when it gets to the big files within a lop processing all the files

rickmoynihan01:11:41

but also do you really need to hold it all in memory… and have you allocated enough?

orestis01:11:01

You are creating a map for every row and then use it only for col lookup and throw it away, right? That’s a lot of garbage...

jjttjj01:11:05

Yeah I've set the JVM options to have enough memory. thanks for the tips!

rickmoynihan01:11:20

you don’t really need an atom or an identity here… just accumulate the rows in an accumulator in a reduce/transduce

jjttjj01:11:22

I think I see how to get rid of the maps and where transients should be used, but do you see a specific spot where transducers are obvious?

jjttjj01:11:38

ok cool so like the outermost thing

rickmoynihan01:11:52

essentially the whole thing… in particular the for over rows to create maps will create a lot of garbage right now… so conj(`!`)ing each row into rows (the accumulator), will remove your GC overhead. You can then probably make extra savings by making the accumulator a transient, to save the cost of the immutable updates. Also 60M swap!s is probably going to have quite a bit of overhead, so using an accumulator will fix that. You don’t need to pay that cost if there’s no concurrency.

jjttjj01:11:45

awesome, this should keep me busy for a bit, thanks so much

rickmoynihan01:11:57

you could also probably parallelise it by file easily enough if you really wanted - but one step at a time and measure the perf gains as you go.

rickmoynihan02:11:09

In my experience with this sort of thing the GC cost is the main one… Raw I/O is pretty good these days, particularly on SSD. When you really start pushing perf here the next big bottleneck you’ll hit is that java readers convert UTF-8 strings into 16bit chars… so then things get trickier to optimise. To get really fast you have to start hard coding knowledge of the file, e.g. column sizes, not converting all cols into strings unless you look at them, avoiding readers altogether etc…

👍 4
elliot03:11:49

if people haven’t seen it

elliot03:11:45

I think it’s interesting that the comparisons are usually to type systems

elliot03:11:20

when there’s a bunch of key concepts that also seem to overlap more with runtime contracts

elliot03:11:40

I know racket contracts must have been looked at (well, I presume, since I don’t actually really know how these work)

elliot03:11:54

there’s also the somewhat less popular but also interesting eiffel contracts

elliot03:11:05

which have some of the same structural guarantees of “if you give me these minimal requirements I guarantee as a post-condition these minimal provisions

👍 4
Jakub Holý09:11:45

Hello! I want to explore a Java app, so I will start a nREPL server within it. But what is the best way to expose some data to it? Either they must be accessible through a static field/method or I guess I could use Clojure API from Java to create a var pointing to it. Right? Any better ideas? Thanks!

Saikyun09:11:07

you can create the objects you want to look at as well 🙂

Saikyun09:11:42

or maybe that's what you mean by >create a var pointing to it

Saikyun09:11:53

that's what I usually do anyway.

Saikyun09:11:17

then using clojure.reflect/reflect a lot

👍 4
Jakub Holý10:11:49

Yes, but how does clj code run in the repl get access to these objects?

Saikyun12:11:36

if you have a class named org.Person, then you can write:

(import 'org.Person)  ;; import the class
(def p (Person.))                 ;; run the constructor and define the object to a var

Saikyun12:11:50

then (reflect p) or whatever 🙂

Jakub Holý14:11:40

I do not want to create the data in the REPL. I have an instance of Person already, from before I started the nrepl server, how do I expose it to the REPL???

Saikyun15:11:13

well, where do you store the person? I'm guessing you have a reference to it somehow?

Saikyun15:11:54

somewhere in your program, you will have some sort of static reference to a person, if nothing else in your main function, from where all you instantiations originate, and I'm guessing the easiest way would be to from there expose the reference statically

Saikyun15:11:13

not sure if there are other ways to do it 😛

Jakub Holý13:12:02

FYI clojure.core/intern is the solution

Saikyun12:12:41

oh, cool. thanks for reporting back 🙂

👍 4
Jakub Holý14:11:48

Is it possible to somehow invoke def from java? I expect Clojure.var("clojure.core", "def").invoke("xxx", 12345); not to work...? Thanks!

athos14:11:52

Probably you can do that using clojure.core/intern.

4
❤️ 4
erwinrooijakkers14:11:08

Hi all, I am getting a strange error with which I hope someone can help me. When I connect to the MySQL database via the application that is running (in an API call) I get the error Could not create connection to database server. com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server.. But when I call the same functions from within the REPL the database queries work fine. What can be the reason?

erwinrooijakkers14:11:27

Stack trace:

Could not create connection to database server.
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server.
(...)
Caused by: java.lang.ClassCastException: java.io.BufferedWriter cannot be cast to java.io.PrintWriter
 at cider.nrepl.middleware.out$print_stream$fn__52939.invoke (out.clj:99)

miro14:11:02

Hi guys! Anybody knows if there is a way how to have clojure.lang.DynamicClassLoader as the base classloader in clojure.lang.RT ? Currently it seems that the default clojure's classloader is sun.misc.Launcher.AppClassLoader (during runtime - so not in repl). I am having troubles loading some maven dependencies during runtime with JDK 1.9 and trying to find a way around it...

erwinrooijakkers14:11:46

Alright it works when starting with lein repl and lein figwheel. Must be something in my Emacs configuration… I think with setting the printer somewhere.

Saikyun15:11:40

is there any documentation regarding exposing c libraries to clojure through jna/i? is it practical?

dpsutton15:11:25

i think @lvh had a talk on that at a conj a few years back

mpcarolin15:11:31

Hi everyone. Does anyone know why Clojure’s read-string inserts spaces for consecutive quotations, as in this example here:

(read-string "(\"\"1\"\")")   
=> ("" 1 "")
My expectation is that it would return something like: ("\"1\"")

potetm15:11:26

@mpcarolin Those are three separate tokens.

potetm15:11:48

The spaces in the printed value are just for readability.

potetm15:11:10

iow ("" 1 "") and (""1"") are the same data structure

Saikyun15:11:14

I guess a follow up question is; has anyone tried using graalvm with clojure and calling c? seems like it'd be useful, maybe?

sparkofreason15:11:07

Trying to run REBL, getting java.lang.ClassNotFoundException: javafx.fxml.FXMLLoader. Any thoughts?

reborg16:11:43

What’s your java -version?

sparkofreason16:11:09

Looks like JavaFX is a separate install for OpenJDK on ubuntu, sudo apt-get install openjfx fixed it.

benzap16:11:41

@dave.dixon JavaFX got separated out since Java 10+ as a separate project, so makes sense

sparkofreason16:11:37

I'm on 1.8.0_191, but whatever it's working. However everything renders really small on my HIDPI display. Is there a JVM flag or something to set the scaling?

Charleshd16:11:40

Hi all, I'm experiencing a very strange behavior with Monger

Robert Randolph16:11:27

@dave.dixon REBL only works on java 8 currently

Robert Randolph16:11:50

I also run it fine with a retina display, I'm unsure what'd cause that

Charleshd16:11:22

basically I have a function feeded with hashmaps that are equals according to data/diff, with equal type on every key/value but yet that yields differents result.

Charleshd16:11:50

have you ever experienced smthg similar ?

orestis16:11:32

@chu can you share a snippet? Is this as part of a find query?

Charleshd16:11:46

yes, it is part of a find query !

Charleshd16:11:21

Let me do a snippet of it

orestis16:11:51

In these cases I usually fire up the mongo shell and see what results does that give.

orestis16:11:35

I’m on mobile, so can’t help with code snippets, but I’d look to see what the final query to mongo looks like.

orestis16:11:01

I think there is something called to-db-object that converts a Clojure data structure to something that mongo understands. It should be easy to find the monger source responsible for that.

Charleshd16:11:05

how do you bridge what you do in your clojure repl and what you see in the mongo shell

orestis16:11:26

See above :)

Charleshd16:11:42

@orestis according to "to-db-object" they are equals, lets try them in mongo shell

orestis16:11:13

Unfortunately I don’t know how readable is the output of that :(

orestis16:11:46

Wild guess - something with map key order? I know that sometimes mongo cares about that for some obnoxious reason.

Charleshd16:11:39

hey that would be strangely possible since the key order between the two maps is different

orestis16:11:40

A snippet would be useful now :)

orestis16:11:09

I guess you are creating those programmatically rather than typing them out?

Charleshd16:11:12

there is one which is read from a file programmatically generated. The second is just the same fields in the existing database.

Charleshd16:11:11

I just nailed down to the two problematic key-value pairs. In both case the key ordering change. So your theory holds.

Charleshd16:11:01

but why, and more importantly how to fix that behavior :thinking_face:

orestis16:11:52

You can use array-map or sorted-map but both have caveats AFAICT.

orestis16:11:23

I would look also at the documents themselves and see if there are mongo predicates that can overcome the ordering thing.

orestis16:11:31

Can’t help much more from mobile, sorry.

Charleshd16:11:26

yeah, thanks you really helped on this one. According to mongodb documentation, nested map order matters.

Charleshd17:11:53

It seems I have to convert the nested maps using dot notation for cracking that stuff

orestis17:11:46

Oh good one. Do you mind opening a monger issue documenting this? I might have a crack in the coming time of a new mongo api wrapper and this is good to keep in mind.

Charleshd17:11:29

Ok, let's dot it right away.

markbastian17:11:25

Anyone know if there's an official advent of code for Clojure?

markbastian17:11:31

for this year

lilactown17:11:06

what do you mean by "official"?

lilactown17:11:13

#adventofcode is a place

seancorfield17:11:05

First very rough datafy/`nav` additions to clojure.java.jdbc are working: given the result of query, REBL (that Stu showed off yesterday) can navigate through it, into other tables via foreign keys! (locally, not committed to git, so don't go looking for it just yet!)

🎉 20
👀 12
dpsutton17:11:18

that is super cool. theres a tool for C# that is amazing kinda like this called LINQPad

seancorfield17:11:41

Now I have a proof of concept -- and a whole stack of questions/problems to figure out before I can actually turn it into useful, committable code 🙂

dominicm19:11:30

Dynamic resolution of a protocol? 🙈

seancorfield19:11:50

Very much "subject to change" right now...

dominicm19:11:26

If you used ' instead of ` then it would work even without requiring datafy!

dominicm19:11:31

just occurred to me

seancorfield19:11:32

Yeah, that's sort of leftover from when I was developing it and was actually calling datafy and nav to test that things worked 🙂

seancorfield19:11:24

Er, the back tick is for clojure.core.protocols as p tho', right?

seancorfield20:11:25

Removed the clojure.datafy require since it wasn't needed. And dumped my thoughts about "schema" lookup into the ns docstring.

dominicm20:11:03

'clojure.core.protocols/datafy

dominicm20:11:16

going long form means it doesn't look for an actual existing var

seancorfield21:11:20

I want it to look for an existing var -- so it will break for folks who accidentally try to load this on Clojure 1.9 or earlier 🙂

dominicm21:11:48

If it doesn't break with clojure <1.10, then it can be always loaded!

seancorfield23:11:06

Right, but this is still experimental right now. Once I've figured out a "final" solution, I'll remove the datafy namespace and add the metadata directly inside the main jdbc namespace code instead (and use the full symbol names so it loads on < 1.10) and document it etc etc...

seancorfield23:11:14

I'm currently contemplating a new, streamlined API (`clojure.java.jdbc2` probably) that removes a lot of the "knobs and dials" and speeds up a lot of the result set handling (which is quite an overhead right now), and the datafication may just go in there instead.

lilactown17:11:30

that's awesome. I'm super hyped about nav

lilactown17:11:40

I'm very eager to get both of those in CLJS

lilactown17:11:52

I somehow have managed to distill most of my work to tree walking operations... React... GraphQL... I can smell something maybe useful cooking if I had something like nav and datafy

NoahTheDuke18:11:19

is this a good place to ask architectural questions? as in, how one might structure their code?

NoahTheDuke18:11:12

I have a two-player online card game, and the core of a given session's game state is a hash atom. It's stateless, so only when players take actions do functions get called, passing the state as the first variable, and changes are applied as swaps or whatnot depending. I inherited the codebase, and currently the main engine logic is in a single namespace spread across 11 files, which are load-file by the top-level core.clj. There's some 6k lines in the core logic and 16k in the card definitions that relies on the core logic

NoahTheDuke18:11:34

Is it worth it for me to separate and move the core stuff into different namespaces?

rickmoynihan10:12:37

I gave some advice on related issues here, which you can obviously take or leave 🙂 https://clojurians-log.clojureverse.org/clojure-uk/2018-11-10/1541861665.543000 The resulting discussion I think was quite interesting though.

NoahTheDuke16:12:01

i'll read through it! thanks!

NoahTheDuke18:11:03

Is using a atom the smartest way to maintain game state?

dpsutton18:11:57

i have no idea what a stateless atom is. an atom is a mechanism to ensure atomic changes to state

NoahTheDuke18:11:04

let me remove that word then. What I meant it to say is that the atom doesn't contain any information about which part of the game we're at, so players are free to perform nearly any action whenever they want

NoahTheDuke18:11:35

kind of a mess of a question on reread, so my apologies

NoahTheDuke18:11:42

let me reduce the question before i walk away in shame, lmao: Is it better to have 5-10 different namespaces? Or 1 giant namespace? for roughly 6k lines of code

noisesmith18:11:12

The thing that matters is ergonomics for the software author. Do you find yourself getting lost and not knowing which file to edit? Having to edit a list of files to do "one" change?

benoit18:11:19

@nbtheduke I would argue that it depends whether you can break down those 6K lines into clear modules with a small interface between them. So it depends more on the logical organization of the code rather than the number of lines IMO

👍 8
noisesmith18:11:42

The convention is definitely small namespaces (ideally under 200 lines each(?)) separated by focus

noisesmith18:11:18

Honestly combining all the files that are currently done via load-file into one file might even be an improvement. Eg. load-file means you can't use the code from a jar without unpacking the jar

👍 4
NoahTheDuke18:11:18

200! Dang. So for a project of this size, if I even double that to 400, that's 15! Okay then. There's definitely issues with knowing where things are and having to (declare X Y Z) at the top of files because functions are used before they're loaded

noisesmith18:11:03

yeah - you might want to make a graph of calls / dependencies and see if natural clusters form...

andrea.crotti18:11:31

😮 I could have troubles coping with 6k lines in one namespace

noisesmith18:11:35

I assume you mean declare above, of course

NoahTheDuke18:11:41

yes, so sorry

andrea.crotti18:11:52

in the projects I've worked on is never > 500 probably

noisesmith18:11:12

@nbtheduke Zachary Tellman's book about Clojure style has some really good material about organizing modules / namespaces / abstractions.

NoahTheDuke19:11:19

just bought it! can't wait to dive into this

noisesmith18:11:28

there's also talks on youtube where he presents some of the chapters

NoahTheDuke18:11:29

Oh that's great, I'll read it! Def need to expand my wider clojure knowledge, cuz I only learned it for this project, haha

noisesmith18:11:03

yeah - I also had to revisit my namespaces / module boundaries in a pretty radical way on my first real clojure project

noisesmith18:11:38

it's especially an issue coming from languages where the rules and conventions around eg. classes tend to make a large number of decisions for you

noisesmith18:11:29

with clojure (or any other lispy language) you then need to learn new conventions for organizing things, and the infrastructure of the language itself only puts weak pressure on those decisions

NoahTheDuke18:11:54

Agreed, and good to know. Thanks for the input y'all, I really appreciate it

schmee19:11:43

looks like metadata is having a revival: datafy, nav, metadata protocols, REBL and the new AWS lib all use metadata in interesting ways

schmee19:11:58

from what I’ve seen not many clojure libs make use of metadata, but maybe it’s time to re-evaluate what metadata can (and should) be used for?

👍 4
schmee19:11:35

I particularly like the idea of attaching the original version of a transformed object as metadata, that’s something I might stea… erhm, borrow, for my own libs 😄

😄 4
schmee19:11:40

kinda like namespaced keywords had a revival when spec was released

schmee19:11:31

very cool! 😄 how does it play with older clojure versions? will it only take effect if you explicitly require the datafy namespace?

slipset20:11:26

The trick is to separate what’s side-effecting and what’s pure. @ericnormand has also explored this territory lately in his podcast ver at http://purelyfunctional.tv

NoahTheDuke20:11:18

Oh hell yeah, thanks for the links!

dominicm20:11:41

Is there a neat way to tidy up something like this?

(str x (when y (str "__" y)) …)
Where is not a consistent separator (you can't use str/join!)

dpsutton20:11:06

(defn my-joiner
  [start pairs]
  (let [suffix (->> pairs
                    (filter #(some? (second %)))
                    flatten)]
    (apply str start suffix)))
#'cljs.user/my-joiner
cljs.user> (my-joiner "blah" [["--" nil] ["**" 3]])
"blah**3"

dpsutton20:11:18

simple function to filter the nils out and then join them?

dominicm20:11:07

This is the logical thing, although I guess it's not as pretty as the current (format %s__%s-%s") (which doesn't work due to nil.)

dominicm20:11:01

I'm betting there's something in cl-format

slipset20:11:37

Yes, same difference :)

slipset20:11:01

He’s got one on stratified architecture (from sicp) which is something very different from your run of the mill three tiers ejb mess.

slipset20:11:45

@dominicm could you calculate your separators and interleaved them with the strings and then join at the end?

dpsutton20:11:25

yeah just filter out pairs of string/separator where the string is nil and apply str to them