This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # adventofcode (22)
- # announcements (7)
- # beginners (32)
- # calva (56)
- # cider (96)
- # cljdoc (7)
- # cljs-dev (50)
- # clojure (173)
- # clojure-austin (8)
- # clojure-brasil (7)
- # clojure-europe (10)
- # clojure-greece (2)
- # clojure-italy (10)
- # clojure-nl (9)
- # clojure-spec (18)
- # clojure-uk (143)
- # clojurebridge (3)
- # clojurescript (9)
- # cursive (14)
- # data-science (1)
- # datascript (4)
- # datomic (9)
- # docker (7)
- # emacs (2)
- # figwheel-main (4)
- # fulcro (18)
- # garden (1)
- # graphql (13)
- # hyperfiddle (4)
- # juxt (2)
- # off-topic (43)
- # pathom (1)
- # pedestal (17)
- # portkey (163)
- # re-frame (4)
- # reitit (7)
- # rum (4)
- # shadow-cljs (139)
- # spacemacs (5)
- # sql (14)
- # unrepl (2)
After initially being blown away by the implications of datafy/nav and the 1.10.0 metadata extensions wrt REBL I thought there was a snag applying it to some usecases I have resolving non edn data objects but no - there’s no snag at all… it should just work. I’m actually stunned at the simplicity and beauty of the design. Amazing work!
I’ve had a nagging feeling that a sidecar REPL viewer like this was the missing piece of clojure for quite a while; but I didn’t think it would be this good! it feels like a game changer… especially also with the predicate driven viewers… will they be extensible at all?
Looks like you can start it from nrepl with
(cognitect.rebl/ui) and send it values with
(cognitect.rebl/inspect [1 2 3 4 5 6]) — though I suspect there are much better ways to do this
Something that would be really cute is a version of scope-capture that works with REBL
I'm trying to parse about 1000 seperate csv files containing 60million rows total (all of the shape, with 10 columns) into one big in-memory datastructure, a map of columns. Currently the code I have to parse each file looks like this:
(defn ptx [a f] (let [csv (.read csvr f StandardCharsets/UTF_8) cols (map keyword (.getHeader csv)) rows (.getRows csv) rows (for [row rows] (zipmap header (.getFields row)))] (doseq [col cols] (let [col-vals (map col rows)] (swap! a update col (fnil concat ) col-vals)))))
abeing an atom to accumulate the results across files I think what's killing me is GC. Are there any general strategies I should use here?
it'll get through about 900 of the CSV files quickly but the later ones are drastically larger the csvs start at about 30kb but by the last 200 files they're about 30MB each and this is where things start to slow to a crawl. But when I run this on a single 30MB csv file it's considerably faster than when it gets to the big files within a lop processing all the files
but also do you really need to hold it all in memory… and have you allocated enough?
You are creating a map for every row and then use it only for col lookup and throw it away, right? That’s a lot of garbage...
you don’t really need an atom or an identity here… just accumulate the rows in an accumulator in a reduce/transduce
I think I see how to get rid of the maps and where transients should be used, but do you see a specific spot where transducers are obvious?
essentially the whole thing… in particular the
for over rows to create maps will create a lot of garbage right now… so
conj(`!`)ing each row into
rows (the accumulator), will remove your GC overhead.
You can then probably make extra savings by making the accumulator a transient, to save the cost of the immutable updates. Also 60M
swap!s is probably going to have quite a bit of overhead, so using an accumulator will fix that. You don’t need to pay that cost if there’s no concurrency.
you could also probably parallelise it by file easily enough if you really wanted - but one step at a time and measure the perf gains as you go.
In my experience with this sort of thing the GC cost is the main one… Raw I/O is pretty good these days, particularly on SSD. When you really start pushing perf here the next big bottleneck you’ll hit is that java readers convert UTF-8 strings into 16bit chars… so then things get trickier to optimise. To get really fast you have to start hard coding knowledge of the file, e.g. column sizes, not converting all cols into strings unless you look at them, avoiding readers altogether etc…
when there’s a bunch of key concepts that also seem to overlap more with runtime contracts
I know racket contracts must have been looked at (well, I presume, since I don’t actually really know how these work)
which have some of the same structural guarantees of “if you give me these minimal requirements I guarantee as a post-condition these minimal provisions”
Hello! I want to explore a Java app, so I will start a nREPL server within it. But what is the best way to expose some data to it? Either they must be accessible through a static field/method or I guess I could use Clojure API from Java to create a var pointing to it. Right? Any better ideas? Thanks!
if you have a class named
org.Person, then you can write:
(import 'org.Person) ;; import the class (def p (Person.)) ;; run the constructor and define the object to a var
I do not want to create the data in the REPL. I have an instance of
Person already, from before I started the nrepl server, how do I expose it to the REPL???
well, where do you store the person? I'm guessing you have a reference to it somehow?
somewhere in your program, you will have some sort of static reference to a person, if nothing else in your
main function, from where all you instantiations originate, and I'm guessing the easiest way would be to from there expose the reference statically
Is it possible to somehow invoke
def from java? I expect
Clojure.var("clojure.core", "def").invoke("xxx", 12345); not to work...? Thanks!
Hi all, I am getting a strange error with which I hope someone can help me. When I connect to the MySQL database via the application that is running (in an API call) I get the error
Could not create connection to database server. com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server.. But when I call the same functions from within the REPL the database queries work fine. What can be the reason?
Could not create connection to database server. com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. (...) Caused by: java.lang.ClassCastException: java.io.BufferedWriter cannot be cast to java.io.PrintWriter at cider.nrepl.middleware.out$print_stream$fn__52939.invoke (out.clj:99)
Hi guys! Anybody knows if there is a way how to have
clojure.lang.DynamicClassLoader as the base classloader in
clojure.lang.RT ? Currently it seems that the default clojure's classloader is
sun.misc.Launcher.AppClassLoader (during runtime - so not in repl). I am having troubles loading some maven dependencies during runtime with JDK 1.9 and trying to find a way around it...
Alright it works when starting with
lein repl and
lein figwheel. Must be something in my Emacs configuration… I think with setting the printer somewhere.
is there any documentation regarding exposing c libraries to clojure through jna/i? is it practical?
Hi everyone. Does anyone know why Clojure’s read-string inserts spaces for consecutive quotations, as in this example here:
My expectation is that it would return something like:
(read-string "(\"\"1\"\")") => ("" 1 "")
I guess a follow up question is; has anyone tried using graalvm with clojure and calling c? seems like it'd be useful, maybe?
Trying to run REBL, getting
java.lang.ClassNotFoundException: javafx.fxml.FXMLLoader. Any thoughts?
Looks like JavaFX is a separate install for OpenJDK on ubuntu,
sudo apt-get install openjfx fixed it.
@dave.dixon JavaFX got separated out since Java 10+ as a separate project, so makes sense
I'm on 1.8.0_191, but whatever it's working. However everything renders really small on my HIDPI display. Is there a JVM flag or something to set the scaling?
basically I have a function feeded with hashmaps that are equals according to data/diff, with equal type on every key/value but yet that yields differents result.
In these cases I usually fire up the mongo shell and see what results does that give.
I’m on mobile, so can’t help with code snippets, but I’d look to see what the final query to mongo looks like.
I think there is something called
to-db-object that converts a Clojure data structure to something that mongo understands. It should be easy to find the monger source responsible for that.
how do you bridge what you do in your clojure repl and what you see in the mongo shell
Wild guess - something with map key order? I know that sometimes mongo cares about that for some obnoxious reason.
hey that would be strangely possible since the key order between the two maps is different
there is one which is read from a file programmatically generated. The second is just the same fields in the existing database.
I just nailed down to the two problematic key-value pairs. In both case the key ordering change. So your theory holds.
I would look also at the documents themselves and see if there are mongo predicates that can overcome the ordering thing.
yeah, thanks you really helped on this one. According to mongodb documentation, nested map order matters.
It seems I have to convert the nested maps using dot notation for cracking that stuff
Oh good one. Do you mind opening a monger issue documenting this? I might have a crack in the coming time of a new mongo api wrapper and this is good to keep in mind.
First very rough
datafy/`nav` additions to
clojure.java.jdbc are working: given the result of
query, REBL (that Stu showed off yesterday) can navigate through it, into other tables via foreign keys! (locally, not committed to git, so don't go looking for it just yet!)
that is super cool. theres a tool for C# that is amazing kinda like this called LINQPad
Now I have a proof of concept -- and a whole stack of questions/problems to figure out before I can actually turn it into useful, committable code 🙂
Yeah, that's sort of leftover from when I was developing it and was actually calling
nav to test that things worked 🙂
clojure.datafy require since it wasn't needed. And dumped my thoughts about "schema" lookup into the
I want it to look for an existing var -- so it will break for folks who accidentally try to load this on Clojure 1.9 or earlier 🙂
Right, but this is still experimental right now. Once I've figured out a "final" solution, I'll remove the
datafy namespace and add the metadata directly inside the main
jdbc namespace code instead (and use the full symbol names so it loads on < 1.10) and document it etc etc...
I'm currently contemplating a new, streamlined API (`clojure.java.jdbc2` probably) that removes a lot of the "knobs and dials" and speeds up a lot of the result set handling (which is quite an overhead right now), and the datafication may just go in there instead.
I somehow have managed to distill most of my work to tree walking operations... React... GraphQL... I can smell something maybe useful cooking if I had something like
is this a good place to ask architectural questions? as in, how one might structure their code?
I have a two-player online card game, and the core of a given session's game state is a hash atom. It's stateless, so only when players take actions do functions get called, passing the state as the first variable, and changes are applied as swaps or whatnot depending. I inherited the codebase, and currently the main engine logic is in a single namespace spread across 11 files, which are
load-file by the top-level core.clj. There's some 6k lines in the core logic and 16k in the card definitions that relies on the core logic
Is it worth it for me to separate and move the core stuff into different namespaces?
I gave some advice on related issues here, which you can obviously take or leave 🙂 https://clojurians-log.clojureverse.org/clojure-uk/2018-11-10/1541861665.543000 The resulting discussion I think was quite interesting though.
i have no idea what a stateless atom is. an atom is a mechanism to ensure atomic changes to state
let me remove that word then. What I meant it to say is that the atom doesn't contain any information about which part of the game we're at, so players are free to perform nearly any action whenever they want
let me reduce the question before i walk away in shame, lmao: Is it better to have 5-10 different namespaces? Or 1 giant namespace? for roughly 6k lines of code
The thing that matters is ergonomics for the software author. Do you find yourself getting lost and not knowing which file to edit? Having to edit a list of files to do "one" change?
@nbtheduke I would argue that it depends whether you can break down those 6K lines into clear modules with a small interface between them. So it depends more on the logical organization of the code rather than the number of lines IMO
The convention is definitely small namespaces (ideally under 200 lines each(?)) separated by focus
Honestly combining all the files that are currently done via load-file into one file might even be an improvement. Eg. load-file means you can't use the code from a jar without unpacking the jar
200! Dang. So for a project of this size, if I even double that to 400, that's 15! Okay then. There's definitely issues with knowing where things are and having to
(declare X Y Z) at the top of files because functions are used before they're loaded
yeah - you might want to make a graph of calls / dependencies and see if natural clusters form...
@nbtheduke Zachary Tellman's book about Clojure style has some really good material about organizing modules / namespaces / abstractions.
Oh that's great, I'll read it! Def need to expand my wider clojure knowledge, cuz I only learned it for this project, haha
yeah - I also had to revisit my namespaces / module boundaries in a pretty radical way on my first real clojure project
it's especially an issue coming from languages where the rules and conventions around eg. classes tend to make a large number of decisions for you
with clojure (or any other lispy language) you then need to learn new conventions for organizing things, and the infrastructure of the language itself only puts weak pressure on those decisions
looks like metadata is having a revival: datafy, nav, metadata protocols, REBL and the new AWS lib all use metadata in interesting ways
from what I’ve seen not many clojure libs make use of metadata, but maybe it’s time to re-evaluate what metadata can (and should) be used for?
I particularly like the idea of attaching the original version of a transformed object as metadata, that’s something I might stea… erhm, borrow, for my own libs 😄
Very rough work in progress https://github.com/clojure/java.jdbc/blob/master/src/main/clojure/clojure/java/jdbc/datafy.clj
very cool! 😄 how does it play with older clojure versions? will it only take effect if you explicitly require the datafy namespace?
@nbtheduke Zach’s writing on processes in that book is gold. As is https://www.destroyallsoftware.com/talks/boundaries and https://www.jamesshore.com/Blog/Testing-Without-Mocks.html#a-frame-arch
Is there a neat way to tidy up something like this?
Where is not a consistent separator (you can't use str/join!)
(str x (when y (str "__" y)) …)
(defn my-joiner [start pairs] (let [suffix (->> pairs (filter #(some? (second %))) flatten)] (apply str start suffix))) #'cljs.user/my-joiner cljs.user> (my-joiner "blah" [["--" nil] ["**" 3]]) "blah**3"
This is the logical thing, although I guess it's not as pretty as the current
(format %s__%s-%s") (which doesn't work due to
A bit of a temptation to define something magical using macros like https://github.com/clojure/core.incubator/blob/master/src/main/clojure/clojure/core/strint.clj
He’s got one on stratified architecture (from sicp) which is something very different from your run of the mill three tiers ejb mess.
@dominicm could you calculate your separators and interleaved them with the strings and then join at the end?