This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-11-30
Channels
- # adventofcode (22)
- # announcements (7)
- # beginners (32)
- # calva (56)
- # cider (96)
- # cljdoc (7)
- # cljs-dev (50)
- # clojure (173)
- # clojure-austin (8)
- # clojure-brasil (7)
- # clojure-europe (10)
- # clojure-greece (2)
- # clojure-italy (10)
- # clojure-nl (9)
- # clojure-spec (18)
- # clojure-uk (143)
- # clojurebridge (3)
- # clojurescript (9)
- # cursive (14)
- # data-science (1)
- # datascript (4)
- # datomic (9)
- # docker (7)
- # emacs (2)
- # figwheel-main (4)
- # fulcro (18)
- # garden (1)
- # graphql (13)
- # hyperfiddle (4)
- # juxt (2)
- # off-topic (43)
- # pathom (1)
- # pedestal (17)
- # portkey (163)
- # re-frame (4)
- # reitit (7)
- # rum (4)
- # shadow-cljs (139)
- # spacemacs (5)
- # sql (14)
- # unrepl (2)
After initially being blown away by the implications of datafy/nav and the 1.10.0 metadata extensions wrt REBL I thought there was a snag applying it to some usecases I have resolving non edn data objects but no - there’s no snag at all… it should just work. I’m actually stunned at the simplicity and beauty of the design. Amazing work!
I’ve had a nagging feeling that a sidecar REPL viewer like this was the missing piece of clojure for quite a while; but I didn’t think it would be this good! it feels like a game changer… especially also with the predicate driven viewers… will they be extensible at all?
For now, no. But that could change.
Looks like you can start it from nrepl with (cognitect.rebl/ui)
and send it values with (cognitect.rebl/inspect [1 2 3 4 5 6])
— though I suspect there are much better ways to do this
Something that would be really cute is a version of scope-capture that works with REBL
Looks like you can also tap>
it values
I'm trying to parse about 1000 seperate csv files containing 60million rows total (all of the shape, with 10 columns) into one big in-memory datastructure, a map of columns. Currently the code I have to parse each file looks like this:
(defn ptx [a f]
(let [csv (.read csvr f StandardCharsets/UTF_8)
cols (map keyword (.getHeader csv))
rows (.getRows csv)
rows (for [row rows] (zipmap header (.getFields row)))]
(doseq [col cols]
(let [col-vals (map col rows)]
(swap! a update col (fnil concat []) col-vals)))))
with a
being an atom to accumulate the results across files
I think what's killing me is GC. Are there any general strategies I should use here?transducers on a transient
it'll get through about 900 of the CSV files quickly but the later ones are drastically larger the csvs start at about 30kb but by the last 200 files they're about 30MB each and this is where things start to slow to a crawl. But when I run this on a single 30MB csv file it's considerably faster than when it gets to the big files within a lop processing all the files
but also do you really need to hold it all in memory… and have you allocated enough?
You are creating a map for every row and then use it only for col lookup and throw it away, right? That’s a lot of garbage...
you don’t really need an atom or an identity here… just accumulate the rows in an accumulator in a reduce/transduce
I think I see how to get rid of the maps and where transients should be used, but do you see a specific spot where transducers are obvious?
essentially the whole thing… in particular the for
over rows to create maps will create a lot of garbage right now… so conj
(`!`)ing each row into rows
(the accumulator), will remove your GC overhead.
You can then probably make extra savings by making the accumulator a transient, to save the cost of the immutable updates. Also 60M swap!
s is probably going to have quite a bit of overhead, so using an accumulator will fix that. You don’t need to pay that cost if there’s no concurrency.
you could also probably parallelise it by file easily enough if you really wanted - but one step at a time and measure the perf gains as you go.
In my experience with this sort of thing the GC cost is the main one… Raw I/O is pretty good these days, particularly on SSD. When you really start pushing perf here the next big bottleneck you’ll hit is that java readers convert UTF-8 strings into 16bit chars… so then things get trickier to optimise. To get really fast you have to start hard coding knowledge of the file, e.g. column sizes, not converting all cols into strings unless you look at them, avoiding readers altogether etc…
when there’s a bunch of key concepts that also seem to overlap more with runtime contracts
I know racket contracts must have been looked at (well, I presume, since I don’t actually really know how these work)
which have some of the same structural guarantees of “if you give me these minimal requirements I guarantee as a post-condition these minimal provisions”
Hello! I want to explore a Java app, so I will start a nREPL server within it. But what is the best way to expose some data to it? Either they must be accessible through a static field/method or I guess I could use Clojure API from Java to create a var pointing to it. Right? Any better ideas? Thanks!
Yes, but how does clj code run in the repl get access to these objects?
if you have a class named org.Person
, then you can write:
(import 'org.Person) ;; import the class
(def p (Person.)) ;; run the constructor and define the object to a var
I do not want to create the data in the REPL. I have an instance of Person
already, from before I started the nrepl server, how do I expose it to the REPL???
well, where do you store the person? I'm guessing you have a reference to it somehow?
somewhere in your program, you will have some sort of static reference to a person, if nothing else in your main
function, from where all you instantiations originate, and I'm guessing the easiest way would be to from there expose the reference statically
FYI clojure.core/intern is the solution
Is it possible to somehow invoke def
from java? I expect Clojure.var("clojure.core", "def").invoke("xxx", 12345);
not to work...? Thanks!
Hi all, I am getting a strange error with which I hope someone can help me. When I connect to the MySQL database via the application that is running (in an API call) I get the error Could not create connection to database server. com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server.
. But when I call the same functions from within the REPL the database queries work fine. What can be the reason?
Stack trace:
Could not create connection to database server.
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server.
(...)
Caused by: java.lang.ClassCastException: java.io.BufferedWriter cannot be cast to java.io.PrintWriter
at cider.nrepl.middleware.out$print_stream$fn__52939.invoke (out.clj:99)
Hi guys! Anybody knows if there is a way how to have clojure.lang.DynamicClassLoader
as the base classloader in clojure.lang.RT
? Currently it seems that the default clojure's classloader is sun.misc.Launcher.AppClassLoader
(during runtime - so not in repl). I am having troubles loading some maven dependencies during runtime with JDK 1.9 and trying to find a way around it...
Alright it works when starting with lein repl
and lein figwheel
. Must be something in my Emacs configuration… I think with setting the printer somewhere.
is there any documentation regarding exposing c libraries to clojure through jna/i? is it practical?
I found it! thanks 🙂 https://www.youtube.com/watch?v=Lf-M1ZH6KME
Hi everyone. Does anyone know why Clojure’s read-string inserts spaces for consecutive quotations, as in this example here:
(read-string "(\"\"1\"\")")
=> ("" 1 "")
My expectation is that it would return something like: ("\"1\"")
@mpcarolin Those are three separate tokens.
I guess a follow up question is; has anyone tried using graalvm with clojure and calling c? seems like it'd be useful, maybe?
Trying to run REBL, getting java.lang.ClassNotFoundException: javafx.fxml.FXMLLoader
. Any thoughts?
Looks like JavaFX is a separate install for OpenJDK on ubuntu, sudo apt-get install openjfx
fixed it.
@dave.dixon JavaFX got separated out since Java 10+ as a separate project, so makes sense
I'm on 1.8.0_191, but whatever it's working. However everything renders really small on my HIDPI display. Is there a JVM flag or something to set the scaling?
@dave.dixon REBL only works on java 8 currently
I also run it fine with a retina display, I'm unsure what'd cause that
basically I have a function feeded with hashmaps that are equals according to data/diff, with equal type on every key/value but yet that yields differents result.
In these cases I usually fire up the mongo shell and see what results does that give.
I’m on mobile, so can’t help with code snippets, but I’d look to see what the final query to mongo looks like.
I think there is something called to-db-object
that converts a Clojure data structure to something that mongo understands. It should be easy to find the monger source responsible for that.
how do you bridge what you do in your clojure repl and what you see in the mongo shell
Wild guess - something with map key order? I know that sometimes mongo cares about that for some obnoxious reason.
hey that would be strangely possible since the key order between the two maps is different
there is one which is read from a file programmatically generated. The second is just the same fields in the existing database.
I just nailed down to the two problematic key-value pairs. In both case the key ordering change. So your theory holds.
I would look also at the documents themselves and see if there are mongo predicates that can overcome the ordering thing.
yeah, thanks you really helped on this one. According to mongodb documentation, nested map order matters.
It seems I have to convert the nested maps using dot notation for cracking that stuff
Oh good one. Do you mind opening a monger issue documenting this? I might have a crack in the coming time of a new mongo api wrapper and this is good to keep in mind.
Anyone know if there's an official advent of code for Clojure?
for this year
thanks
First very rough datafy
/`nav` additions to clojure.java.jdbc
are working: given the result of query
, REBL (that Stu showed off yesterday) can navigate through it, into other tables via foreign keys! (locally, not committed to git, so don't go looking for it just yet!)
that is super cool. theres a tool for C# that is amazing kinda like this called LINQPad
Now I have a proof of concept -- and a whole stack of questions/problems to figure out before I can actually turn it into useful, committable code 🙂
https://github.com/clojure/java.jdbc/blob/master/src/main/clojure/clojure/java/jdbc/datafy.clj
Very much "subject to change" right now...
Yeah, that's sort of leftover from when I was developing it and was actually calling datafy
and nav
to test that things worked 🙂
Er, the back tick is for clojure.core.protocols
as p
tho', right?
Removed the clojure.datafy
require since it wasn't needed. And dumped my thoughts about "schema" lookup into the ns
docstring.
I want it to look for an existing var -- so it will break for folks who accidentally try to load this on Clojure 1.9 or earlier 🙂
Right, but this is still experimental right now. Once I've figured out a "final" solution, I'll remove the datafy
namespace and add the metadata directly inside the main jdbc
namespace code instead (and use the full symbol names so it loads on < 1.10) and document it etc etc...
I'm currently contemplating a new, streamlined API (`clojure.java.jdbc2` probably) that removes a lot of the "knobs and dials" and speeds up a lot of the result set handling (which is quite an overhead right now), and the datafication may just go in there instead.
I somehow have managed to distill most of my work to tree walking operations... React... GraphQL... I can smell something maybe useful cooking if I had something like nav
and datafy
is this a good place to ask architectural questions? as in, how one might structure their code?
I have a two-player online card game, and the core of a given session's game state is a hash atom. It's stateless, so only when players take actions do functions get called, passing the state as the first variable, and changes are applied as swaps or whatnot depending. I inherited the codebase, and currently the main engine logic is in a single namespace spread across 11 files, which are load-file
by the top-level core.clj. There's some 6k lines in the core logic and 16k in the card definitions that relies on the core logic
Is it worth it for me to separate and move the core stuff into different namespaces?
I gave some advice on related issues here, which you can obviously take or leave 🙂 https://clojurians-log.clojureverse.org/clojure-uk/2018-11-10/1541861665.543000 The resulting discussion I think was quite interesting though.
i'll read through it! thanks!
Is using a atom the smartest way to maintain game state?
i have no idea what a stateless atom is. an atom is a mechanism to ensure atomic changes to state
let me remove that word then. What I meant it to say is that the atom doesn't contain any information about which part of the game we're at, so players are free to perform nearly any action whenever they want
kind of a mess of a question on reread, so my apologies
let me reduce the question before i walk away in shame, lmao: Is it better to have 5-10 different namespaces? Or 1 giant namespace? for roughly 6k lines of code
The thing that matters is ergonomics for the software author. Do you find yourself getting lost and not knowing which file to edit? Having to edit a list of files to do "one" change?
@nbtheduke I would argue that it depends whether you can break down those 6K lines into clear modules with a small interface between them. So it depends more on the logical organization of the code rather than the number of lines IMO
The convention is definitely small namespaces (ideally under 200 lines each(?)) separated by focus
Honestly combining all the files that are currently done via load-file into one file might even be an improvement. Eg. load-file means you can't use the code from a jar without unpacking the jar
200! Dang. So for a project of this size, if I even double that to 400, that's 15! Okay then. There's definitely issues with knowing where things are and having to (declare X Y Z)
at the top of files because functions are used before they're loaded
yeah - you might want to make a graph of calls / dependencies and see if natural clusters form...
😮 I could have troubles coping with 6k lines in one namespace
I assume you mean declare
above, of course
yes, so sorry
in the projects I've worked on is never > 500 probably
@nbtheduke Zachary Tellman's book about Clojure style has some really good material about organizing modules / namespaces / abstractions.
just bought it! can't wait to dive into this
there's also talks on youtube where he presents some of the chapters
Oh that's great, I'll read it! Def need to expand my wider clojure knowledge, cuz I only learned it for this project, haha
yeah - I also had to revisit my namespaces / module boundaries in a pretty radical way on my first real clojure project
it's especially an issue coming from languages where the rules and conventions around eg. classes tend to make a large number of decisions for you
with clojure (or any other lispy language) you then need to learn new conventions for organizing things, and the infrastructure of the language itself only puts weak pressure on those decisions
Agreed, and good to know. Thanks for the input y'all, I really appreciate it
looks like metadata is having a revival: datafy, nav, metadata protocols, REBL and the new AWS lib all use metadata in interesting ways
from what I’ve seen not many clojure libs make use of metadata, but maybe it’s time to re-evaluate what metadata can (and should) be used for?
I particularly like the idea of attaching the original version of a transformed object as metadata, that’s something I might stea… erhm, borrow, for my own libs 😄
Very rough work in progress https://github.com/clojure/java.jdbc/blob/master/src/main/clojure/clojure/java/jdbc/datafy.clj
very cool! 😄 how does it play with older clojure versions? will it only take effect if you explicitly require the datafy namespace?
@nbtheduke Zach’s writing on processes in that book is gold. As is https://www.destroyallsoftware.com/talks/boundaries and https://www.jamesshore.com/Blog/Testing-Without-Mocks.html#a-frame-arch
The trick is to separate what’s side-effecting and what’s pure. @ericnormand has also explored this territory lately in his podcast ver at http://purelyfunctional.tv
Oh hell yeah, thanks for the links!
Is there a neat way to tidy up something like this?
(str x (when y (str "__" y)) …)
Where is not a consistent separator (you can't use str/join!)(defn my-joiner
[start pairs]
(let [suffix (->> pairs
(filter #(some? (second %)))
flatten)]
(apply str start suffix)))
#'cljs.user/my-joiner
cljs.user> (my-joiner "blah" [["--" nil] ["**" 3]])
"blah**3"
This is the logical thing, although I guess it's not as pretty as the current (format %s__%s-%s")
(which doesn't work due to nil
.)
A bit of a temptation to define something magical using macros like https://github.com/clojure/core.incubator/blob/master/src/main/clojure/clojure/core/strint.clj
@slipset @nbtheduke I believe you are referring to https://lispcast.com/podcast
He’s got one on stratified architecture (from sicp) which is something very different from your run of the mill three tiers ejb mess.