Fork me on GitHub

if I have [1 2 [3 4 [5 6 7]]], can I get a pointer to [5 6 7]?


yes, but the answer depends on what rule leads you to it, and what you mean by "pointer"


(peek (peek v)) returns that specific immutable vector, for example, and in java all first class non primitive values are pointers into the heap


but you might want an actual pointer reference you could pass to a C program by address?


(that's possible, depends on which lib you use for ffi)


@noisesmith I have a tree with each node keyed like [:node1 "foo" [:child1 "bar" [:child1-1 "a"] [:child1-2 "b"]] [:child2 "baz"]] and I want to return a node and its children given a key.


The plan was to have a hashmap with the node keys paired to node pointers for near constant speed access.


a binding is a pointer, you can reduce over the tree-seq of the collection then call group-by


that gives close-enough-to-constant-time access via binding each key in the map to the corresponding subtree



user=> (->> [:node1 "foo" [:child1 "bar" [:child1-1 "a"] [:child1-2 "b"]] [:child2 "baz"]] (tree-seq coll? seq) (filter coll?) (group-by second) pprint)
   [:child1 "bar" [:child1-1 "a"] [:child1-2 "b"]]
   [:child2 "baz"]]],
 "bar" [[:child1 "bar" [:child1-1 "a"] [:child1-2 "b"]]],
 "a" [[:child1-1 "a"]],
 "b" [[:child1-2 "b"]],
 "baz" [[:child2 "baz"]]}


due to the mechanics of group-by each val has an extra vector around it (in case the same key came up twice)


it's straightforward to write a "key-by" that works like group-by without the vector


looks good speedwise


especially if you do multiple lookups it should be good


looks like it can consume quite a bit of space if the tree is deep


it uses only one kv pair worth of memory per entry beyond your input


it's not copying the data, just indexing into it


that's safe, since the input is immutable


hmm ok that's perfect then


I'll digest it a bit


@deadghost possible definition of "key-by" plus the output when using it instead of group-by

user=> (defn key-by [f coll] (into {} (map (juxt f identity)) coll))
user=> (->> [:node1 "foo" [:child1 "bar" [:child1-1 "a"] [:child1-2 "b"]] [:child2 "baz"]] (tree-seq coll? seq) (filter coll?) (key-by second) pprint)
  [:child1 "bar" [:child1-1 "a"] [:child1-2 "b"]]
  [:child2 "baz"]],
 "bar" [:child1 "bar" [:child1-1 "a"] [:child1-2 "b"]],
 "a" [:child1-1 "a"],
 "b" [:child1-2 "b"],
 "baz" [:child2 "baz"]}


dumb question: is clojure.lang.Box a thing one could use or is it an implementation detail?

Alex Miller (Clojure team)05:08:49

It is impl and you shouldn't use it :)


First time I've ever heard of it so I'd say "implementation detail".


yeah i first heard of it 5 minutes ago in this stack overflow post


the top answer quotes some source that mentions it


It's a weird class. A single public member val of type Object 👀


all of the clojure implementation feels like it was written in a way to intentionally make people who write java mad


Hahaha... yeah, almost all Java devs hate Rich's indentation/brace style in his Java code too 🙂


i hate the huge blocks of commented out code


I can see a parallel between that and Rich Comment Forms... Java doesn't really have a way to say "Here's some code that I want to use for testing/benchmarking things but I don't want it in the main compiled code"...


It's not worth the effort of trying to shoehorn that into some sort of standalone test code, given the way Java build tools work, so I can understand this as a compromise.


(hate was too strong of a word - more a strong eh)


found an example not related to test code


Given the evolution of Clojure, I actually find that useful: it shows how the code was and what it is now, at a glance. That heads off people who think they've found a bug and go "Hey! prepRet should probably check Integer and Float right?"


Since now they'll look at it and go "Oh! It used to check those, so there's got to be a good reason why it doesn't now"


That's a really interesting perspective! Although I still don't think code should be left in, because you can't leave all changes in just in case someone finds the history useful. If I find something that seems like a bug, checking git history is part of the investigation process before reporting it. Still, its an interesting use case of leaving commented out code in.


The cost of leaving code in is that it makes reading the code that is actually used harder to read and it brings up questions of why there's code left there (is it important for it to be put back, but has some issues? was it a temp fix for something? was it an old no-longer relevant idea? who knows)


As someone who has spent a lot of time debugging other people’s code, browsing/grepping through a codebase with commented out code is a total distraction.


I certainly wouldn't advocate the practice for my code/my team -- and in fact that's one of the things I always pick up in a pull request code review: version control is the way to handle removal of code -- but I suspect Rich is using the commented out code as a visual reminder of why the current behavior is as it is.


The "why" use case seems to be exactly what comments should be for. I mean, I'm not going to argue with Rich, if it works for him, great, but in a more general discussion I would say version control to remove code and comments to explain why the implementation is how it is.


(mind you, commenting out vs removing isn't consistent)


And, to be fair, even the Clojure code in the compiler/runtime isn't idiomatic a lot of the time -- and that's almost always true of compiler/runtime system code. I've worked with a lot of compilers over the years and you often have to do some serious contortions to produce the right result in a performant manner.


Back in the 80's I wrote large parts of a C compiler and runtime, based on a VM architecture, and we chose to avoid all implementation-defined, undefined, and unspecified behavior in the system itself -- so you end up writing a lot of peculiar C code for that.


I've also worked on a COBOL compiler (written in COBOL!) and several other compilers/interpreters/etc. So much non-idiomatic code.


Package and tool maintainers: Do you pre-compile your .class files and bundle them into the Clojars release? What are the tradeoffs here, I feel like you could run into JVM version issues on user's machines?


My theory is for libraries, don't bother. The user can decide how they manage AOT. For tools where you value ease of use and startup time (I'm thinking *.main namespaces with a CLI), it might be worth it?


As long as different JVM/JDK versions won't cause it to barf.


An alternative route for CLI tools is Graal native image, like clj-kondo does. But not all of Clojure is supported of course.


True, I would rather keep it as just a Clojars JAR + a main namespace, but that's an option.

👍 4

It doesn't need to be that fast.


And the distribution of binaries keeps me up at night.


Just interested in if there are risks of horrible subtle errors if you AOT for your users or not.


Like will compiling on the oldest/newest JDK I have access to make a difference in compatibility.

Alex Miller (Clojure team)12:08:07

There are risks and it’s preferable to release as source

Alex Miller (Clojure team)12:08:59

In particular you should only aot if everything you depend on is also aot (otherwise you can get into loading issues, particularly around protocols)

Alex Miller (Clojure team)12:08:28

Jdk version is usually not an issue (unless you use APIs that were just added)

Alex Miller (Clojure team)12:08:24

Clojure version does matter (you are baking in compiled references as of that Clojure version). We try to never remove or break that stuff, but it’s possible to encounter breakage

Alex Miller (Clojure team)12:08:19

It’s really best to leave as source so the consumer can make the choice

Alex Miller (Clojure team)12:08:17

If you do want to aot, then you can also do both and publish under a classifier. We do this with a few contrib libs like tools.reader for clojurescript


👍 sounds good. One of my tools is AOT and can stay that way because I control the whole thing. The other one will be a dep so I'll leave that.


Thanks for the insight!

Alex Miller (Clojure team)13:08:11

those seem like good choices - aot is best at level when you're in control


does anyone have any tips/resources/libraries/examples for mechanically generating Clojure wrappers for large Java APIs?


Probably quite clever but a comprehensive example.


just what I’m looking for, thanks! :thumbsup:


No problem!

Alex Miller (Clojure team)12:08:03

My tip would be to not do that :) just use interop


yeah, I’m leaning towards that as well


since Clojure doesn’t support overloading you need to special case any method with overloading anyway (which in my case is most methods)


as often, the solution is “don’t be smart” 🙂

Alex Miller (Clojure team)12:08:18

The whole point of having non-wrapper interop in Clojure was to avoid wrappers

Alex Miller (Clojure team)12:08:31

Adding wrappers just makes everything slower through var indirection and/or reflection


in this case it is warranted IMO (the Java Vector API) due to the trickiness of type hinting / inlining you need to use that effectively

Alex Miller (Clojure team)12:08:07

That’s a reasonable argument


@U064X3EF3 - when you say avoid wrappers do you mean auto generated wrappers? A lot of times, I 'wrap' my interop in CLJ functions that take->return CLJ data structures - I consider this to be wrappers. Am I missing something?

Alex Miller (Clojure team)14:08:35

auto generated wrappers are particularly prone to this. a lot of times people hand-create clj wrappers that don't add additional value

👍 4
Alex Miller (Clojure team)14:08:05

but that does not necessarily apply to all wrappers, some do more and provide value

Alex Miller (Clojure team)14:08:59

for example java.jdbc / next.jdbc are not "just wrappers", they provide a lot of additional functionality


It's in a very early stage but I'm attempting this with my library Iboga for Interactive Brokers API In this case interop was just so painful over the years


(although this also adds functionality)


thanks for the clarification @U064X3EF3


clojure spec uses a global registry of keys. also uses a global registry of keys. (There's also spec coerce and a few other things that lean on spec). Are there any other examples of libraries or open source apps that use a registry of keys with functionality assigned to each key?


true but I'm thinking more about things that implement functionality for "nouns" at the application level


spec lets you to assign specs to these "things" and then you get a set of operations for those things. Integrant let's you start with data associated the things then implement multimethods for them

Alex Miller (Clojure team)15:08:45

they are both global addressing schemes for locating something


yeah i'm starting to think about if there's a way to separate the registry itself from the operations


like so you could just have a big map atom of the globals keys then each value would be a map that could look like

{:spec int? :ig-init #(.start %) :render render-fn} 
(horrible example sorry these would never really go together)

Alex Miller (Clojure team)15:08:39

why not just use a multimethod or protocol in that case?

Alex Miller (Clojure team)15:08:46

a protocol is actually implemented as a map of functions


hmmm yeah that might work. I was sort of thinking that it also might be useful to store data here and not just functions, and to have this be enumerable. This might be mixing up too much stuff though

Alex Miller (Clojure team)15:08:49

there are cases where doing stuff like this is useful, but if you can mix the stuff already in the box, it's likely to be less fragile over time

Alex Miller (Clojure team)15:08:59

protocols have a lot of special advantages (perf, callsite caching, integration with java, inline implementation in records+types+reify, etc) so think carefully about the tradeoffs


by protocol you mean using a record type to implement the protocol on instead of using global keys right?

Alex Miller (Clojure team)15:08:06

doesn't have to be a record type. protocols can be extended to other things and as of 1.10, can be supplied as metadata

👍 4

woah, didn't know that about 1.10, cool thanks!


So you'd need a record/type for each noun


For thoose of you who are doing "non-blocking" coding with e.g core.async or manifold. How do you tame jdbc? Do you hide things behind e.g an ExecutorService?


jdbc is always synchronous, so yeah your only option is to submit it to a threadpool and return futures/channels


if we had to do some jdbc stuff we would use a manifold future, which uses an ExecutorService behind the scenes and presents the same promise-like interface of manifold deferred


You can use core.async/thread


it is likely that a future JVM that includes Fibers will not have to do anything special to be non-blocking, even with JDBC


it will be possible to have the semantics of <! >! within an ordinary thread, no need for go blocks

Graham Seyffert18:08:57

> it is likely that a future JVM that includes Fibers will not have to do anything special to be non-blocking, even with JDBC Project Loom can’t come soon enough 🙏


How can I use rlwrap for sane arrow behavior but also use jline for reading a single character in Java?


it seems once you use jline to read a single char, rlwrap behavior stops working


can't jline also give you arrow behavior?


maybe, but I don't know how it works. I just googled around for how you read a single char in clojure and then I found that


I think on a TTY level, once you want character as opposed to line oriented actions it's one or the other


right, maybe I shouldn't go down this rabbit hole...


they both want to control the terminal IO


also line-based right?


any reason tools.deps CLI is using rlwrap vs this?

Alex Miller (Clojure team)18:08:10

they have their pros and cons but the main thing is that it's an os dep rather than a java dep so using a tool wrapper approach worked out better


but it has jline interop so you should be able to get character input too once it's set up, one would hope


I'm not sure. Perhaps it's that rlwrap is a simpler solution out of the box, and pulls in less libs.


going with rlwrap for now and just lines... this is for another time. thanks


there's also lanterna for more detailed terminal control


I stumbled on that, but the source code seems to have disappeared


ah thanks, I got a couple of dead links to bitbucket 😄


but if using jline for line+characters is already out of scope, lanterna is even further left of field


yeah, I'm going to figure this out for a next iteration


(.read *in*)


I think rlwrap even interferes with that


user=> (char (.read *in*))


it interferes by requiring a newline


it's not a single character consumed


doesn't work on my machine, fwiw


what does "doesn't work" mean?


you want to read a single character without enter, as @noisesmith suggests, or some other error?


the most common way to break that is running in nrepl


without enter, correct


you need true terminal controls to get that unbuffered access


(aka jline or lanterna or the like)


I just wrote a tool for that


@ahiguera coll - is it cross platform? (noticing it's a C lib...)


oh, docker :D


How can I remove a map from a vector? Let's say I want to remove the map with :id 2 and I have:

(def fruits [{:id 1 :name "apple"} {:id 2 :name "orange"} {:id 3 :name "water melon"}])
Is it ok or should I use another data design?

Alex Miller (Clojure team)19:08:33

you can't remove things from the middle of a vector (efficiently)


With a correctly-working core.rrb-vector you can ! I would still recommend a map if it is all keyed by :id, though, and they are not equal to the vector index.


Heavy heavy aside, here, I know.


hum what is the constant complexity like with those, compared to clojure's persistent vectors?


Yet to be measured, once they are working and stable. They should start out very close to Clojure's persistent vectors, but I am hoping from the research paper that they will not degrade by more than a factor of 2 or so as you do more subvec (aka "slice") and/or catvec (aka concatenate or "splice") operations on them.


sounds like some interesting properties, make sure to advertise on completion !


Definitely will. I've been focusing on getting some bugs fixed in it first, before measuring performance or looking for optimization opportunities.

Alex Miller (Clojure team)19:08:41

why not a map keyed by id?


it cool be nice yeah, but I was lazy to figure how update my forloop 😅


Should I use doseq and map keyed by id as you suggest?


(def fruits [{:id 1 :name "apple"} {:id 2 :name "orange"} {:id 3 :name "water melon"}])
=> #'user/fruits
(def indexed-fruits (into {} (map (juxt :id identity)) 
=> #'user/indexed-fruits
=> {1 {:id 1, :name "apple"}, 2 {:id 2, :name "orange"}, 3 {:id 3, :name "water melon"}}
(dissoc indexed-fruits 2)
=> {1 {:id 1, :name "apple"}, 3 {:id 3, :name "water melon"}}
;; lazy
(remove (comp #{2} :id) fruits)
=> ({:id 1, :name "apple"} {:id 3, :name "water melon"})


I think I'm going with

(def fruits {:1 "apple" :2 "orange" :3 "water melon"})
(for [[k v] fruits]
Edit: it was just a question of destructuring


map "keys" can be any clojure "data" (aka you can do {1 "apple" ...}


is there a way to signify that java sources should be compiled with deps.edn?


deps.edn isn't in that business


also, what are the "rules" for when i point deps.edn to a library on git


if i just had a java project with a pom.xml would that work?


what about a leiningen project?


or does it have to be a pure clojure, deps.edn specified thing


I have used deps.edn to specify deps that are Maven-central published Java libraries just fine.


They had no deps.edn file of their own, but presumably must have a pom.xml


I believe (Alex will chime in with the gospel) that clj will read transitive deps from pom.xml and project.clj -- but note, if the project needs javac compilation (likely with pom.xml) it will not do that for you


a good rule of thumb is you can only use other deps.edn libraries as git dependencies, it is a little more nuanced the that, it does support some other formats


but there are more corner cases in the other formats


it definitely doesn't support anything that requires a build step as a git dependnecy


so java source is right out


thats...somewhat frustrating


I still don't fully understand the point of deps.edn


like leiningen and maven cover |------------------| these use cases


deps.edn covers |------| these


but i dont know what to use for the remaining |----------|


in theory i like that it is a more focused and narrow tool


@emccue I think this is / is meant to be / will be the other remaining part


I haven't tried it, but this might be an option for compiling a java project for deps.edn


using mvn to compile the java looks like a good choice to me


hmm - maybe not, its docs don't actually mention using javac in mixed clojure/java repos


there are tools that add a build step on top of deps.edn, but I think most of those tools are aimed at building your own java source, not building the source of dependencies, but maybe there is one somewhere that does that as well


oh yeah, that is a distinct issue - it seems like the java ecosystem isn't really set up for source deps


its somewhat annoying because deps.edn is the only thing that works with git deps directly afaik


so if i put a library on a private git and that is my method of deployment i am locked into only using it with pure clojure projects


because i can only pull it with another deps.edn

Alex Miller (Clojure team)21:08:10

you can build it and deploy it into a maven repo (public or private) and use it from any build tool


yeah that seems to be the way to go in general

Alex Miller (Clojure team)21:08:55

clojure has the advantage of being a language that can work from source. git deps let you take advantage of that, when possible.

Alex Miller (Clojure team)21:08:04

when not possible, fall back into the "java way"


what options are there for private maven repos these days?


ive used artifactory before but only as a consumer


i have no clue what it takes to set up something like that in a reliable way


you can just stick stuff in an s3 bucket if your needs are pretty minimal


you can mvn install directly to local cache if you aren't depending on an external CI


that s3 bucket option sounds interesting


yeah, the s3 mvn thing works mostly

Alex Miller (Clojure team)21:08:58

deps / clj are set up to work from s3 repos

Alex Miller (Clojure team)21:08:10

so you don't need to add anything additional to do that

Alex Miller (Clojure team)21:08:56

it uses the s3-wagon-private thing but fixes several obnoxious things about it (which yes, I've filed upstream)


what are the cons of using s3?


I know with terraform there is a whole system with a locking table w/ dynamo to maintain atomicity


so i would anticipate race-condition flavored concerns


I don't think people use/think about maven repos in that way


this is simple put into an s3 bucket


the s3 wagon works great

Alex Miller (Clojure team)21:08:59

well, I wouldn't say "great", but it does kind of work


good to hear there are fixes coming to s3 private wagon

Alex Miller (Clojure team)21:08:45

it relies on an ancient version of the amazon stuff and the way it expects creds is totally broken - there's an issue there with all the details. I hacked over it for the deps usage of it


we use it at work, but at random times it reads metadata during our build process, even when just recompiling CLJS. might also be a boot thing, haven't figured it out


oh yeah... I forgot the creds stuff

Alex Miller (Clojure team)21:08:23

I've actually rewritten the whole mess (mostly) using the cognitect aws-api client, just haven't gotten around to finishing it, but will swap that whole thing out at some point


would be cool to have that as a drop-in replacement

Alex Miller (Clojure team)21:08:55

it's not a wagon, I'm actually using the newer maven apis directly

Alex Miller (Clojure team)21:08:19

so it won't be useful for mvn / lein afaik


Also last I checked there's the cosmetic issue that s3-wagon-private prints stack trace logs every time you check it for a dep that isn't in that repo


Do you have a link to a bug? Wasn't aware of that one


this is the closest match I can find in the repo, it appears fixed anyway


I thought it was after 2015 I had this problem though - maybe time flies :/


so you can override repo search order to make sure it's checked last, or sort through lots of spam


a very neat project for someone looking for something to do would would be a way to run maven repos p2p ala syncthing or ipfs

Alex Miller (Clojure team)21:08:49

the usage side and connection to tools.deps for something like that is pretty easy

Alex Miller (Clojure team)21:08:34

it depends whether you want to play the maven game or the tools.deps procurer game

Alex Miller (Clojure team)21:08:30

if you just want to run <handwave> some kind of storage </handwave>, tools.deps doesn't require much

Alex Miller (Clojure team)21:08:05

I've looked at npm for example, and I think it would be trivial to hook that into the resolver. what you actually do with the result of that re building classpaths, I'm not so sure


Yeah, the classpath relationship is a little more dubious I suppose. There's a clear link between clojars and npm dependencies, but npm dependencies don't necessarily belong in the classpath. Sometimes they do though! Eg for serving css or for building sass. It's more questionable when it comes to clojurescript reading npm dependencies from the classpath. But that might actually work out of the box. For something like planck/lumo it gets harder, because they need a single folder I think, but that folder can be selected via a env var.


(import 'leftpad) of course

😉 4
Alex Miller (Clojure team)21:08:36

if it would be useful to cljs to somehow have npm modules (w/transitive deps), I think that's easily within reach, but that's a solution in search of a problem


It definitely would be useful.


Currently it's a pain (impossible?) For libraries to depend on npm dependencies. So you have to specify that the user does an additional install.


I think a lot of node modules surprisingly have a build step


I don't think it's necessarily a "lot". I've been pleasantly surprised at how many do not

Alex Miller (Clojure team)21:08:04

I'm not saying this is good or useful, just an example that transitive resolution only has like 2 or 3 operations to implement