hey I was thinking about a way to minimize the memory requirements of a nested clojure data structure where some substructures are identical, and I came up with this:

(partial clojure.walk/postwalk (memoize identity))

@bcbradley if you just use the same object as an arg to assoc, it won't be duplicated

depending on how the data was created, of course

my use case involves taking an .edn file from disk that is probably too self similar and large to fit in memory

oh, yeah, fun

it would be interesting to try the postwalk identity and then compare the object pointers via jdb maybe(?)

yeah i haven't tested it i was just wondering if anyone else had ideas

what's "fn*"? I cannot find the document of it~

it's an implementation detail of fn

fn is implemented as a macro, and uses the destructuring functions that clojure.core defines for macros

so means: I don't have to care about it?

fn* is implemented in java code

right, remembering that it's an fn that can't destructure is probably enough

could you please give an example? of the difference?

+user=> ((fn [[a]] a) [1])
1
+user=> ((fn* [[a]] a) [1])
CompilerException java.lang.IllegalArgumentException: fn params must be Symbols, compiling:(NO_SOURCE_PATH:2:2)

[a] as a parameter is a destructure that says "bind the first element of this sequencable input to the name a"

fn* doesn't understand that syntax

so the difference is only about the destructuring of the parameters? fn supports it, while fn* doesn't?

that's the main one, I forget if it's the only one

hmm~ in lazy-seq macro:

boot.user=> (source lazy-seq) (defmacro lazy-seq "Takes a body of expressions that returns an ISeq or nil, and yields a Seqable object that will invoke the body only the first time seq is called, and will cache the result and return it on all subsequent seq calls. See also - realized?" {:added "1.0"} [& body] (list 'new 'clojure.lang.LazySeq (list* '^{:once true} fn* [] body)))

boot.user=> (source lazy-seq)
(defmacro lazy-seq
  "Takes a body of expressions that returns an ISeq or nil, and yields
  a Seqable object that will invoke the body only the first time seq
  is called, and will cache the result and return it on all subsequent
  seq calls. See also - realized?"
  {:added "1.0"}
  [& body]
  (list 'new 'clojure.lang.LazySeq (list* '^{:once true} fn* [] body)))

why is fn* preferred here~? is it some performance concern, as fn* is the basic one?

right, destructuring is defined in terms of lazy-seq

so lazy-seq can't use fn, which destructures

source of fn shows you how it builds the form for fn* - it does many things, some of which are lazy seq generating

I will try to read the source of fn I probably digged too deep of the dark magics, as I just started learning Clojure

another difference, discovered by reading the source of fn, is that fn* can't do preconditions - or at least not reasonably

+user=> ((fn [a] {:pre [(number? a)]} a) :a)
AssertionError Assert failed: (number? a)  user/eval36/fn--37 (NO_SOURCE_FILE:7)
+user=> ((fn* [a] {:pre [(number? a)]} a) :a)
:a

thanks for the explanation, I probably need some more time to digest it~

yeah- all you need to remember is fn is fancier, does some things fn* can't, but code that exists before fn is defined (code that fn uses for example...) has to use fn*

got it thx!

So I’m parsing XML. That’s enough for wanting to shoot myself in the face, but on top of it I need to check that it conforms to a certain shape, because the source(s) can be unreliable.

I’m using clojure.xml/parse for this.

Now, spec seems like the right tool for conforming. However, I’ve also heard tales of spec being able to destructure as well.

Where can I read about how the destructuring works, specifically? If I can avoid hand-disassembling the stain upon humanity that is the output of parse, I’d be delighted.

I just discovered zippers

Anyone used clojure transducers for unfolds (anamorphisms)? All talk I’ve seen is on folds (reducers / catamorphisms).

So I'm working on getting a plugin system working in clojure. Based on the loading system on @yogthos' (https://yogthos.net/posts/2015-01-15-A-Plugin-System-in-Clojure.html). I can get the file on the classpath, but when I try and run the function defined by the .edn file packaged with the plugin, it gives me the error: Could not locate 'oss_world_example/core__init.class or 'oss_world_example/core.clj on classpath.

I checked in the jar and oss_world_example/core__init.class is definetely there (you can download the example jar I'm using here: https://filebin.ca/3VLfGd71KesW/project.jar). What gives? How is it finding the .edn file but not the classes?

@donyorm what does the code in those files look like?

@bfabry oss-world-example.core looks like this:

@donyorm You might have an extra quote character somewhere?

fwiw, I can load that ns from that jar fine

11326-storage:look bfabry$ java -cp ~/.m2/repository/org/clojure/clojure/1.8.0/clojure-1.8.0.jar:project.jar clojure.main
Clojure 1.8.0
user=> ( "oss_world_example/core__init.class")
#object[java.net.URL 0x2ca26d77 "jar:file:/private/tmp/look/project.jar!/oss_world_example/core__init.class"]
user=> (require '[oss-world-example.core])
nil
user=> oss-world-example.core/entry
#object[oss_world_example.core$entry 0x24c22fe "oss_world_example.core$entry@24c22fe"]
user=>

user=> (oss-world-example.core/entry)
Entered the example plugin.
nil

Perhaps in the edn file.

so my bet is you don't actually have the jar on the cp

the edn file is using underscores for the namespace, which is incorrect

I'd have to see your loader code to know whether you compensate for that

but, while that's a problem, it's not a problem that should yield the error you got. the error you got very specifically says the file does not exist in the cp

(it could be finding the edn file because it's accessible by some other cp entry like .)

I think the names in could not locate error message should not start with a quote, so check for single quote in the edn file such as:

{:description "Markdown parser"
 :init 'cryogen-markdown.core/init}

^ that's it

I missed the quote

you don't need to quote inside an edn file

(edn/read-string "{:foo bar}")
=> {:foo bar}
(class (first (vals *1)))
=> clojure.lang.Symbol

and in fact it's not part of edn I don't think (' etc are part of the clojure reader)

yeah

(edn/read-string "{:foo 'bar}")
=> {:foo 'bar}

It's surprising in my opinion actually. Maybe it's a bug in the lib, not wrapping with (name symbol) where it should.

Nope, I'm wrong

it's surprising but I dunno about bug... the language reader and the data format reader can't be exactly the same

That's probably it. I'll check when I get back to my computer.

Yep removing the quote fixed it. Thanks for your help!

So I have some code in my -main function (extend-classpath (filter #(.contains (.getName %) ".jar") (file-seq plugin-dir))). If I run this directly in the repl, it changes my classpath as I want it to. If I run the -main function in the repl, it doesn't work. What causes this difference (boot repl if it matters)?

@donyorm regarding extend-class in repl vs -main, I’m wondering if it could be related to boot using pods to isolate classpath (https://github.com/boot-clj/boot/wiki/Boot-for-Leiningen-Users#repl-dependencies). Just getting started with boot myself. Would appreciate an authoritative answer too.

There seems to be an issue with 1.9.0-alpha17 in let blocks with auto-gensymed variables in macros. Is that a known issue?

@mrg could you elaborate?

@moxaj I'm trying to find the example that exploded yesterday 🙂

@moxaj looking again with fresh eyes, I found the bug. I had this:

that fails with something completely unreadable:

trying it in 1.8 I am getting this instead:

that error message seems crystal clear to me /s

so if I do the right thing and use msg# instead of msg it works

So yeah. Not a bug, just a mistake on my part, coupled with an awful error message 😕

what is the general approach to "expanding iteration"?

(path-to-paths
  [:a]
  {:a #{:b :c}
   :b #{:d :e}})
  
;=> [[:a :c]
;    [:a :b :d]
;    [:a :b :e]]

@moxaj i just added this: https://github.com/bhb/expound/blob/master/README.md

and got this:

that's pretty neat...

that's indeed pretty cool!

@mrg https://clojurians.slack.com/archives/C1B1BB2Q3/p1501600794016449

Thanks @misha

is there any recommended way of fixing

java.lang.ClassNotFoundException: clojure.tools.logging.impl.LoggerFactory

@sjagoe you can consider asking in the #luminus channel. I'll see what I can find

ok, will do!

@sjol I seem to remember a similar issue in one of my projects, and the solution was to AOT compile the logging implementation by adding this to my lein project.clj:

:aot [clojure.tools.logging.impl]

If somebody is ever interested in using ULID:s I ported library for Clojure https://github.com/theikkila/clj-ulid

How can I check if something is not equal to one of a few things.

Like (not (= % 1) (= % 2))

(not (#{a few things} thing))

Ah, sets, I knew there would be a simple thing I wasn't thinking of.

Thanks @bfabry

np

you can use contains? with the set if you need to check for nil / false

(along with other non-nill non-false values)

:thumbsup:

is there an elegant way to convert a vector of maps (where each map has the same nested key structure) to a map of vectors? i.e. if i had [{:A {:B 1} :C 2} {:A {:B 3} :C 4}] I would like to get {:A {:B [1 3]} :C [2 4]}

@michaellindon maybe merge-with?

@cjsauer thats a good suggestion, let me check it out

recursive merge-with

I actually think I've written that before

@cjsauer thank you, I will try that, i thought that ':aot :all' would have taken care of that?

@sjol where did you place it? If you put it in your uberjar profile, then it will only take place when you run lein uberjar

@bfabry if you could find it then that would be great!

it's not quite what you want, but it should give you an idea. also it looks like it was written kinda stupidly

(defn merge-metrics
  "Merges a collection of maps together assuming they all have the same layout by recursing
  through any child maps until a non-map is found.
  If the values are a collections, they are appended together.
  Anything other type of value is added."
  [first & rest]
  (if (map? first)
    (apply merge-with (into [merge-metrics first] rest))
    (if (coll? first)
      (into first (flatten rest))
      (+ first (apply + rest)))))

particularly the recursive call could just be (apply merge-with merge-metrics first rest)

thanks for sharing, this looks like a good start for me

@cjsauer yes ':aot :all' was already part of the project.clj file in the uberjar section, when I removed it last night lein uberjar would not work

@michaellindon slightly smaller:

(defn merge*
  [res lat]
  (if (map? res)
    (merge-with merge* res lat)
    (into [] [res lat])))
;; now use it
(apply merge* my-maps)

@sjol you could try placing the :aot [clojure.tools.logging.impl] at the top-level of your project.clj file; not down in a profile. That way it applies to all lein tasks.

@cjsauer ah this is excallent, what a timesaver!

> my-maps
[{:A 1, :B [1 2], :D {:F 10}} {:A 2, :B [3 4], :D {:F 20}}]
> (apply merge* my-maps)
{:A [1 2], :B [[1 2] [3 4]], :D {:F [10 20]}}

thanks, i love lisp

@cjsauer will i need to provide a :main as my entry point?

Question for everyone.... if you want to dynamically define a set of ns level functions, what's the right way? I've always been of the opinion that a macro is right, but a co-worker argues that calling a no-arg function in the ns that calls defn is simpler to understand (which may be true). I've always understood the latter idea to be bad, but haven't seen a really solid explanations as to why. Can anyone enlighten me on this? Thanks!

As an example: Suppose you have a few dbs with names like "foo" and "bar". You might want to dynamically create functions like find-foo, find-bar, write-foo, write-bar rather than use the underlying (db/find "foo" ....) library code every time.

with a macro you can ensure that the name being defined is right there in the call

with the function, it would be a convention at best, right?

also, it’s better to simply not create things like db connections at the global level

well, suppose the first line of the ns were (defn foo[] (do (defn find-foo.....))) then (foo) right after. The function would exist before you used it.

I'm not advocating this, just trying to see why it's much worse than (defstuff "foo") where defstuff is the macro that creates the functions relevant to foo.

that leads to misbehaving code - clojure doesn’t have a compile-only mode, you can’t syntax check or create a jar from that code without creating a db connection

just FYI

(well you can but it’s complex)

Excellent points.

this is why libraries like stuartsierra/component or integrant exist

anyway, def-* is better than defn inside defn

The db example is just one. the main question is: In any case where you'd want to dynamically generate some functions, what are the reasons why a macro is better than blind function call.

Unless the answer is "never do that"

but it seems like something that would occasionally be useful

if you are dynamically generating functions, they shouldn’t have to exist at namespace level - otherwise you end up needing to override some of the few protections clojure provides at compile time to create code that uses them

so, instead letfn or something along those lines as needed?

(let [foo-fn (resolve 'foo)] (comment "foo eventually exists") (foo-fn))

that’s what you end up with

where can i read the source of how keywords act as functions? ie, (:a {:a 1}) Looking at keyword in source, its just interned. But I thought i keywords being proper functions and not just being treated like a function call

@dpsutton all things in call position are treated like a function call - this is where it is defined https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Keyword.java#L137

they are “proper functions” in that they implement IFn

yeah. where is the implementation of IFn

@dpsutton that'

s what I linked to

@cjsauer , there is one minor issue, the vectors become nested when applied to a vector of maps greater than 2

l
> my-maps
[{:A 15, :B [14 24], :D {:F 40}} {:A 1, :B [1 2], :D {:F 10}} {:A 2, :B [3 4], :D {:F 20}}]
> (apply merge-with merge* my-maps)
{:A [[15 1] 2], :B [[[14 24] [1 2]] [3 4]], :D {:F [[40 10] 20]}}

ah ok. just wondering why ("bob" {"bob" "value"}) wouldn't work. seems consistent with "all things in call position are treated like a function call"

so keywords implement ILookup?

@dpsutton because string doesn’t implement IFn so you get a “String isn’t an IFn” error

it’s the source I linked to

they implement invoke, as all IFn must

oh man. i missed the filename i was in

totally my bad

i see now

yeah i'm with you

it’s right here in the “weird error message” - very clear if you know what you are looking for

=> ("hi" {"hi" 1})
ClassCastException java.lang.String cannot be cast to clojure.lang.IFn  peregrine.circle/eval52026 (form-init7640008060394334379.clj:1)

:foo can be cast to IFn, so it works

thanks a bunch

is there a function s.t. (foo [1 2] [3 4]) gives [[1 2] [3 4]] but also (foo [[1 2] [3 4]] [5 6]) gives [[1 2] [3 4] [5 6]]

@michaellindon you can write one if you can articulate what makes the first scenario different than the second

@ghadi I'm not sure how to differentiate between a vector and a vector of vectors

how about (vector? (first coll)) ?

thats a good point

i think i can work with that

thanks

np

it's just a conj away after that

sorry, its been a long day..

XD

what is the good lib for working with amazon aws (whatever that might mean)? asking for a friend™

depends on what you're doing with it

some people like Amazonica (https://github.com/mcohen01/amazonica)

no idea. "asking for a friend" was not really a joke.

are there any noticeable alternatives? or it is pretty much it? @bja

nothing as comprehensive. there are specific wrappers to S3, SQS, SNS, etc but nothing else (that I'm aware of) besides Amazonica that wraps the entire SDK. I personally tend to use specific libraries for S3/SNS/SQS and then use direct interop in the rare case I need something else. Interop with the Java SDK is pleasant enough for a java library.

understood, thank you.

Amazonica goes a decent way towards a "native" feel for using AWS via Clojure. If you're just looking for access to the APIs (particularly if you have experience with them), importing the Java SDK directly is an option. The rationale behind not using Amazonica would be you want to avoid its extensive reflection usage (although how much that actually matters as a remote api wrapper is questionable)

or because you decided to look for the source to that one function you use and discovered the whole thing is a bed of lies

the structure of the project is pretty startling though - all the definitions are created by macros that reflect on the objects in the aws jar to decide what to create

it’s only a concession to sanity that they give you namespace files - they could have faked that up too

… or maybe not, it would make it hard for require to work as expected now that I think of it

that's part of the extensive reflection that I wasn't happy with

my original rationale for not using it was that it didn't let me directly select which client object I was using, but I found out that I just didn't understand what it was doing

by that point, I had already built on top of clj-aws-s3 and bandalore though

Not related to AWS libs, but the recent release of clojure.tools.deps directly interops with Maven Resolver's java API. It's a really nice reminder of how clean and easy interop is.

back in the early days there were a million redis wrapper libraries, but these days I just use Jedis directly.

@ghadi still no videos from euro-clojure?

i'm sure they'll be coming soon!

in the meantime the Java Language Summit is releasing videos, always interesting

pass

Unfortunately no Clojure representation at the summit this year

though Brian Goetz had some very flattering things to say about clojure.spec to a large Java audience (at some other conference)