Fork me on GitHub

ok, so this is the full iceberg: I want to store a json recieved from the web in a nosql store, but I would like to store only the fields Im interested in. So I was planning to use jackson.

(defrecord Foo ^{JsonCreator true} [^{JsonProperty "field1"} field1])
(-> (ObjectMapper.)
    (.readValue json Foo))
I didnt fully investigate if clojure.spec can remove the unwanted field.

Joe Lane00:07:24

Why make a record?


use something like Cheshire or data.json to read in as data @paulocuneo


ok, that will solve the serialization issue, but how would you remove the unwanted field/nested-fields?


data.json takes a json string and gives back a hash map, then you can dissoc and encode again


I was thinking of select-keys actually, writing something like a parser But since the json is kinda big it would have been the same to describe the full structure.

(defn clear-fields[{:keys[field1 field2 ...manyfields]}]
   {:field1 (clear-sub-field1 field1)
    :field2 (clear-sub-field2 field2)

Joe Lane00:07:32

how deep of a tree is it?


clojure.walk ?


maybe 4 levels deep

Joe Lane00:07:29

Is the structure known and static?


yes, it's known and static


I just want to avoid storing garbage into nosql

Joe Lane00:07:32

So it's just an optimization?


it's a security issue too


It's better to whitelist, a priori it's unknwon what an attacker would add to the json, and it's not a recursive structure. I'll go with select-keys and update-in for now. I'll looking into schema an spec. Thanks @U051SS2EU @U050ECB92 @U0CJ19XAM @U0JEFEZH6 !!!

Joe Lane00:07:02

Would it be easier to whitelist things you want to keep or blacklist things you know you don't want?

Joe Lane00:07:48

If you use a whitelist you can reduce over the collection of paths and accumulate into a map

Joe Lane00:07:02

it would be like select keys but with deeply nested paths.


If the keys are predictable and are bad at all levels of nesting, it's easy to sanitize with clojure.walk


user=> (clojure.walk/postwalk (fn [x] (if (map? x) (dissoc x :a :c :e) x)) {:a {:b :f} :b {:c :g :d :l} :d {:d {:d :e}}})
{:b {:d :l}, :d {:d {:d :e}}}


on the other hand if the keys happen at predictable levels, update-in with dissoc might be more straightforward


I am trying out clj-async-profiler on a project, OS X with Oracle/Apple JDK 8, starting it with clojure command. I am trying with the option -Djol.tryWithSudo=true mentioned in clj-async-profiler's README, which causes a password prompt to appear in the terminal window where I started clojure when clj-async-profiler triggers some call that requires superuser privileges. I try to enter my password, but it seems that the clojure REPL is fighting the password prompt for the input characters, and clojure wins. It interprets my password as an unknown Clojure symbol. Any advice for improving on this?


I have tried running the clojure command itself using sudo clojure from the terminal, but then when it gets to the point where it would have prompted, instead I see "Killed: 9" and the clojure process dies.


When I do the clojure command without sudo in an Ubuntu Linux VM, it also prompts for a password, and the first time I type it and press return clojure gets it and says no such symbol, but if I immediately type it again and press return, it works. Weird.


I have also discovered that just as sudo is often configured by default so that if you authenticate with your password once, then within a few minutes later sudo commands do not prompt for a password, on Ubuntu Linux if I do some other sudo command, then clojure within those few minutes, it does not even prompt for a password. Nice.


And on OS X, if I try that same thing, it gets "Killed: 9", the same as if I run sudo clojure


If I rubber duck this enough, I will find a solution I am content with, probably, except perhaps for OS X 🙂

duckie 4

Where's is this tryWithSudo option mentioned (couldn't find anything in or and why is that useful?


I use async profiler but the only thing I added to project.clj is :jvm-opts ["-Djdk.attach.allowAttachSelf=true"]


When I use the allowAttachSelf=true option with at least some combinations of OS and JVM versions (maybe all? I don't recall right now), it mentions the tryWithSudo option, but perhaps that is because I am running my Clojure program using a different library (Java Object Layout) that needs that permission to do things. So perhaps this is unrelated to clj-async-profiler.


how do I access instance fields set in constructor of a deftype?


e.g. in CLJS I can do:

(deftype Foo [bar])

(def foo (->Foo "baz"))

(.-bar foo) ;; => "baz"


@lilactown That works in clj too so I'm not sure what your question is?


I think the issue is that the field I want to access is marked as volatile-mutable


I can’t find much docs but stackoverflow leads me to believe it’s private

Alex Miller (Clojure team)05:07:27

Any mutable fields will be made private

Alex Miller (Clojure team)05:07:15

To access, your deftype needs to extend interfaces or protocols that can expose the data (safely)

Alex Miller (Clojure team)05:07:05

Mutable stuff is inherently unsafe so is not public by default


I find odd that lein cloverage is assigning 100% of test coverage to a file full of specs. But I have literally 0 tests in my project. How is that?

Alex Miller (Clojure team)06:07:40

specs are all macros. maybe just compiling them is "coverage" ?


Maybe, I will investigate this on a empty project later on


I just did some first (naive) tests comparing fressian and transit. It seems Transit is more compact than Fressian. Does this sound correct? I had expected the opposite


doesn't transit cache keys in maps ? could be it. I am quite sure fressian doesn't. On the otherhand fressian allows to define your own handlers for custom types, so I imagine you can optimize things further.


not just keys - any value anywhere in the input, if identical? to a previous value, is replaced by a backreference


so in cases where you have duplicate entries at different points in the same input, it can shrink things drastically


eg. (let [m {:a (range 10000)}] {:x m :y m m [m m m]}) has only one copy of m, and 5 back references, when encoded


the back references are typically two characters long in the output


yeah i guess the extensibility is the prime benefit then (maybe cpu performance, but i don't care too much)


you could also add snappy/lz4 or someting on top of fressian


fressian does some optimisations (varints & co), but nothing substantial


as far as I remember


i should experiment more and see what i can get out of it. for example I havent seen this feature working yet


i hoped a repeated object would be cached but that doesn't seem to be the case


you can end up with quite compact representation of "types" you can control


ex not repeating keys in records and whatnot, but that's not free (as in you need to specify what/how), same goes for caching


I quite like transit nowadays, was super sceptical at first


I was thinking of making my own dictionary lookup for repeated items (for arbitrary objects) and have some kind of reference that needs to be replaced on a read. I have this working with transit, which is already quite compact. But Transit doesn't allow custom objects (forcing me to have hacky things like strings with underscore prefixes), so i was hoping Fressian could do this, but the basic respresentation is much bigger


Maybe i need to try harder 🙂


Transit is super good, but i have a special use case for a very compact representation


if you can trade some ressources for storage size, just compress the content 😛


quite a waste, but I don't know your constraints


hmm yeah actually if it runs on the jvm it can work. Maybe i should try traditional compression techniques instead indeed


ok one mystery for me though, why is the bytesize of (data->transit-bytes [1]) smaller than (data->transit-bytes 1) (i'm using the code in the gist above). It's 2 vs 6 bytes


if I have a symbol/string representing a java method, is it possible to call that method on an object in a macro without using read-string or reflection?


This is what I've attempted (assumes a constructor with no parameters, and setpairs returns a sequence of vectors of [<setter-symbol> <value to set>]

(defmacro m->obj [m class]
  (let [setters (set-pairs m class)]
    `(let [obj# (new ~class)]
       (doseq [[field# v#] ~setters]
         (doto obj# ((memfn field#) v#))
         ;;(doto obj# (. field# v#))


I think that would work, as long as setter-symbol is actually a symbol (you can use the symbol function on a string, of course)


that memfn will reflect when called at runtime btw


it avoids reflection by type-hinting symbol holding an instance: (with-meta (gensym "instance") {:tag type-expr})


ah! thanks, I think that'll be perfect


type-expr would just basically be the class-name or int for primatives right?. Nevermind, got it!


Build tooling power in 2019: deps.edn vs boot? I stopped following the evolution of Clojure build tooling in 2017. At the time, boot was the best choice from a standpoint of power / configurability, and I was quite happy switching to it over lein. Since then, I see that deps.edn and the clojure command line tool are being pushed as the new path forward. I found as a summation of the current "state of the world" from Apr 2018. While trying to understand how tools.deps is used to support dev flow for a backend microservice, I found From observing the current state of tools.deps libraries, it seems that the recommendation for tools.deps.alpha is to use a Makefile and/or aliases to setup CLI executable "tasks" for your project (dev, uberjar, etc.) It seems that tools.deps doesn't provide a nice "Clojure build layer" similar to Boot, but leaves that up to other tools which may eventually wrap it. Unfortunately, those tools don't seem to exist, at least not at the power level of Boot. The best I can find is My conclusion -- Boot is still going to be a lot easier to work with for a nice hot reloading clojure backend service. Many of boot's benefits seems achievable with tools.deps, but the path forward is not apparent and will require a lot of "roll your own" approach. Beyond the improvements in classpath handling, the benefits offered by tools.deps don't seem to get you significantly further than Boot.


tl;dr: tools.deps is still evolving, use Boot if you want to get off the ground (relatively) fast with a good foundation for a microservice ^ My question to this community -- is this conclusion right, or is there a part of the picture I am missing here?

Alex Miller (Clojure team)19:07:12

I think that’s pretty right when you scope it to “hot reloading Clojure backend service”

Alex Miller (Clojure team)19:07:54

But I think there are a large number of projects that are satisfied without using Boot or lein at all


That's fair. I notice tools.deps has made huge leaps WRT raw CLI apps & graalVM


And I'm looking forward to using pure tools.deps someday, but my hunch now is that I'd get more heartache than help

Alex Miller (Clojure team)19:07:10

If you need it, feel free to ask in #tools-deps

👍 4

you will have to make the leap at some point though, there's no stopping this train


^ I also get that impression, and I do agree with the vision behind tools.deps 🙂


i see many projects use a dual approach, use lein and tools.deps together


i'm not familiar with boot though

Alex Miller (Clojure team)19:07:14

I use tools.deps and maven together too (which is pretty easy with -Spom)

Alex Miller (Clojure team)19:07:56

Although that’s mostly in contrib projects which have a long maven history


Haha, maven gets much further out of my realm of know-how. I never got the "proper Java background" much of the community comes from. I do notice & as integrated approaches, though I'm not sure what I'd use 'em for in practice


I see this as a safe transition path, use tools.deps for its own power and services, use lein/boot for what you need that doesn't exist yet in this ecosystem


like your hot reloading


Makes sense. I'm interested in picking up just to get off the ground, but I just noticed... they already made the leap from boot to deps.edn Even found as a SPA example with figwheel reloading.


I can probably run with this example & eventually pull in revolt for a Clojure build layer. Out of respect for the support & work going into tools.deps, I'll give this a shot and see if I can make it work. Thanks @carkh & @alexmiller for the input!


good luck !


@quest Happy to answer questions about workflow/tooling -- we started with lein at work back in 2011, switched completely to boot in late 2015, and switched completely to clj/`deps.edn` last year.


Thank you for the offer, and I may take you up on this later, but sparingly. Knowledgeable help is a precious resource 🙂


I've also switched most of my OSS projects over to deps.edn now (instead of lein or boot).


Still rocking that good old lein... what made you change @seancorfield?


This talks about why we switched from lein to boot

clj 4

Thx Sean. I didnt even know there was a lein or boot alternative


I need to use a macro to call getter methods with no arguments on an object. The following works when passed a string but cannot be passed something that needs to be evaluated.

(defmacro getter [nm obj]
  (let [getter-expr (symbol (str "." nm))]
    `(~getter-expr ~obj)))

(getter "getMyValue" myObject) ;;works

(let [s "getMyValue"] (getter s  myObject)) ;;doesn't work
Any tips that might help me get the second way working?


it can't be done without eval, as s is just the symbol s until runtime


with eval, you'll probably want a lambda that receives s as an argument, as eval doesn't see locals


gotcha, thank you!