Fork me on GitHub

I am attempting to analyze some data that I got from an API. The problem is, when I talked to the API, what I got back are byte arrays. Each byte array “is” a .zip file which has the artifacts for the job in question. If I were analyzing this manually, I would click a button for a job, and my browser would put a .zip file into my ~/Downloads directory. Then I’d move it somewhere convenient, then unzip it to get a folder. Inside the folder, I’d find the files that are going to help shed light on my research question, and I would write Clojure functions to load them, parse them into JSON, and do my analysis. But I have a small # of hundreds of jobs to look at the artifacts for. So what I’ve got now is, an edn file (I chucked all the results of my calls into a file to stop hammering the server with questions), with a sequence of maps that look roughly like {:id 1337 :artifacts [80 75 3 4 20 0 8 8 8 0 110 6 37 86 0 … and I’m not sure, what’s the most straightforward way to turn that into the data I care about?


(for [m ...] (assoc m :artifacts (into-array Byte/TYPE (:artifacts m))))


then do whatever you want with the byte-arrays


wrap them in bytearrayinputstreams and wrap those in a zipinputstream and read the contents


i sort of get streams abstractly


this will teach me a lot


if you aren't familiar with* that might prove to be a good thing because ZipInputStream can be a little different to work with then most kinds of inputstreams


about concrete streams


zipinputstream is actually maybe the old way to work with zipfiles, there is a newer api that gives you more of a filesystem like abstraction


when working with inputstreams, do those need to be paired with a reader? i have mostly only used slurp/spit


but i got help writing the edn i mentioned, then today i did successfully read it


input/outputstreams are for binary data


reader and writers are for character data


my big accomplishment for the day was getting, and caching as edn, my data — tomorrow I will get into streams, see how far I can get, then probably come running right back to clojure slack . Thank you for the signpost!


> there is a newer api that gives you more of a filesystem like abstraction @U0NCTKEV8 do you recall the name of the api?

Søren Sjørup06:03:12

When (read-string "'()") becomes (quote ()) . Why does (read-string "()")` become (clojure.core/list) and not (syntax-quote ())?

Søren Sjørup06:03:36

Because the reader does quote expansion in Clojure seems to be the answer.


I'm little confuse with 'headings forms'. In e.g: 1. (fn name? [params*] exprs*) 2. (defn name doc-string? [params*] exprs*) 3. (def symbol doc-string? init?) Why at 3. heading use 'symbol'? And not 'name' as at 1. & 2. ? As I understand 'symbol' will be created after we initialize it with 'def'. But first we have to pass a 'name'. So, for me it should be (def name doc-string? init?) not (def symbol doc-string? init?). What do you think?


probably because def

Creates and interns or locates a global var with the name of symbol and a
namespace of the value of the current namespace (*ns*).
main part is "the name of symbol"


It's kind of confusing. As I understand it, you are passing a symbol, but it gets also treated as a name that you can look up. See:

(defn foo [x] x)
(-> #'foo meta :name) ; foo
(def bar 1)
(-> #'bar meta :name) ; bar

👍 1

also def is a special form and defn is macro


(macroexpand '(def foo 42))
;; => (def foo 42)

(macroexpand '(defn bar []))
;; => (def bar (clojure.core/fn ([])))


> As I understand 'symbol' will be created after we initialize it with 'def'. But first we have to pass a 'name'. > So, for me it should be (def name doc-string? init?) not (def symbol doc-string? init?). > What do you think?


A symbol stands on its own. Yes, it's often a name for a var. You can't evaluate it until you assigned it to a var, except you quote it like 'foo Symbol and name are kind of interchangeable concepts. But be aware that a symbol is actually a thing and not just a label that is assigned to something else. You can for example use symbols in macros that get treated at a first class thing and assigned to a var or multiples later on. The confusing thing is rather why they used both name and symbol in the docs. I assume it is because we think in terms of "function names" both for fn and defn.


So it is more of a high level cultural expression there (function name).


symbol and name are not interchangeable concepts. def expect symbol and will use its name in combination with the value of *ns* to create and intern global var.


That's why I said "kind of". What I'm personally not sure about is what the "name" in fn is. It definitely doesn't assign a symbol to a var in your namespace. Maybe it's just for debugging purposes? But it's very different from def and defn


"also def is a special form and defn is macro" so what fnis also special form and use 'name'


"If a name symbol is provided, it is bound within the function definition to the function object itself, allowing for self-calling, even in anonymous functions."


> symbol and name are not interchangeable concepts. def expect symbol and will use its name in combination with the value of *ns* to create and intern global var. Symbols are definitely first class names though - just to be sure! def takes the symbol and assignes a name (the symbol) for it in your namespace. The name in fn is something entirely different it seems. Not sure what it does.


for fn name is bound to that function only and available only within its definition.


also symbol is just symbol. depending on the context it will be used differently: to intern a global var, define local binding, etc.

Sam Ritchie15:03:47

Huh I didn’t realize def supported a docstring!

👀 1
Sam Ritchie15:03:42

Has that always been the case?


> Support for doc-string was added in Clojure 1.3.

👀 2
Sam Ritchie16:03:13 haha I had 100s of docstrings attached via :doc… backed them all out here, this looks way better!!

👍 1
😂 2

I didn't know that either! Also that's a huuuge commit :)

Sam Ritchie17:03:02

I’m a docstring addict

👏 1

I lit a bit confuse when I use function set and how this function shaking element in sets

(set {:a 1 :s 2 :y 3}) => #{[:y 3] [:a 1] [:s 2]}

(set {:a 1 :s 2 :y 3 :o 8 :p 6}) => #{[:y 3] [:a 1] [:p 6] [:s 2] [:o 8]}
Why this function change position of element? Where to read about this behavior? Thanks


because in set and hashed-map order of elements are not same


I made a simple comparison table, maybe it will be useful

🙌 1

Good example, thank you!


"This is because of the internal structure of the HashSet. The value is transformed in a unique hash, which allows fast access but does not keep the insertion order. If you care about the order in which the elements are added, you need to use a different data structure," from book The Clojure Workshop: Use functional programming to build data-centric applications with Clojure and ClojureScript

👍 1

What the key of elements in set. In vector key is a index (position) from 0 In map - keyword, string in set? hash of element value?


sets and maps have no order, indexing in them doesn't apply


the only thing that is guaranteed is for maps, iteration order is consistent between (seq m), (keys m), (vals m), where m is a map


> What the key of elements in set. > sets and maps have no order, indexing in them doesn't apply So @U050ECB92 is being accurate, but slightly pedantic. A better term for what I think you mean by this question, that I hope @U050ECB92 will not also object to?, is "lookup value". For vectors, you can look up an element by index. For maps you can look up an element by key. For sets you can look up an element by itself, which is a constant time operation.

(:foo #{:foo :bar})
; =>

(#{:foo :bar} :foo)
; =>
Like maps, sets have no specified order. And any circumstantial appearance of any consistency in ordering is purely implementation detail, subject to change at any time, and should not be relied upon.


> Sets support...get, the latter returning the object that is held in the set which compares equal to the key, if found:

(def s #{:a :b :c :d})
(get s :a)
-> :a


(yeah I'm being pedantic: lookup is for finding by arbitrary keys, indexing is finding by an integer in a sequential collection)


I think that distinction is fair if you consider indexing to be a specific less-arbitrary subset of lookup in general. Either way, lookup by self in a set is every bit as arbitrary as lookup by key in a map.

🙌 1

Awesome! It's what I want to knew. What the index/key of set. Before, I don't understand what the different between hash-map and sets as structure data. Can you show me example where you use set or hash-map in real code? For what kind of data using set ?


A few pretty good succinct examples here. Quite common to use sets as functions for existence lookups rather than something like data storage.


The usage of hash-maps, sets, lists, vectors, etc. isn't really Clojure-specific as much as it is general computer science knowledge. Barring a few tricks, the way these data structures are applied will be much the same in any programming language. Just clarifying since it might be good to look up a more general source of information too.


Two common uses for maps are domain objects and lookup tables. Typical examples in real code: • assoc used to create an "object" map with certain properties, to be passed around: fn in • a nested lookup table

Jakub Šťastný15:03:52

How can I run Clojure so it passes command line arguments as they are to a given fn? From deps.edn:

From src/jakub_stastny/et/runner.clj:
(defn custom-runner [args]
  (println "custom runner") (prn args))
If I call clj -X:run a b c it passes a b c into args, but then if I give a file path it does something weird, clearly Clojure is doing some processing on these arguments. How can I get command-line arguments as an argument to a function I'm running? I know about *command-line-args* but I'd prefer an argument to a main-like function rather than a global variable.

Jakub Šťastný15:03:01

I wasn't able to figure out whether defining -main fn is still a thing in Clojure (nor is it particularly clear to me how to do it).

Jakub Šťastný15:03:34

As much as Clojure CLI is a nice system, it's bloody complex. I found Babashka a great first contact with Clojure since running it is much simpler TBH. This is my first time trying to set up a project using deps.edn and my head is spinning like mad.

Alex Miller (Clojure team)15:03:52

-main is still a thing and if you do that, you can call it from the cli with clj -M -m my.ns

Alex Miller (Clojure team)15:03:08

those args are passed as strings to the function and then you can do whatever you want with it

Alex Miller (Clojure team)15:03:48

-X / -T expect to invoke a function that takes a map, and thus take k v pairs, and will edn/read each of the arguments

Jakub Šťastný15:03:27

I see. So -M -m is really what I want then.

Jakub Šťastný15:03:40

I was confused by both -M and -m, I thought it's one or the other.

Alex Miller (Clojure team)15:03:25

-M is clojure.main, which takes an argument -m to pass the main class

🙏 1

And just to round this out: with -M you can also use -e, -i, and -r which are also options understood by clojure.main


Interesting. I used this as a challenge to write a deep-merge (fn my-deep-merge' below). Then I saw the splendid deep-merge in @delaguardo’s link (enc as deep-merge below). Now I am trying to work out how I could possibly produce that code in deep-merge ; clearly I have not got the where-withal! How do you visualise the recursion here to see the deconstruction in the fn params - I get <Basis> <induction> type recurses but cant see how you'd design this?

Jakub Šťastný17:03:08

I found myself doing various versions of (conj (butlast acc) (last acc)) when doing reduce: Adding a key to a map:

(let [acc [{:paths []}]] 
  (conj (butlast acc) (assoc-in (last acc) [:name] "name")))
; => ({:paths [], :name "name"})
Pushing to an array under a key:
(let [acc [{:paths []}]]
  (conj (butlast acc) (update-in (last acc) [:paths] #(conj % "path"))))
; => ({:paths ["path"]})
Since I've been doing it over and over, I wondered whether there might be a shorter version of this? I was wondering whether it could be handled by assoc-in/`update-in` only, but I haven't find a way to say "update on last index". Any suggestions appreciated 🙏:skin-tone-3:


you are treating the vector like a stack. And you can use peek and pop for quick access


or just use a list like a stack with first and rest


last is 0(n) in the size of the input; it creates a seq from the input and walks the seq to the end


butlast is also converting your vector to a seq, so conj is adding to the front


so butlast is dropping the last item in the seq, and then conj is adding at the front

Alex Miller (Clojure team)18:03:40

If you need a stack, a list is the fastest option

Jakub Šťastný18:03:47

Right...I see what you're saying. Ruby/JS has only one type of arrays, so I'm quite confused with all these 🙈

Jakub Šťastný18:03:40

OK I thought I got what you guys were saying, but maybe interpretation was: "use list and it all somehow work" and I realised that if list adds to the first of it, then I could just do assoc-in with index 0 which will be the last item (as in the item I added last, but actually the first one "physically"). However:

user=> (assoc-in [{:name "test"}] [0 :name] "a")
[{:name "a"}]
user=> (assoc-in '({:name "test"}) [0 :name] "a")
Execution error (ClassCastException) at user/eval3 (REPL:1).
So clearly my interpretation is incorrect. I still understand that list is the way to go, as @U064X3EF3 says, it's the fastest way, but then...? I just want some elegant way with assoc-in/`updated-in` so I don't spend ages writing this repetitive code.


lists are not associative

Alex Miller (Clojure team)18:03:00

You just need cons (push), first (peek), and rest (pop)

Jakub Šťastný18:03:02

No I get that, they behave like vectors...but these two functions works on vectors, so why not on lists?


you have what look like logic errors in your code (using last to get the last element, but then conj'ing it back on to the front) that you may want to address before getting concerned about repetitive code


it is possible you might want something like (circular ring buffer type thing), but in that case you still don't want to be using last

Jakub Šťastný18:03:08

@U0NCTKEV8 I'm writing a prototype of something. I'm more worried about how much code I need to write than about performance. I get your point though, but my priority is elsewhere. I do want to learn Clj properly and write proper code, but as of now "it works" is more important than anything else.


I am not just saying performance, I am saying if the intent of the code is to treat accum like a stack, then it isn't doing that correctly


it is "popping" from the end of the list, and adding the beginning, more like a queue

Jakub Šťastný18:03:56

Anyway the right code is then:

(prn (reduce
       (fn [acc i] (cons {:name i} acc))
       ["a" "b" "c"]))
That and first and rest when applies.

Jakub Šťastný18:03:12

But essentially the code will be doing exactly the same as my original code in terms of code structure. It might be more correct and fast, but no chance for less code then?


you can destructure acc into the first and the rest

Jakub Šťastný18:03:20

OK that's a fair start (let [[f & r] '(1 2 3)] [f r]) ; [1 (2 3)].


you can even abstract the whole thing out (fn [f] (fn [a b] (cons (f (first a) b) (rest a))))


use it as a transducer with transduce


presumably at some point you want to push a new element onto the stack instead of updating the top

Jakub Šťastný19:03:54

@U0NCTKEV8 Would you have an example of how to use it as a transducer? That one's a bit out of my CLJ league still 🙈 But it's something I wanted to learn, but I'm still not getting it.


(defn stack [g]
  (fn [rf]
     (fn [accum item]
       (let [accum (g accum)]
         (cons (rf (first accum) item) (rest accum)))))))

 (stack (fn [stack]
          (if (> (count (:path (first stack))) 2)
            (cons {:path []} stack)
  (fn [a b]
    (update-in a [:path] conj b)))
 '({:path []})
 (range 10))


output of the above is

({:path [9]} {:path [6 7 8]} {:path [3 4 5]} {:path [0 1 2]})

Jakub Šťastný19:03:36

Thank @U0NCTKEV8, I really appreciate your help.

Jakub Šťastný19:03:04

Going to read up on transducers now, because this is over my head still 🙈

Jakub Šťastný19:03:16

I see there's a Rich's video on them, great, I love these.


Ignoring all the above discussion about stacks and whatnot, and focusing on your original question and isolated code samples: It's not clear from the docs, but you can use update on a vector, with an index as the "key", as shown in the This does not save you any characters, but I think it is a bit more declarative representation of what your code is doing, and feels less awkward.

(defn last-idx [v] (-> v count dec))

(let [acc [{:paths []}]]
  (update acc (last-idx acc)
          #(assoc % :name "name")))
; =>
({:paths [], :name "name"})

(let [acc [{:paths []}]]
  (update acc (last-idx acc)
          #(assoc % :paths "path")))
; =>
({:paths ["path"]})


Is there any way I can reprovide a macro in clojurescript? Like, if file internal.clj has a macro, but I want it to be brought in whenever frontend.clj is required, is there anyway I can do that.


If it was a value I could just do something like:

(ns frontent
  (:require [backend])

(def a-value backend/a-value)
But for a macro...well that obviously doesn't work...


I suppose I could eta expand it to get:

(defmacro a-macro [& args] `(backend/a-macro [email protected]))
But that me.


"Is there any way I can reprovide a macro in clojurescript?" Scary. I spent a couple hours doing this today. "Trying", I should say. gave up.

"(def a-value backend/a-value)"

But for a macro...well that obviously doesn't work...
Did you try it? I made the same assumption and did not even try either, but one never knows.
(defmacro a-macro [& args] `(backend/a-macro [email protected]))
That's what I did, as well as just copy the whole damn macro definition. I seemed to making headway, but I do an anaphoric variable injection of sth akin to JS "this" and it works great, but it got lost when I did the reprovisioning -- CLJS now complained the anaphor was undefined. btw, I was just trying to reduce the "requires" in a client, so I did not care how gross was my solution. And my API is small, so there would not be a lot of it. Hoping a better answer comes along!


> Did you try it? I made the same assumption and did not even try either, but one never knows. Ya...I did, works about as well as you'd expect. In racket its easy enough to do with make-rename-transformer, but alas, does not seem to be a thing here.


> I seemed to making headway, but I do an anaphoric variable injection of sth akin to JS "this" and it works great, but it got lost when I did the reprovisioning -- CLJS now complained the anaphor was undefined. I have a hunch that its due to the auto-namespace injection that clojure's quasiquote inserts. Care to share your current headway?


Let me put it back in and get a fresh fail. The headway may have been that lein test passes, but that just tests Clojure. (I have a big CLJC project, never got round to automated testing for CLJS.) brb


(defmacro c-fn-var-ex [[c] & body]
  `(fn [~c]
     (let [~'me (c-model ~c) ;; <================ me
           ~'_cell ~c
           ~'_slot-name (c-slot ~c)
           ~'_cache (c-value ~c)]
       [email protected])))

(defmacro c-fn [& body]
  `(c-fn-var (~'slot-c#) [email protected]))

(defmacro cF [& body]
     :code '~body
     :value tiltontec.cell.base/unbound
     :rule (tiltontec.cell.core/c-fn [email protected])))

[Figwheel:WARNING] Compile Warning   src/web_mx_quickstart/core.cljs   line:64  column:45

  Use of undeclared Var web-mx-quickstart.core/me

  59                            {:default     :ignore
  60                             :on-navigate (fn [route params query]
  61                                            (when-let [mtx @md/matrix]
  62                                              ;; todo mset! me
  63                                              (mset! mtx :route route)))}))
  64       :selected-lesson (cF (let [route (mget me :route)]
  65                              (some (fn [lesson]
  66                                      (when (= route (:route lesson))
  67                                        lesson)) lessons)))
  68       :keydowner       (cF+ [:watch (fn [_ me new _ _]
  69                                       (.addEventListener js/document "keydown" new))]
It then fails at run time because mget receives nil in that position. Let me try modifying one of the CLJ tests to use the repro NS. brb...


OMG. I thought :refer-macros was no longer needed. The CLJS works now with:

     :refer-macros [cF]
     :refer [mpar mget mset! mswap! mset! fasc fmu] :as mx]
Did I misunderstand about :refer-macros being vestigial?


IIRC, you need to use :include-macros true now.


So it would be


[tiltontec.matrix.api :refer [mpar mget ...elided... cF] :include-macros true]]

👀 1

OK, :include-macros does the trick (thx!). Trying again with:

(defmacro cF [& body]
  `(tiltontec.cell.core/cF [email protected]))
...instead of reiterating the whole macro, but my thinking was that this code is crazy mature (so not changing much) and then when the user does a find definition, their first hop won't be completely useless. As for the original question, I have kicked off a couple threads recently on how to spare users (and me!) huge requires to use my library, given my preference for small files. No good answer was found, so I am leaving them where they are and doing a separate tiltontec.matrix.api namespace where I repro. Looking forward to more input on this thread, which has already salvaged my morning's work!


Glad to hear it.

Sam Ritchie20:03:25

You can still avoid include-macros!!


Oh cool, how?

Sam Ritchie20:03:01

On my phone, searching for an example, one sec …

Sam Ritchie20:03:23

If you add a :require-macros call to the cljs side of the namespace where the macro is written, no one else requiring that namespace has to do anything beyond :refer


Oh, sure.


But if you've already been given a clojurescript file which you don't want to change, it seems like include-macros is still required, ya?

Sam Ritchie20:03:27

Ah yes sorry I misunderstood. You are right


Rats, oh well, thanks.

Sam Ritchie20:03:05

Or push a PR upstream, that’s what I did here since i wanted all of those pesky include-macros calls gone from my library


One day perhaps clojurescript will never require include-macros, even for libraries that haven't added require-macros. Until then, thanks anyway. 🙂

❤️ 1

Although I guess while I have your attention, is there any way (at runtime) to get a list of all the macros a namespace provides?


It seems like Clojure can do it.


But in clojurescript, the only thing I've found that does it is cljs.repl/dir.


Which would be fine, except it prints it to stdout.

Sam Ritchie20:03:47

Yeah I think all of the var info / metadata is gone at runtime


Errr...I should say in bootstrapped cljs.


In which case its not because...ya. 🙂


oddly enough though, it seems like cljs.repl/dir uses ns-publics, so not sure why usig ns-publics directly doesn't work.

Sam Ritchie20:03:07

You can call ns-publics in a macro, since that will run at compile time

Sam Ritchie20:03:08

But the macroexpansion won’t have a runtime reference to ns-publics left over


Ah, so your saying that dir works because its a defmacro, but if it was defined with defn, then it wouldn't work?

Sam Ritchie20:03:25

That’s right


Which I guess would explain why ClojureScript's implementation of dir doesn't have a similar dir-fn.


Hmm...strange, looks like even when I use it in a macro, I still am not getting the macros:


cljs.user=> (defmacro foo [arg] (ns-publics 'shadow.cljs.modern))
cljs.user=> (foo)
cljs.user=> (clojure.repl/dir shadow.cljs.modern)


(Note that even though I'm referring to shadow.cljs.modern, I'm actually not using Shadow CLJS, I'm just using that one file.)

Sam Ritchie20:03:24

Huh, I am not sure what is going on there!


Rats, well thanks anyway. 🙂


You know, I wonder if it has something to do with the fact that I defined it in the repl.

Sam Ritchie21:03:34

That sounds likely!


So I tried @U017QJZ9M7W’s trick and in a CLJ test it works fine, but in my CLJS build I get:

Could not Analyze: Library name must be specified as a symbol in :require / :require-macros; offending spec: (:require-macros [tiltontec.matrix.api]) at line 1 jar:file:/Users/kennethtilton/.m2/repository/com/tiltontec/matrix/4.3.1-SNAPSHOT/matrix-4.3.1-SNAPSHOT.jar!/tiltontec/matrix/api.cljc  jar:file:/Users/kennethtilton/.m2/repository/com/tiltontec/matrix/4.3.1-SNAPSHOT/matrix-4.3.1-SNAPSHOT.jar!/tiltontec/matrix/api.cljc   line:1  column:1
It almost sounds as if it thinks I am doing CLJS (require-macros [tiltontec.matrix.api]), which would require the tick. In tiltontec.matrix.api.cljc I have:
(ns tiltontec.matrix.api
       (:require-macros [tiltontec.matrix.api]))
I am looking at Sam's PR, not seeing a difference. 😞

Sam Ritchie21:03:58

Move it out of the :require block

Sam Ritchie21:03:03

It should sit at the same level


Also side note, @U017QJZ9M7W yup, it seems to only show up in the repl. When I make a new .clj file and put the defmacro in there, it works fine.

👍 1

Ah, coulda stared at that all day. Thx!

🎉 1

I've generated a ".png" file on disk. Is there a way of "opening" it in an image viewer from clojure. Said in other words, I would like to run a function that launches the "default" app for viewing that file type with that file loaded (so I don't have to go into the folder and double click on it and can stay in the REPL).

Jakub Šťastný20:03:30

A possibility is to use the open command if you're on mac. But hopefully someone provides a more Clojure-y solution.

Jakub Šťastný20:03:56

That obviously with the shell just as @U064X3EF3 said.

Jakub Šťastný20:03:06

open opens file in the default app.

Alex Miller (Clojure team)20:03:12

( "open" "foo.png")

👍 1
Jakub Šťastný21:03:09

I'm just learning about transducers. Very cool. My question is how often do you use the traditional map/`reduce` etc vs. reducers and for what use-cases? It seems to me that if the reducing logic becomes complex enough, it's where it makes most sense using transducers. I'm aware of the documentation, i. e., but I'd like to hear "from the field" how, why & when are these used in comparison to the traditional map/`reduce` and friends. That and also in comp. with recursion as a mechanism of mapping/reducing.

👀 1

These days, unless I want a lazy sequence, I am leaning toward using transducers for most transformations that are more than a trivial map or mapv. I don't always remember but I'm getting better about it (after using Clojure for a long time before transducers existed!).

👍 1
Sam Ritchie01:03:44

I think I have the same heuristic, transducers whenever I am doing anything I used to use ->> for with chained transformations . With many exceptions of course!!

👍 1
Sam Ritchie01:03:33

I also try to make my “reducing” functions implement the three arities transducers need

Jakub Šťastný13:03:02

Thanks for the insights!