This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-03-09
Channels
- # announcements (4)
- # aws (3)
- # babashka (86)
- # babashka-sci-dev (31)
- # beginners (171)
- # biff (15)
- # calva (3)
- # clerk (47)
- # cljdoc (11)
- # clojure (57)
- # clojure-dev (11)
- # clojure-europe (122)
- # clojure-losangeles (1)
- # clojure-nl (2)
- # clojure-norway (4)
- # clojure-uk (2)
- # clojurescript (40)
- # cursive (5)
- # data-science (3)
- # datahike (1)
- # datomic (5)
- # fulcro (9)
- # graalvm (8)
- # hyperfiddle (17)
- # introduce-yourself (1)
- # java (28)
- # jobs (1)
- # malli (11)
- # membrane (9)
- # missionary (1)
- # nbb (1)
- # off-topic (5)
- # other-languages (1)
- # pedestal (1)
- # re-frame (4)
- # reagent (16)
- # releases (3)
- # remote-jobs (3)
- # shadow-cljs (83)
- # spacemacs (1)
- # sql (5)
- # tools-deps (28)
- # xtdb (15)
I am attempting to analyze some data that I got from an API. The problem is, when I talked to the API, what I got back are byte arrays. Each byte array “is” a .zip file which has the artifacts
for the job
in question.
If I were analyzing this manually, I would click a button for a job, and my browser would put a .zip file into my ~/Downloads directory. Then I’d move it somewhere convenient, then unzip it to get a folder. Inside the folder, I’d find the files that are going to help shed light on my research question, and I would write Clojure functions to load them, parse them into JSON, and do my analysis. But I have a small # of hundreds of jobs
to look at the artifacts
for.
So what I’ve got now is, an edn file (I chucked all the results of my calls into a file to stop hammering the server with questions), with a sequence of maps that look roughly like {:id 1337 :artifacts [80 75 3 4 20 0 8 8 8 0 110 6 37 86 0 …
and I’m not sure, what’s the most straightforward way to turn that into the data I care about?
wrap them in bytearrayinputstreams and wrap those in a zipinputstream and read the contents
if you aren't familiar with http://java.io.* that might prove to be a good thing because ZipInputStream can be a little different to work with then most kinds of inputstreams
zipinputstream is actually maybe the old way to work with zipfiles, there is a newer api that gives you more of a filesystem like abstraction
when working with inputstreams, do those need to be paired with a reader? i have mostly only used slurp/spit
my big accomplishment for the day was getting, and caching as edn, my data — tomorrow I will get into streams, see how far I can get, then probably come running right back to
. Thank you for the signpost!
https://github.com/clojure/tools.build/blob/master/src/main/clojure/clojure/tools/build/util/zip.clj#L85 has some unzip code to crib from

> there is a newer api that gives you more of a filesystem like abstraction @U0NCTKEV8 do you recall the name of the api?
https://docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/zipfilesystemprovider.html
When (read-string "'()")
becomes (quote ())
. Why does (read-string "
()")` become (clojure.core/list)
and not (syntax-quote ())
?
Because the reader does quote expansion in Clojure seems to be the answer.
backtick is a https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/LispReader.java#L95
I'm little confuse with 'headings forms'. In e.g: 1. (fn name? [params*] exprs*) 2. (defn name doc-string? [params*] exprs*) 3. (def symbol doc-string? init?) Why at 3. heading use 'symbol'? And not 'name' as at 1. & 2. ? As I understand 'symbol' will be created after we initialize it with 'def'. But first we have to pass a 'name'. So, for me it should be (def name doc-string? init?) not (def symbol doc-string? init?). What do you think?
probably because def
Creates and interns or locates a global var with the name of symbol and a
namespace of the value of the current namespace (*ns*).
main part is "the name of symbol"It's kind of confusing. As I understand it, you are passing a symbol, but it gets also treated as a name that you can look up. See:
(defn foo [x] x)
(-> #'foo meta :name) ; foo
(def bar 1)
(-> #'bar meta :name) ; bar
also def
is a special form and defn
is macro
(macroexpand '(def foo 42))
;; => (def foo 42)
(macroexpand '(defn bar []))
;; => (def bar (clojure.core/fn ([])))
> As I understand 'symbol' will be created after we initialize it with 'def'. But first we have to pass a 'name'. > So, for me it should be (def name doc-string? init?) not (def symbol doc-string? init?). > What do you think?
A symbol stands on its own. Yes, it's often a name for a var. You can't evaluate it until you assigned it to a var, except you quote it like 'foo
Symbol and name are kind of interchangeable concepts. But be aware that a symbol is actually a thing and not just a label that is assigned to something else.
You can for example use symbols in macros that get treated at a first class thing and assigned to a var or multiples later on.
The confusing thing is rather why they used both name and symbol in the docs. I assume it is because we think in terms of "function names" both for fn and defn.
symbol and name are not interchangeable concepts. def
expect symbol and will use its name in combination with the value of *ns*
to create and intern global var.
That's why I said "kind of".
What I'm personally not sure about is what the "name" in fn
is. It definitely doesn't assign a symbol to a var in your namespace. Maybe it's just for debugging purposes? But it's very different from def
and defn
"also def
is a special form and defn
is macro" so what fn
is also special form and use 'name'
"If a name symbol is provided, it is bound within the function definition to the function object itself, allowing for self-calling, even in anonymous functions."
> symbol and name are not interchangeable concepts. def
expect symbol and will use its name in combination with the value of *ns*
to create and intern global var.
Symbols are definitely first class names though - just to be sure! def
takes the symbol and assignes a name (the symbol) for it in your namespace.
The name in fn
is something entirely different it seems. Not sure what it does.
for fn
name is bound to that function only and available only within its definition.
also symbol is just symbol. depending on the context it will be used differently: to intern a global var, define local binding, etc.
Has that always been the case?
https://github.com/mentat-collective/emmy/pull/110 haha I had 100s of docstrings attached via :doc
… backed them all out here, this looks way better!!
I lit a bit confuse when I use function set and how this function shaking element in sets
(set {:a 1 :s 2 :y 3}) => #{[:y 3] [:a 1] [:s 2]}
(set {:a 1 :s 2 :y 3 :o 8 :p 6}) => #{[:y 3] [:a 1] [:p 6] [:s 2] [:o 8]}
Why this function change position of element? Where to read about this behavior? Thanks"This is because of the internal structure of the HashSet. The value is transformed in a unique hash, which allows fast access but does not keep the insertion order. If you care about the order in which the elements are added, you need to use a different data structure," from book The Clojure Workshop: Use functional programming to build data-centric applications with Clojure and ClojureScript
What the key of elements in set. In vector key is a index (position) from 0 In map - keyword, string in set? hash of element value?
the only thing that is guaranteed is for maps, iteration order is consistent between (seq m), (keys m), (vals m), where m is a map
> What the key of elements in set. > sets and maps have no order, indexing in them doesn't apply So @U050ECB92 is being accurate, but slightly pedantic. A better term for what I think you mean by this question, that I hope @U050ECB92 will not also object to?, is "lookup value". For vectors, you can look up an element by index. For maps you can look up an element by key. For sets you can look up an element by itself, which is a constant time operation.
(:foo #{:foo :bar})
; =>
:foo
(#{:foo :bar} :foo)
; =>
:foo
Like maps, sets have no specified order. And any circumstantial appearance of any consistency in ordering is purely implementation detail, subject to change at any time, and should not be relied upon.> Sets support...get, the latter returning the object that is held in the set which compares equal to the key, if found:
(def s #{:a :b :c :d})
(get s :a)
-> :a
https://clojure.org/reference/data_structures#Sets(yeah I'm being pedantic: lookup is for finding by arbitrary keys, indexing is finding by an integer in a sequential collection)
I think that distinction is fair if you consider indexing to be a specific less-arbitrary subset of lookup in general. Either way, lookup by self in a set is every bit as arbitrary as lookup by key in a map.
Awesome! It's what I want to knew. What the index/key of set. Before, I don't understand what the different between hash-map and sets as structure data. Can you show me example where you use set or hash-map in real code? For what kind of data using set ?
A few pretty good succinct examples here. Quite common to use sets as functions for existence lookups rather than something like data storage. https://blog.jdriven.com/2020/06/clojure-goodness-using-sets-as-functions/
The usage of hash-maps, sets, lists, vectors, etc. isn't really Clojure-specific as much as it is general computer science knowledge. Barring a few tricks, the way these data structures are applied will be much the same in any programming language. Just clarifying since it might be good to look up a more general source of information too.
Two common uses for maps are domain objects and lookup tables. Typical examples in real code:
• assoc
used to create an "object" map with certain properties, to be passed around: https://github.com/mhuebert/maria/blob/main/shapes/src/shapes/core.cljs#L174 fn in maria.cloud
• a nested lookup table https://github.com/daveliepmann/uruk/blob/master/src/uruk/core.clj#L120
How can I run Clojure so it passes command line arguments as they are to a given fn?
From deps.edn
:
:aliases
{:run
{:exec-fn jakub-stastny.et.runner/custom-runner}}
From src/jakub_stastny/et/runner.clj
:
(defn custom-runner [args]
(println "custom runner") (prn args))
If I call clj -X:run a b c
it passes a b c
into args
, but then if I give a file path it does something weird, clearly Clojure is doing some processing on these arguments.
How can I get command-line arguments as an argument to a function I'm running? I know about *command-line-args*
but I'd prefer an argument to a main-like function rather than a global variable.I wasn't able to figure out whether defining -main
fn is still a thing in Clojure (nor is it particularly clear to me how to do it).
As much as Clojure CLI is a nice system, it's bloody complex. I found Babashka a great first contact with Clojure since running it is much simpler TBH.
This is my first time trying to set up a project using deps.edn
and my head is spinning like mad.
-main is still a thing and if you do that, you can call it from the cli with clj -M -m my.ns
those args are passed as strings to the function and then you can do whatever you want with it
-X / -T expect to invoke a function that takes a map, and thus take k v pairs, and will edn/read each of the arguments
I see. So -M -m
is really what I want then.
I was confused by both -M and -m, I thought it's one or the other.
-M is clojure.main, which takes an argument -m to pass the main class
Thanks @U064X3EF3!
Now it works.
And just to round this out: with -M
you can also use -e
, -i
, and -r
which are also options understood by clojure.main
Interesting. I used this as a challenge to write a deep-merge (fn my-deep-merge' below). Then I saw the splendid deep-merge in @delaguardo’s link (enc as deep-merge below). Now I am trying to work out how I could possibly produce that code in deep-merge ; clearly I have not got the where-withal! How do you visualise the recursion here to see the deconstruction in the fn params - I get <Basis> <induction> type recurses but cant see how you'd design this?
I found myself doing various versions of (conj (butlast acc) (last acc))
when doing reduce:
Adding a key to a map:
(let [acc [{:paths []}]]
(conj (butlast acc) (assoc-in (last acc) [:name] "name")))
; => ({:paths [], :name "name"})
Pushing to an array under a key:
(let [acc [{:paths []}]]
(conj (butlast acc) (update-in (last acc) [:paths] #(conj % "path"))))
; => ({:paths ["path"]})
Since I've been doing it over and over, I wondered whether there might be a shorter version of this?
I was wondering whether it could be handled by assoc-in
/`update-in` only, but I haven't find a way to say "update on last index".
Any suggestions appreciated 🙏:skin-tone-3:you are treating the vector like a stack. And you can use peek
and pop
for quick access
last is 0(n) in the size of the input; it creates a seq from the input and walks the seq to the end
so butlast is dropping the last item in the seq, and then conj is adding at the front
If you need a stack, a list is the fastest option
Right...I see what you're saying. Ruby/JS has only one type of arrays, so I'm quite confused with all these 🙈
OK I thought I got what you guys were saying, but maybe not...my interpretation was: "use list and it all somehow work" and I realised that if list adds to the first of it, then I could just do assoc-in
with index 0
which will be the last item (as in the item I added last, but actually the first one "physically").
However:
user=> (assoc-in [{:name "test"}] [0 :name] "a")
[{:name "a"}]
user=> (assoc-in '({:name "test"}) [0 :name] "a")
Execution error (ClassCastException) at user/eval3 (REPL:1).
So clearly my interpretation is incorrect.
I still understand that list is the way to go, as @U064X3EF3 says, it's the fastest way, but then...?
I just want some elegant way with assoc-in
/`updated-in` so I don't spend ages writing this repetitive code.You just need cons (push), first (peek), and rest (pop)
No I get that, they behave like vectors...but these two functions works on vectors, so why not on lists?
Lists are not indexed
you have what look like logic errors in your code (using last to get the last element, but then conj'ing it back on to the front) that you may want to address before getting concerned about repetitive code
it is possible you might want something like (circular ring buffer type thing), but in that case you still don't want to be using last
@U0NCTKEV8 I'm writing a prototype of something. I'm more worried about how much code I need to write than about performance. I get your point though, but my priority is elsewhere. I do want to learn Clj properly and write proper code, but as of now "it works" is more important than anything else.
I am not just saying performance, I am saying if the intent of the code is to treat accum like a stack, then it isn't doing that correctly
it is "popping" from the end of the list, and adding the beginning, more like a queue
Anyway the right code is then:
(prn (reduce
(fn [acc i] (cons {:name i} acc))
'()
["a" "b" "c"]))
That and first
and rest
when applies.But essentially the code will be doing exactly the same as my original code in terms of code structure. It might be more correct and fast, but no chance for less code then?
OK that's a fair start (let [[f & r] '(1 2 3)] [f r]) ; [1 (2 3)]
.
you can even abstract the whole thing out (fn [f] (fn [a b] (cons (f (first a) b) (rest a))))
presumably at some point you want to push a new element onto the stack instead of updating the top
Yes of course.
@U0NCTKEV8 Would you have an example of how to use it as a transducer? That one's a bit out of my CLJ league still 🙈 But it's something I wanted to learn, but I'm still not getting it.
(defn stack [g]
(fn [rf]
(completing
(fn [accum item]
(let [accum (g accum)]
(cons (rf (first accum) item) (rest accum)))))))
(transduce
(stack (fn [stack]
(if (> (count (:path (first stack))) 2)
(cons {:path []} stack)
stack)))
(completing
(fn [a b]
(update-in a [:path] conj b)))
'({:path []})
(range 10))
output of the above is
({:path [9]} {:path [6 7 8]} {:path [3 4 5]} {:path [0 1 2]})
Thank @U0NCTKEV8, I really appreciate your help.
Going to read up on transducers now, because this is over my head still 🙈
I see there's a Rich's video on them, great, I love these.
Ignoring all the above discussion about stacks and whatnot, and focusing on your original question and isolated code samples:
It's not clear from the docs, but you can use update
on a vector, with an index as the "key", as shown in the https://clojuredocs.org/clojure.core/update#example-58075cdce4b001179b66bdcf.
This does not save you any characters, but I think it is a bit more declarative representation of what your code is doing, and feels less awkward.
(defn last-idx [v] (-> v count dec))
(let [acc [{:paths []}]]
(update acc (last-idx acc)
#(assoc % :name "name")))
; =>
({:paths [], :name "name"})
(let [acc [{:paths []}]]
(update acc (last-idx acc)
#(assoc % :paths "path")))
; =>
({:paths ["path"]})
Is there any way I can reprovide a macro in clojurescript?
Like, if file internal.clj
has a macro, but I want it to be brought in whenever frontend.clj
is required, is there anyway I can do that.
If it was a value I could just do something like:
(ns frontent
(:require [backend])
(def a-value backend/a-value)
But for a macro...well that obviously doesn't work...I suppose I could eta expand it to get:
(defmacro a-macro [& args] `(backend/a-macro [email protected]))
But that feels...off...to me."Is there any way I can reprovide a macro in clojurescript?" Scary. I spent a couple hours doing this today. "Trying", I should say. gave up.
"(def a-value backend/a-value)"
But for a macro...well that obviously doesn't work...
Did you try it? I made the same assumption and did not even try either, but one never knows.
(defmacro a-macro [& args] `(backend/a-macro [email protected]))
That's what I did, as well as just copy the whole damn macro definition.
I seemed to making headway, but I do an anaphoric variable injection of sth akin to JS "this" and it works great, but it got lost when I did the reprovisioning -- CLJS now complained the anaphor was undefined.
btw, I was just trying to reduce the "requires" in a client, so I did not care how gross was my solution. And my API is small, so there would not be a lot of it.
Hoping a better answer comes along!> Did you try it? I made the same assumption and did not even try either, but one never knows.
Ya...I did, works about as well as you'd expect. In racket its easy enough to do with make-rename-transformer
, but alas, does not seem to be a thing here.
> I seemed to making headway, but I do an anaphoric variable injection of sth akin to JS "this" and it works great, but it got lost when I did the reprovisioning -- CLJS now complained the anaphor was undefined. I have a hunch that its due to the auto-namespace injection that clojure's quasiquote inserts. Care to share your current headway?
Let me put it back in and get a fresh fail. The headway may have been that lein test
passes, but that just tests Clojure. (I have a big CLJC project, never got round to automated testing for CLJS.) brb
(defmacro c-fn-var-ex [[c] & body]
`(fn [~c]
(let [~'me (c-model ~c) ;; <================ me
~'_cell ~c
~'_slot-name (c-slot ~c)
~'_cache (c-value ~c)]
[email protected])))
(defmacro c-fn [& body]
`(c-fn-var (~'slot-c#) [email protected]))
(defmacro cF [& body]
`(tiltontec.cell.core/make-c-formula
:code '~body
:value tiltontec.cell.base/unbound
:rule (tiltontec.cell.core/c-fn [email protected])))
[Figwheel:WARNING] Compile Warning src/web_mx_quickstart/core.cljs line:64 column:45
Use of undeclared Var web-mx-quickstart.core/me
59 {:default :ignore
60 :on-navigate (fn [route params query]
61 (when-let [mtx @md/matrix]
62 ;; todo mset! me
63 (mset! mtx :route route)))}))
64 :selected-lesson (cF (let [route (mget me :route)]
^---
65 (some (fn [lesson]
66 (when (= route (:route lesson))
67 lesson)) lessons)))
68 :keydowner (cF+ [:watch (fn [_ me new _ _]
69 (.addEventListener js/document "keydown" new))]
It then fails at run time because mget
receives nil
in that position.
Let me try modifying one of the CLJ tests to use the repro NS. brb...OMG. I thought :refer-macros
was no longer needed. The CLJS works now with:
[tiltontec.matrix.api
:refer-macros [cF]
:refer [mpar mget mset! mswap! mset! fasc fmu] :as mx]
Did I misunderstand about :refer-macros
being vestigial?OK, :include-macros
does the trick (thx!).
Trying again with:
(defmacro cF [& body]
`(tiltontec.cell.core/cF [email protected]))
...instead of reiterating the whole macro, but my thinking was that this code is crazy mature (so not changing much) and then when the user does a find definition, their first hop won't be completely useless.
As for the original question, I have kicked off a couple threads recently on how to spare users (and me!) huge requires to use my library, given my preference for small files. No good answer was found, so I am leaving them where they are and doing a separate tiltontec.matrix.api
namespace where I repro.
Looking forward to more input on this thread, which has already salvaged my morning's work!You can still avoid include-macros!!
On my phone, searching for an example, one sec …
If you add a :require-macros call to the cljs side of the namespace where the macro is written, no one else requiring that namespace has to do anything beyond :refer
But if you've already been given a clojurescript file which you don't want to change, it seems like include-macros
is still required, ya?
Ah yes sorry I misunderstood. You are right
Or push a PR upstream, that’s what I did here since i wanted all of those pesky include-macros calls gone from my library
One day perhaps clojurescript will never require include-macros, even for libraries that haven't added require-macros
. Until then, thanks anyway. 🙂
Although I guess while I have your attention, is there any way (at runtime) to get a list of all the macros a namespace provides?
Yeah I think all of the var info / metadata is gone at runtime
oddly enough though, it seems like cljs.repl/dir
uses ns-publics
, so not sure why usig ns-publics
directly doesn't work.
(looking at https://github.com/clojure/clojurescript/blob/r1.11.60/src/main/clojure/cljs/repl.cljc )
You can call ns-publics in a macro, since that will run at compile time
On the JVM
But the macroexpansion won’t have a runtime reference to ns-publics left over
Ah, so your saying that dir
works because its a defmacro
, but if it was defined with defn
, then it wouldn't work?
That’s right
Which I guess would explain why ClojureScript's implementation of dir
doesn't have a similar dir-fn
.
Hmm...strange, looks like even when I use it in a macro, I still am not getting the macros:
cljs.user=> (defmacro foo [arg] (ns-publics 'shadow.cljs.modern))
#'cljs.user/foo
cljs.user=> (foo)
{}
cljs.user=> (clojure.repl/dir shadow.cljs.modern)
defclass
js-await
js-template
nil
cljs.user=>
(Note that even though I'm referring to shadow.cljs.modern
, I'm actually not using Shadow CLJS, I'm just using that one file.)
Huh, I am not sure what is going on there!
You know, I wonder if it has something to do with the fact that I defined it in the repl.
That sounds likely!
So I tried @U017QJZ9M7W’s trick and in a CLJ test it works fine, but in my CLJS build I get:
Could not Analyze: Library name must be specified as a symbol in :require / :require-macros; offending spec: (:require-macros [tiltontec.matrix.api]) at line 1 jar:file:/Users/kennethtilton/.m2/repository/com/tiltontec/matrix/4.3.1-SNAPSHOT/matrix-4.3.1-SNAPSHOT.jar!/tiltontec/matrix/api.cljc jar:file:/Users/kennethtilton/.m2/repository/com/tiltontec/matrix/4.3.1-SNAPSHOT/matrix-4.3.1-SNAPSHOT.jar!/tiltontec/matrix/api.cljc line:1 column:1
It almost sounds as if it thinks I am doing CLJS (require-macros [tiltontec.matrix.api])
, which would require the tick.
In tiltontec.matrix.api.cljc I have:
(ns tiltontec.matrix.api
(:require
#?(:cljs
(:require-macros [tiltontec.matrix.api]))
...etc)
I am looking at Sam's PR, not seeing a difference. 😞Move it out of the :require block
It should sit at the same level
Also side note, @U017QJZ9M7W yup, it seems to only show up in the repl. When I make a new .clj file and put the defmacro in there, it works fine.
I've generated a ".png" file on disk. Is there a way of "opening" it in an image viewer from clojure. Said in other words, I would like to run a function that launches the "default" app for viewing that file type with that file loaded (so I don't have to go into the folder and double click on it and can stay in the REPL).
you can use http://clojure.github.io/clojure/clojure.java.shell-api.html#clojure.java.shell/sh to shell out with open
A possibility is to use the open
command if you're on mac. But hopefully someone provides a more Clojure-y solution.
That obviously with the shell just as @U064X3EF3 said.
open
opens file in the default app.
I'm just learning about transducers. Very cool. My question is how often do you use the traditional map
/`reduce` etc vs. reducers and for what use-cases? It seems to me that if the reducing logic becomes complex enough, it's where it makes most sense using transducers.
I'm aware of the documentation, i. e. https://clojure.org/guides/faq#transducers_vs_seqs, but I'd like to hear "from the field" how, why & when are these used in comparison to the traditional map
/`reduce` and friends.
That and also in comp. with recursion as a mechanism of mapping/reducing.
These days, unless I want a lazy sequence, I am leaning toward using transducers for most transformations that are more than a trivial map
or mapv
. I don't always remember but I'm getting better about it (after using Clojure for a long time before transducers existed!).
I think I have the same heuristic, transducers whenever I am doing anything I used to use ->> for with chained transformations . With many exceptions of course!!
I also try to make my “reducing” functions implement the three arities transducers need
Thanks for the insights!