Fork me on GitHub
#beginners
<
2020-01-17
>
Chase01:01:38

so when you realize a lazy sequence like (take 10 (iterate inc 0)) or whatever, it only realizes a small chunk of it right? I think I read its 32 elements or something?

bfabry01:01:08

depends on which particular lazy sequence iirc. but yes either exactly 10 or 32. not all

alexmiller03:01:50

iterate by itself is not chunked, so here 10

Chase01:01:41

offshoot lazyness question. If I declare a variable like (def my-lazy-seq (iterate inc 0)) or (def my-range (range)) and then go (realized? my-range) I get true. Why is it already realized if I haven't told it to do anything?

bfabry01:01:40

that should not be what you get

bfabry01:01:49

it's also what I get though... must be something new

Chase01:01:20

Those docs did mention a change in 1.7 I think. Says it returns a LongRange now instead of a lazy seq

Chase01:01:55

Not sure why my iterate example is realized though. Did you try that one?

bfabry01:01:08

the first element is 0

bfabry01:01:12

which is realized as a literal

bfabry01:01:55

so specific to iterate, in that it already knows its first element

Chase01:01:15

interesting. ok. Yeah, because when I do (def x (map inc (iterate inc 0))) it is not realized

bfabry01:01:24

user=> (realized? my-lazy-seq)
true
user=> (realized? (rest my-lazy-seq)
)
false

Chase02:01:27

I am still seeing the doc string for range saying it is a lazy seq

bfabry02:01:09

I'd say that's pretty close to a bug on the docs

bfabry02:01:33

although.. I dunno

Chase02:01:35

oh, but the source says a zero arity call to range is (iterate inc' 0) so now we are back to it knowing the first element...

bfabry02:01:48

(range) actually returns an Iterate object

bfabry02:01:02

so it's more like iterate, a special case where it happens to know the head already

Chase02:01:54

Cool. I think we've solved these mysteries. Whether my understanding of lazyness has improved is inconclusive. Lol. Thanks!

Chase02:01:46

And I will never grow tired of just asking the repl for docs and/or source of core functions and actually being able to understand some of what's happening. Clojure is the best!

alexmiller03:01:04

in general, I consider the use of realized? to generally be a smell. we went back and forth several times on exactly what the customized impls of iterate and range should do here and whether it means "I know the first element" or "first has been forced" and I'm not sure there is a good and useful answer.

👍 4
Chase02:01:20

So I was just reading the source of range and see this: (if (and (instance? Long start) (instance? Long end))... . instance? is new to me. Is this an example of a little ad hoc type checking in clojure? Is this something people find themselves using or am I completely misunderstanding what is going on?

Chase02:01:18

Considering the else branch of that if statement carries on as if it was still given a number (I think?) I think I am misunderstanding the situation.

andy.fingerhut02:01:31

It is run time type checking

andy.fingerhut02:01:23

You will find it here and there in the implementation of Clojure. It can be useful in library and application code, too, in some circumstances.

andy.fingerhut02:01:36

In the JVM, Long is a 64-bit integer. There are plenty of other kinds of numbers that are not those, that the else branch handles, e.g. ratios, doubles, BigInteger, BigDecimal, ...

Chase02:01:33

Thanks for the clarifications!

alexmiller03:01:03

the special case of all long values for range is a) very common and b) highly optimizable, so this is a case where that special case returns an optimized specialized impl

alexmiller03:01:43

btw, these are all excellent questions. However, questions in this slack fall off the archive and do, admittedly, show up in external logs and zulip (login-walled) logs, but don't seem to be well google-indexed. A site exists, run by the core team, specifically for Clojure questions at https://ask.clojure.org. This site is well-indexed (and I actually have spent some time tweaking things to increase that) and will surface old questions if you start to ask a new one that is similar, so I would encourage you to consider using that site instead for tight, well-scoped questions like this.

clj 24
tzzh09:01:25

Not sure what channel to ask but what are the best natively compiled formatters and linters ? (I have seen clj-kondo for linting and a couple of projects seem to be trying to run cljfmt with graalvm)

mloughlin09:01:10

best in which scenario? I happily use clj-kondo in my editor.

tzzh09:01:26

basically I’d like to setup a pre-commit hook for formatting and linting, so speed is quite important. clj-kondo looks sick but I am not sure what the equivalent for formatting would be.

borkdude09:01:00

maybe zprint?

tzzh10:01:26

thanks will take a look - this https://github.com/cespare/goclj/tree/master/cljfmt looks interesting as well

didibus10:01:57

I use cljfmt built with GraalVM

didibus10:01:05

for formatting, pretty fast

littleli10:01:50

is there a graalvm build somewhere?

didibus10:01:03

But it doesn't have pre-compiled binaries, so you need to build it yourself. And last time I did, that was like a year ago, it was using tools.cli which wasn't compatible with GraalVM native image, so I had to rewrite that part to not use tools.cli myself.

didibus10:01:52

I think zprint does offer pre-built binaries for linux and mac

didibus10:01:24

So for an easy download and run experience it is probably better. Never used it though

didibus10:01:47

clj-kondo is definitly the best native linter otherwise

bartuka10:01:57

I have a question about macros, code example:

(s/def ::spec-value int?)
(def spec-maps {:value ::spec-value})

(defmacro def-parser-1
  [name]
  `(s/keys :req-un [~name]))

(s/valid? (def-parser-1 ::spec-value) {:spec-value "13123"})
(s/valid? (def-parser-1 (:value spec-maps)) {:spec-value "1323"})
The second example raises an exception because s/keys need his argument to be a full qualified keyword, how can I compute (:value spec-maps) before the s/keys realizes?

mloughlin12:01:52

looking into the spec.alpha codebase it seems like s/keys is a macro (https://github.com/clojure/spec.alpha/blob/cd4aeb7edccbf7a881dcdc3c947313359508db3a/src/main/clojure/clojure/spec/alpha.clj#L416), which is why (:value spec-maps) isn't evaluating like you'd expect with a function

mloughlin12:01:29

what happens if you modify def-parser-1 to

(defmacro def-parser-1
  [name]
  `(let [realised-name ~name]
	 (s/keys :req-un [realised-name])))
? Once you pass a form to a macro, the rules on whether or when it gets evaluated become completely up to the macro-writer.

bartuka12:01:34

If I try your version of the code, I get this error of let binding:

Call to clojure.core/let did not conform to spec.
   #:clojure.spec.alpha{:problems
                        ({:path [:bindings :form :local-symbol],
                          :pred clojure.core/simple-symbol?,
                          :val leitor.core/realised-name,
                          :via
                          [:clojure.core.specs.alpha/bindings
                           :clojure.core.specs.alpha/bindings
                           :clojure.core.specs.alpha/binding
                           :clojure.core.specs.alpha/binding-form
                           :clojure.core.specs.alpha/binding-form
                           :clojure.core.specs.alpha/local-name],
                          :in [0 0]}
                         {:path [:bindings :form :seq-destructure],
                          :pred clojure.core/vector?,
                          :val leitor.core/realised-name,
                          :via
                          [:clojure.core.specs.alpha/bindings
                           :clojure.core.specs.alpha/bindings
                           :clojure.core.specs.alpha/binding
                           :clojure.core.specs.alpha/binding-form
                           :clojure.core.specs.alpha/binding-form
                           :clojure.core.specs.alpha/seq-binding-form],
                          :in [0 0]}
                         {:path [:bindings :form :map-destructure],
                          :pred clojure.core/map?,
                          :val leitor.core/realised-name,
                          :via
                          [:clojure.core.specs.alpha/bindings
                           :clojure.core.specs.alpha/bindings
                           :clojure.core.specs.alpha/binding
                           :clojure.core.specs.alpha/binding-form
                           :clojure.core.specs.alpha/binding-form
                           :clojure.core.specs.alpha/map-binding-form
                           :clojure.core.specs.alpha/map-bindings],
                          :in [0 0]}
                         {:path [:bindings :form :map-destructure],
                          :pred clojure.core/map?,
                          :val leitor.core/realised-name,
                          :via
                          [:clojure.core.specs.alpha/bindings
                           :clojure.core.specs.alpha/bindings
                           :clojure.core.specs.alpha/binding
                           :clojure.core.specs.alpha/binding-form
                           :clojure.core.specs.alpha/binding-form
                           :clojure.core.specs.alpha/map-binding-form
                           :clojure.core.specs.alpha/map-special-binding],
                          :in [0 0]}),
                        :spec
                        #object[clojure.spec.alpha$regex_spec_impl$reify__2509 0x726cc2a1 "[email protected]"],
                        :value
                        ([leitor.core/realised-name (:value spec-maps)]
                         (clojure.spec.alpha/keys
                          :req-un
                          [leitor.core/realised-name])),
                        :args
                        ([leitor.core/realised-name (:value spec-maps)]
                         (clojure.spec.alpha/keys
                          :req-un
                          [leitor.core/realised-name]))}
                 alpha.clj:  705  clojure.spec.alpha/macroexpand-check
                 alpha.clj:  697  clojure.spec.alpha/macroexpand-check

bartuka12:01:39

(macroexpand-1 '(def-parser-3 (:value spec-maps)))
;; => (clojure.core/let [leitor.core/realised-name (:value spec-maps)] (clojure.spec.alpha/keys :req-un [leitor.core/realised-name]))

mloughlin14:01:15

let me load up a repl and see if I got my syntax wrong

bartuka14:01:36

Thanks for helping. I tried several things already. ;)

mloughlin15:01:47

to state the obvious, we need a way to trick the var spec-maps to evaluate at compile time.

(defmacro def-parser-1
    [name]
    `(s/keys :req-un [~(eval name)]))

mloughlin15:01:59

idk how much of a Clojure sin that is

bartuka15:01:39

Haha I swear I went to my job thinking about this exact solution

bartuka15:01:18

I saw this on one PR from spec-tools where @U055NJ5CC made the usage of this bad guy. Still, we can live with that?

mloughlin16:01:19

I wonder if the eval works differently in different contexts. I only tested it in REPL.

Adrian Smith11:01:38

Doing an informal survey: When learning Clojure what features of Clojure drew you in the most? And what language(s) did you come from?

bartuka11:01:31

Community, Python & Academic Algorithms in Fortran 77 :)

Joaquin Iglesias12:01:12

Its a Lisp with Java interop. I wanted to learn a Lisp, and Clojure seemed the most practical. Python and R

mloughlin12:01:28

• Persistent data structures/immutability by default. • Data oriented, lack of OO. • Ruthless focus on simplicity and pragmatism. E.g. Not being afraid of introducing reader macros into Lisp for heavily used datastructures {} [] • Rich Hickey. I professionally use C#, JS & Python. I already knew Common Lisp before learning Clojure. Clojure feels like it equips you with all the tools you need to avoid shooting yourself in the foot and will get you 3/4 of the way to a well organised code-base without even trying.

jaihindhreddy12:01:23

From Python, PHP and JS with some exposure to Haskell and Go. Here's the things I love about Clojure (and where I encountered or understood them) • Simplicity (Thought Go was simple. Saw someone say Go's authors don't know what simple means, then saw Simple Made Easy) • Realised syntax in text instead of data is purely made up with no advantages. (From Rich's interview with Brian Beckman, and Clojure for Java Programmers) • Realised you can avoid playing both the inheritance and/or mutability game of OO, and the types game of FP, and just do data. (From Clojure, Made Simple) • Reified names • Macros (Guy Steele's Growing a Language spoke to me) • Identities built on values instead of the other way around (from Are We There Yet?) • Polymorphism a la carte (from Clojure for Lisp Programmers, which made the idea of Polymorphism itself click in my head) over pattern matching • Immutability. This should be present in every line, because it enables most of these things. • The possibility of doing systems without the tyranny of the container.

shaun-mahood15:01:03

The deep thought that went into so many of the conference talks (I spent about 2 years convincing myself to try Clojure seriously), and the search for a programming language that could replace my use of C# and JS and reduce the amount of context switching when building full-stack applications.

chepprey16:01:37

15 years ago, drowning in an ocean of "enterprise Java" and OO at the height of its fashion, feeling like "this just can't be right". Started looking for better ways to do things. Discovered Erlang, got some books and started tinkering; discovered Paul Graham's "On Lisp", started reading about Haskell, .... thought ok, yes, there ARE better ways. Shortly after that Rich Hickey drops Clojure and the amazing conference talks, esp. Simple Made Easy. Welp, we have a winner!

Joaquin Iglesias12:01:31

I have a macro question. I am developing a microservice, I will get a json, that I need to use to instantiate some ugly Java class that got written 10 years ago. The class has no setters or getters so I am using (set! (. class-instance field) value). Most parameters are optional, and I dont want to write a ton of conditionals checking weather the parameter is in the json and whatnot. I have managed to get the thing working with no macros, some metaprogramming and an eval. I know eval is bad form, so I would like to know if any of you can spot a cleaner way to do this.

(def a-map {"tipoInmueble" "VPT" "superficie" 100.0 "codPostal" (int 28014)})

(defn write-code [some-map]
  (let [some-entrada (new PretasacionInt)]
    (for [km (keys some-map)]
      `(set! (.  ~'some-entrada ~(symbol km)) ~(some-map km)))))

(def some-code (write-code a-map))
(pp/pprint some-code)
Yields
((set! (. some-entrada tipoInmueble) "VPT")
 (set! (. some-entrada superficie) 100.0)
 (set! (. some-entrada codPostal) 28014))
Which is the bulk of the work. Now no matter what parameters get passed in the map, I can send them to the Java object To finish up
(def better-code `(let [~'some-entrada (new PretasacionInt)]
                    [email protected]
                    ~'some-entrada))
(pp/pprint better-code)
Yields
(clojure.core/let
 [some-entrada (new objeto.PretasacionInt)]
 (set! (. some-entrada tipoInmueble) "VPT")
 (set! (. some-entrada superficie) 100.0)
 (set! (. some-entrada codPostal) 28014)
 some-entrada)
Now its a matter of (eval better-code) and i am done. I would think that just setting better-code into a macro with some parameters would do the job, and get rid of my eval but
(defmacro my-not-working-macro [random-code]
  `(let [~'some-entrada (new PretasacionInt)]
                    [email protected]
                    ~'some-entrada))
Yields
Don't know how to create ISeq from: clojure.lang.Symbol
I have done some fidgeting with the unquoting/quotining-unquoting/etc of random-code to no success. I have also tried to call the first function within the macro, but then it got really confusing whith the Java class, its symbol and local and gloval environments. Finally, I know metaprogramming should be avoided, when doing something like
(let [some-entrada (new PretasacionInt)]
    (set! (.  some-entrada (symbol "superficie")) (a-map "superficie")))
I got
Caused by java.lang.IllegalArgumentException
   Invalid assignment target

Kamuela16:01:29

Is there a small tutorial to understand dependencies managed with just an .edn file rather than say leiningen? Got it: https://clojure.org/guides/deps_and_cli

alexmiller16:01:23

if you have questions or suggestions, feel free to log issues here: https://github.com/clojure/clojure-site/issues

Kamuela18:01:37

Thanks for that. Seems alright as it is. Can I make PR's to do things like update REPL's to show latest versions instead? Just various cosmetic things (right now example REPL's are at 1.9)

bartuka16:01:42

@alanfborba parrot🎉clj🚀

parrot 4
🎉 8
clj 4
🚀 4
bibiki16:01:40

can anyone please advice, buddy or friend for authentication?

shaun-mahood16:01:40

I believe that friend is pretty much unmaintained, everything I've seen for the past few years has recommended buddy (though I haven't looked for a while so something may have changed since then)

bibiki17:01:42

Yeah, friend's repo has not been active for 3 years now. Seems like buddy is what I will go with. But at first it was difficult to decide as google brought friends up first, but compojure repo seemed to have example with buddy.

Nico Hartto13:01:36

buddy is good, just recently used it in a project (sign, jwe, jwt). bouncycastle is beneath the surface

JoshLemer20:01:33

do you guys know any clojure library for an implementation of a persistent map, which preserves key order? I know there is array map in the official docs, but that’s only suitable for small maps.

Kamuela22:01:05

Is there supposed to be a way to generate new projects with the clojure tool? Like clojure new or something? Or should I set it up myself?

seancorfield22:01:40

@kamuela Yes. clj-new. It's listed on the Tools page on the tools.deps.alpha wiki.

seancorfield22:01:01

Also, look at my dot-clojure repo for lots of examples of tools to use with clojure

Jeremy Plichta22:01:28

what about lein new app <myapp> ?

seancorfield22:01:56

@jplichta That will create a Leiningen-based project, not a deps.edn-based project.

Jeremy Plichta22:01:17

ah, thanks. I am pretty new to this and have only used Leiningen projects so far.

Kamuela22:01:34

Same @jplichta, but that's why I'm trying to dive head-first into the new tools

Jeremy Plichta22:01:50

ill check it out. thanks!

seancorfield22:01:24

We switched from lein to boot at work back in 2015 and then to the CLI/`deps.edn` in 2018. I never use Leiningen these days (unless I'm helping someone here debug a problem with their lein project 🙂 )

JoshLemer22:01:41

Don’t you think it’s a little odd that ordered maps are not more commonly used in programming? Seems like it would have a lot of benefits in terms of deterministic behaviour

Kamuela22:01:02

So is the direction of travel modular (positive)/hodgepodge (negative) with regard to the clojure official tooling? Is the ultimate state intended to require the use of many separate tools?

seancorfield23:01:45

@kamuela Since the main channel has gone off on ordered maps, happy to chat in a thread more about clojure tools if you have any additional questions!

Kamuela23:01:50

Just got the alias installed actually, trying it all out now

seancorfield23:01:00

Simple, composable tooling fits with the Clojure mindset.

seancorfield23:01:26

@joshlemer I've never needed an ordered hash map in Clojure, after nearly nine years of production use.

seancorfield23:01:48

There is a sorted hash map that I've used maybe once or twice.

seancorfield23:01:36

@kamuela The other issue is that Clojure doesn't need compiled, packaged artifacts, in Maven-like repos: it can run entirely from source, using local or git coordinates.

JoshLemer23:01:01

I mean, you could just use an ordered hashmap wherever you use hashmap, and you'd get a bit worse prformance, but your keys would always be in the same order they're added to the map which is like a little bit more predictability right

hiredman23:01:10

There is a continuim from hash maps to databases, sorted maps are somewhere in the middle

JoshLemer23:01:33

Like rather than having keys scrambled, that's a bit of a win.

hiredman23:01:27

So the tend to get left out, because generally people for one end or the other

hiredman23:01:33

The cases where I have considered using a sorted map have been for indices of in memory databases

seancorfield23:01:34

@joshlemer What is the use case for wanting to track insertion order in a hash map? If I needed things in a particular order, I'd be inclined to use a vector either instead of, or as well as, a hash map.

hiredman23:01:59

Yeah, insertion order is never the order I want from a sorted map

JoshLemer23:01:53

So like, imagine parsing json and you represent objects as maps naturally. If you don't use an ordered map, then a roundtrip JSON -> AST -> JSON will scramble all the json objects.

andy.fingerhut23:01:09

The main times I have wanted an ordered map of any kind are when printing out maps and doing diffs between previous-version-of-code debug output and possible-new-version-of-code debug output.

andy.fingerhut23:01:58

to make the diffs smaller

JoshLemer23:01:20

So that's not great for say, writing a json formatter, or displaying errors to users (they would typically like to see output corresponding to input) Or think about representing arbitrary "row" datatypes like from JDBC or Cassandra. You would like to keep the row's columns in the same order they exist in the DB

seancorfield23:01:21

In all my years of working with both JDBC databases and document-based (MongoDB) with Clojure, that's never been an issue in real world production code.

hiredman23:01:36

Why? The whole point of jdbc producing maps is to avoid order

seancorfield23:01:39

(and both clojure.java.jdbc and next.jdbc offer a vector-based result set format that preserves column order: vector of (vector of column names) followed by (vector of column values) for each row)

JoshLemer23:01:04

I mean, imagine an API where rows are not represented like that, but instead by maps. In that case, it would be easier to deal with maps that follow the same order as the column order.

JoshLemer23:01:27

it would be easier to, say, convert between the map representation and your vector-of-vectors representation, than if you had to carry around the column ordering on the side. IMO anyways

JoshLemer23:01:31

Doesn't sound like I will convince anyone though hehe

seancorfield23:01:46

In real world production code, the lack of order of column names does not matter.

Kamuela23:01:36

Insertion order isn't that algorithmic use case of a hash map generally, you get O(1) because of the hash function that allows you to store data anywhere

andy.fingerhut23:01:16

The Clojure implementation of insertion order map that I am aware of, https://github.com/clj-commons/ordered, does preserve O(1) search time, because it maintains the insertion order separately from a normal Clojure map, in its implementation.

Kamuela23:01:03

Sorry, which was the one linked?

andy.fingerhut23:01:44

I edited my message to include the link. I saw it linked earlier today, but maybe not this channel.

andy.fingerhut23:01:20

Thus that implementation preserves fast lookup/update operations, like normal Clojure maps, with the tradeoff of some extra memory for each map to store the key insertion order.

Kamuela23:01:19

I'm inclined to agree with @seancorfield's suggestion of adapting a vector-ish solution

Kamuela23:01:11

So I'm looking at this and not understanding the syntax sometimes: https://github.com/clj-commons/ordered/blob/master/src/flatland/ordered/map.clj, line 48, 50, 70, 76 just seem to have unbounded literals

andy.fingerhut23:01:56

Line 48 contains MapEquivalence which is a class name without its package qualifier. If you look at the import line in the ns form near the top, you will find the full package name of that class

andy.fingerhut23:01:43

more precisely, some of those may be Java interface names, but those are represented as classes in the JVM, with extra flags identifying them as interfaces, I believe.

hiredman23:01:17

The few times I recall working on a project were we need to hash some canonical serialization we implemented a custom serializer for that purpose that did thinks like sorting map keys

hiredman23:01:01

So it didn't matter what the concrete type of the map was, it would serialized to a canonical format

hiredman23:01:32

There are open source clojure libraries that do something similar

hiredman23:01:35

I think you can do interesting things with sorted maps, but insertion order sorting is the worst least useful sorting you can pick

hiredman23:01:22

But maybe that is why people don't use sorted maps, you have to pick 1 sorting and no one agrees on that one sorting

andy.fingerhut23:01:22

A utility that modifies a JSON input file and produces a possibly-modified JSON output file is at least one special case where you could perhaps understand wanting to maintain insertion order?

hiredman23:01:03

Which is why I said they exist on a continuum with databases, since a database can be seen as a map with multiple sorts (each index)

hiredman23:01:05

Maybe? I would question if sorting shouldn't be a feature of the serializer

hiredman23:01:51

Because it is very easy to do some operation sorted map in -> unsorted map out, and then you loosing that guarantee

hiredman23:01:35

If that sort is important to you, you should consider enforcing it

noisesmith23:01:49

I've also needed an insertion ordered map for a case where an API took a json payload, but errored if top level keys were not in the expected order, it was either ordered map, or create the json from a string template instead of a serialization lib

andy.fingerhut23:01:55

Clojure operations on maps do make it easy to lose any sorting guarantees, if you are not careful, certainly. You can be careful about it if you are aware of the issue.

JoshLemer23:01:57

I mean even just take a simple example, maybe a bit contrived, that might happen in simple every day coding

(into [] (into {} [["hello" 1] ["world" 2]]))
It's completely nondeterministic (well it is deterministics but, not really meaningful) whether you get [["hello" 1] ["world" 2]] I know I'm stating the obvious here but I personally find this scrambling at least a slight annoyance

hiredman23:01:26

sure, but maps are unordered, and I know that, so I don't do that

JoshLemer23:01:15

I know, you have to know not to do that, and work around that. That is what I am calling an annoyance.

andy.fingerhut23:01:25

Replace the {} with a function call that creates an empty insertion order map, and then as long as you avoid operations on maps that do not return a normal unordered map (I don't have a complete list handy right now), you can avoid that annoyance.

hiredman23:01:39

you have to know they are insertion order sorted to make that safe

andy.fingerhut23:01:40

In exchange for the annoyance of knowing which operations will return unordered maps.

hiredman23:01:43

either way you have to know

seancorfield23:01:03

(and the "annoyance" of poorer key access performance than a regular hash map 🙂 )

seancorfield23:01:21

(and probably poorer key insertion/deletion too)

andy.fingerhut23:01:22

no. They have same lookup/assoc performance as regular hash maps

hiredman23:01:48

so "insertion order" preserving isn't more natural and not annoying in some way, you just have a personal expection of it, and clojure as a language doesn't match your personal expectations

andy.fingerhut23:01:55

I mean, it isn't O(n), for sure. It does have to update its internal stored version of the key insertion order if you add a new key, of course.

JoshLemer23:01:07

Yeah typically they have the same lookup, but worse assoc/dissoc

seancorfield23:01:18

@andy.fingerhut That depends on how they are implemented -- there are multiple possibilities, depending on what you want to optimize for.

JoshLemer23:01:22

at least, the persistent versions that is

andy.fingerhut23:01:41

I am talking about this implementation, which I have examined carefully and fixed some bugs in: https://github.com/clj-commons/ordered

seancorfield23:01:50

Ah, fair enough.

hiredman23:01:39

https://github.com/tonsky/persistent-sorted-set is very cool (the sorted set behind datascript), but not insertion ordered

hiredman23:01:46

that reminds me, we do use insertion order sets at work, the graphql library we use uses them because it returns clojure data and leaves json serialization up to you, and I believe the graphql spec makes order matter or something

noisesmith23:01:34

yeah, json maps are ordered so sometimes when talking to json apis you hit things that break with reordering

andy.fingerhut23:01:03

Really? Wow, that seems so wrong somehow.

andy.fingerhut23:01:21

I mean, that implementations of APIs that take JSON as input, depend upon the order of keys

hiredman23:01:47

not every api actually parses json into maps

noisesmith23:01:49

it's obnoxious to deal with - but hey that's the point of an API - some company owns access to the data, they aren't going to let me rewrite their code

andy.fingerhut23:01:39

Out of curiosity, I just checked https://www.json.org and it says on the English home page: "An object is an unordered set of name/value pairs." I know that doesn't stop people from making implementations that are more fragile/strict/whatever than that.

noisesmith23:01:51

I guess the API is very broken then?

noisesmith23:01:04

it must be relying on implementation details of mainstream consumers

noisesmith23:01:22

I know for a fact it breaks if the object keys are not in the expected order

hiredman23:01:07

the actual ecma standard for json (the rfc standard says something slightly different) says

hiredman23:01:13

The JSON syntax does not impose any restrictions on the stringsused  as  names,  does  not  require  that  name stringsbe  unique,  and  does  not  assign  any significance to the orderingof name/value pairs. These are all semantic considerations that may be defined by JSON processors or in specifications defining specific uses of JSON for data interchange.

hiredman23:01:53

so the syntax doesn't give meaning to order, but processors of json may

hiredman23:01:32

(and yeah, different standards have slightly different definitions of json)

andy.fingerhut23:01:37

I wouldn't be surprised if the RFC was written that way because at the time of writing, some implementations out in the wild were order-dependent

hiredman23:01:36

the above is from ecma, the rfc says something like "maybe don't depend on order, if you do you can't guarantee interoperability"

hiredman23:01:31

there is actually another json rfc for something called I-JSON where I stands for "Internet" for some reason instead of interoperability (which seems to be what that rfc cares about)

hiredman23:01:16

the I-JSON rfc says no duplicate members in json objects, and the meaning of the json (the semantics) must be order independent

andy.fingerhut23:01:29

Sounds like par for the course for such a widely used thing as JSON, to have slightly different standards/interpretations/definitions that will probably forever remain different.

JoshLemer23:01:31

I don't know maybe it is purely a personal expectation but it jus feels more natural to represent data unscrambled, all else equal. Just as, columns in excel do not scramble themselves, input fields in an html form are both labeled and remain unscrambled in order. Just seems that all else equal, you definitely would like unscrambled to be the default 🤷 sorry for belabouring the point.

andy.fingerhut23:01:13

Clojure makes it possible to do that in your own code, but given the prevalence of the use of various Clojure core functions operating on maps in many Clojure libraries, some of which can take an ordered map as input but return a normal unordered map as output, you are likely to at least sometimes, if not often, be frustrated by various libraries you use forgetting your preserved orders.

andy.fingerhut23:01:53

I am almost sure that this is not the main reason Clojure defaults to unordered maps, but specifically regarding your comment "all else equal", forgetting insertion gives the implementer of maps freedom to avoid maintaining state that insertion-ordered maps must maintain.

andy.fingerhut23:01:31

Out of curiosity, is there some programming language you have used where it defaults to maintaining insertion order? Python doesn't in its default built-in dictionaries, but I know it provides ordered dictionaries in its standard library.

JoshLemer00:01:52

Hey there thanks for the comments. I don't know of such a language but I have primarily a background in scala and even am a core contributor to the standard library, which in the most recent release, includes a new abstract collection type SeqMap, which preserves key ordering

JoshLemer00:01:14

The primary motivation for its inclusion was in order to represent things like JSON objects.

andy.fingerhut00:01:35

Also out of curiosity, would there be any kind of Scala operations in its library that could take a SeqMap, and an unordered map (assuming Scala has unordered map -- I do not know), that would return an unordered map?

andy.fingerhut00:01:53

If so, that sounds perhaps very similar to Clojure's situation -- you can opt in to an insertion-ordered map or sorted-map, but then you have to be cautious to avoid accidentally "opting out" by the choice of operations you do on them.

JoshLemer00:01:53

Generally all operations have their return type determined by the type on the left, so myHashMap.concat(mySeqMap) Will return a hashmap of all keys in both maps, with the SeqMap values overwriting the hashmap's, if that answers your question

JoshLemer00:01:46

But any method on seqmaps will return a SeqMap, for instance mySeqMap.concat(myHashMap)

andy.fingerhut00:01:23

Clojure merge is similar to concat, at least in terms of whether the returned map is sorted or not.

andy.fingerhut00:01:41

... I believe. You'd want to double-check that before relying on it in an application.

didibus03:01:39

I find as an abstraction, it helps to consider unordered vs ordered I feel, even as a tool to reason about problem.

didibus03:01:06

I was going to say, its not just about preserving the type though, sometimes its also about the implementation detail of modifications to a ordered map

didibus03:01:04

Like say you do a merge, which key should come first?

didibus03:01:51

Also in clojure sorted-map is not insertion order, it sorts the keys based on a comparator, but it doesn't remember the comparator. So even if you use it with a type preserving function which return you another sorted-map, it won't have the same sort

didibus03:01:56

array-map is insertion order

andy.fingerhut04:01:11

array-map is O(n) time for lookup, and becomes a hash-map if you add a 9th or later key to it, so pretty fragile as an ordered map. Highly disrecommended for that purpose. (9th is from memory, perhaps off by a factor of 2. It is a pretty small limit)

andy.fingerhut04:01:37

sorted-maps do remember the comparator, so not sure what you mean by "doesn't remember the comparator"

didibus22:01:39

Are you sure? Array-map seem to stay array maps no matter how many elements I initialize them with.

didibus22:01:45

I know they have slow lookup. But isn't all order preserving maps with fast lookups not just a map wrapping an array map and a hash map pretty much?

andy.fingerhut22:01:17

I guess I could have been a bit more detailed in how I said that about array-maps. You can create them as large as you want, but then if you conj or assoc a key on them, and the result would be over some small limit, the return value is a hash-map. So they are very fragile in what operations you can do on them and still have an array-map.

andy.fingerhut22:01:37

The implementation of order preserving maps in this library: https://github.com/clj-commons/ordered is, inside of the implementation, a regular Clojure map, plus a sequence (or maybe vector? I'd have to review the code to be sure) to remember the insertion order. Lookup is the same speed as for a regular Clojure map. Adding/removing keys is the time to update the Clojure map, plus a single operation on the vector to update it, too.

andy.fingerhut22:01:54

There is no array-map

andy.fingerhut22:01:50

Just checked, it is a regular Clojure map (which could be an array-map when it is small, but becomes a hash map when it grows larger, just like most Clojure maps do), plus a Clojure vector.

didibus22:01:52

I guess I meant a map where you double index the keys

didibus22:01:45

Ya you're right about array-map, for some reason I thought assoc would preserve it if you explicitly created one

didibus22:01:18

What I meant about sorted-map, is I think they don't preserve sort function on modification

didibus22:01:35

Some modification

didibus22:01:04

Might be wrong

andy.fingerhut22:01:13

If you assoc onto a sorted-map you get back a sorted-map with the same comparator function.

andy.fingerhut22:01:34

If you merge two maps, the returned type will be whatever the first one is, IIRC

didibus22:01:51

Is that true of all fns?

didibus22:01:58

Like how do you grab the sorting fn out of a sorted map?

didibus22:01:25

Say I wanted to do a reduce from one to another

andy.fingerhut22:01:27

There is no Clojure API for getting the comparator function from a sorted-map that I know of.

andy.fingerhut22:01:12

reduce from one to another what?

didibus22:01:34

Would transducers over sortedmap maintain the sort fn?

didibus22:01:44

I still the whole Clojure type preserving vs non type preserving machinery is still something I don't fully grasp

andy.fingerhut22:01:44

I am not saying that sorted-maps are the most robust of all things in Clojure in terms of what operations can return something that isn't sorted. I am saying array-maps are even more fragile than that 🙂

👌 4
andy.fingerhut22:01:18

To me, it is a bit odd that there are separate functions array-map and hash-map. It would seem perfectly reasonable if there were only one (but it would have to be called something other than map, of course)

andy.fingerhut22:01:23

only one function that constructed maps, and used the same heuristic that assoc and other operations use to decide whether the implementation returned is an array or hash map, based upon size.

didibus22:01:26

I agree. Thats why I was confused in fact. I thougg if explicitly created with array-map it would maintain the type

didibus22:01:37

Versus using the literal {}

andy.fingerhut22:01:00

It doesn't. array-map are O(n) for assoc of a new key, and that gets really, really bad

didibus22:01:53

Right, but I mean, there's a reason I would explictly use (array-map)

andy.fingerhut22:01:48

If it is for ordering guarantees, I personally wouldn't use it for that. I'd use the library linked above, because it guarantees ordering across all sizes, and assoc and many other operations, more like sorted-map

didibus22:01:17

Its a bit unintuitive. Like you said, if it didn't want me using array-maps on my own term, it could just have had a single constuctor like (dictionary) or wtv

didibus22:01:25

Anyway, thanks for the info.

andy.fingerhut22:01:32

Sure, no problem.

andy.fingerhut22:01:52

I know you answer lots of questions on various Clojure fora, so always good if such people know more 🙂

didibus22:01:05

Ya, definitely. Don't want to misguide people. I thought using array-map was type preserving on assoc and all. Which if it was, would be an okay choice for the JSON style order preserving entity representation. Since its rare in that case that you have that many keys, since they model mostly objects on payloads. But since they don't preserve type, it can't it doesn't work.

hiredman23:01:49

the ui presentation is not the same thing as internal representation

hiredman23:01:18

you can do things like swap the default printer in the repl to do things like sort maps

andy.fingerhut00:01:54

Good to know, thanks. Of course, the special case of insertion order is a key sort order that, once forgotten, cannot be "reintroduced later" at print time or any other time, once forgotten.