Fork me on GitHub
#clojure
<
2021-01-23
>
didibus00:01:39

Clojure core inlines a lot of functions, but from a quick glance, most of it seems only to avoid var indirection, its not actually computing things at compile time. Since we have direct linking now, is that really necessary?

hiredman00:01:43

the inlining clojure feature isn't really about computing things at compile time, it is about exposing information to the compiler

hiredman00:01:55

so for example for unchecked-add, when that gets inlined as the static method call with the right argument types, the compiler then recognizes the static method and then instead of emitting a call to the static method it knows it can just emit an ladd instruction

hiredman00:01:04

I think constant folding what you usually call computing static expressions at compile time

caumond12:01:48

Hi guys, I am building a DSL in my application. In some configuration use cases, some expert users will need to assemble the higher level part of our language: check the grammar, and easily modify the sentences. Do you have any advise on how to do that ? the approach and tooling. I have in mind to start to build the language, small sentences first, and when it will be done, try to search for tools to present graphically the language to end users. Do you have better way to do?

simongray13:01:49

I have been exploring this are lately, but I do not have much concrete advice. The 17th chapter of The Joy of Clojure is about making DSLs, might be of interest to you.

simongray13:01:36

I also compiled this list which you may find helpful: https://github.com/simongray/clojure-dsl-resources

caumond17:01:42

Thx simon, very interesting, even if I am still having a detailed look of all information available there, I don't think there are any resource to present the dsl to the end user, did I miss something?

simongray18:01:58

Not sure what you mean.

caumond18:01:12

I mean that I dont see any lib helping an end user to build something with my underlying dsl

simongray19:01:51

Now I am really confused. How can there be a library to help a user build something with a DSL that you're making? Aren't you the one who needs to make that library??

caumond19:01:24

For sure, but I was suspecting the existence of tools to help to do that. But its true I found nothing. Intellij idea mps is an example for instance.

caumond19:01:24

They provide some functionalities to present a simple ide. As a developer you setup mps to tell what is your dsl. I was not searching for all that features, I just mention to answer your question.

zendevil.eth14:01:57

I am receiving multipart params that contain a file in my handler function in the server:

(defn upload-shot-video [req]
  (prn "uploading video")

  (prn "video is " (-> req :params))
  (prn "video is " (-> req :body))
  ( (-> req :body) ( "./resources/public/video.mov"))


  (let [filename (str (rand-str 100) ".mov")]
    (s3/put-object
     :bucket-name "humboi-videos"
     :key filename
     :file "./resources/public/video.mov"
     :access-control-list {:grant-permission ["AllUsers" "Read"]})
    (db/add-video {:name (-> req :params :name)
                   :uri (str "" filename)}))
  (r/response {:res "okay!"}))

zendevil.eth14:01:27

(-> req :params) prints:

{:_parts [["video" {:_data {:__collector nil, :offset 0, :name "55B5E1A0-8B6F-4E5D-AB7C-75CB163BC57D.mov", :type "video/quicktime", :size 1624035, :blobId "243B8CBE-15F0-4271-9BC4-5C9A9CA15FA3"}}] ["key" "VAL"]]}
(-> req :body) prints:
#object[java.io.ByteArrayInputStream 0x284cfb69 "java.io.ByteArrayInputStream@284cfb69"]

zendevil.eth14:01:59

But the file that I’m saving in my handler using

( (-> req :body) ( "./resources/public/video.mov")) 
is coming to be 0 bytes.

zendevil.eth14:01:03

How to fix this error?

zendevil.eth14:01:43

Whereas

(-> req :multipart-params)
is
{}

kwladyka15:01:48

@ps I had similar issue yesterday. Probably another wrapper already read from body and when you run upload-shot-video the body is already used. Try to move upload-shot-video to a different order. The best to run as first one, so the last one in -> macro. Then you should be able to read from body but other wrappers will fail. In general ring wrappers suc** a little. This is the worst part of ring. I started to make my own even for JSON parse - the devil is in details, but this is another topic. So if I am right there is a .reset which you can use before copy. (.reset (:body req)) will reset position of read body to the beginning. Reminder - position is on the end, because another wrapper already read from body.

kwladyka15:01:54

This is the code which I wrote yesterday. See .reset? Default wrappers don’t do that. Using it you can read body many times.

(defn ring-debug-request [handler]
  (fn [request]
    (if (#{nil "application/json"} (get-in request [:headers "content-type"]))
      (do (l/debug "request\n" (pr-str (update request :body slurp)))
          (.reset (:body request)))
      (l/debug "request\n" (pr-str request)))
    (handler request)))

kwladyka15:01:05

In general there are conflicts between wrappers, because of that, so the order of wrappers really matter.

zendevil.eth15:01:23

@kwladyka wrapping (-> req :body) with .reset gives nil

kwladyka15:01:20

Try

(println (slurp (:body req)))
vs
(.reset (:body req))
(println (slurp (:body req)))

zendevil.eth15:01:54

@kwladyka Before .reset I’m getting “”

zendevil.eth15:01:03

After I’m getting “{\“parts\“:[[\“video\“,{\“data\“:{\“size\“:2649685,\“blobId\“:\“7296F767-8880-4E69-8933-C386071C651C\“,\“offset\“:0,\“type\“:\“video/quicktime\“,\“name\“:\“56970ABA-F1A6-4BF9-A7AB-86AF895EDB5F.mov\“,\“collector\“:null}}],[\“key\“,\“VAL\“]]}”

zendevil.eth15:01:26

But after the copy operation, the saved video is just 216 bytes

zendevil.eth15:01:54

suggesting that something else is being copied to the file. Perhaps the map and not the file.

zendevil.eth15:01:46

I opened the video.mov file in textedit and got the following:

{"_parts":[["video",{"_data":{"size":2971246,"blobId":"D002459C-47C5-4403-ABC6-A2DE6A46230A","offset":0,"type":"video/quicktime","name":"DCDE604A-954F-4B49-A1F9-1BCC2C2F37BC.mov","__collector":null}}],["key","VAL"]]}

zendevil.eth15:01:09

Whereas I want to save the actual data in the video key to the file

kwladyka15:01:19

Unfortunately I don’t have experience with uploading files to server. Sorry I can’t help more here.

zendevil.eth15:01:54

The issue isn’t solved yet

noisesmith19:01:16

as I pointed out the other day, the wrap-multipart-params middleware uses a disk store, you can use the info in the request map to find that file in the disk store, or reconfigure (or stop using) the middleware in order to handle the data yourself instead see the docs for :store here https://ring-clojure.github.io/ring/ring.middleware.multipart-params.html

zendevil.eth03:01:57

It says This adds a file key to the :params map of the request where :tempfile is a http://java.io.File object that contains the uploaded data. But my request map doesn’t have the tempfile key in (-> req :params)

zendevil.eth03:01:30

furthermore, the multipart-params key contains the empty string in place of the file

jaide17:01:52

The other day I was thinking I’d love a Clojure that compiles to PHP so I could use it on dirt cheap web servers, fortunately some contacts suggested using a cgi-script might be close. Looking into it, my cheap bluehost account does support cgi so it indeed may work. Has anyone tried setting that up or seen any articles on it? Currently I’m thinking it might work with Graal VM or Babashka but I’m not sure how to get the binary on there. Just SFTP up the binary?

didibus03:01:09

I don't think you can use it with shared hosting though, cause I don't think you can setup the scgi server on shared hosting.

jaide05:01:55

Seems like some cgi support is offered but not sure if that applies here 😅

didibus05:01:16

No, because bluehost would need to have Clojure installed, and their webserver should be setup to know how to call Clojure scripts

jaide05:01:49

I’ve registered .clj to run as a cgi-script. Could I specify a shebang #!../bin/babashka to run a script through that?

didibus05:01:16

Yes, if they installed babashka

jaide05:01:34

Couldn’t I just upload the binary and point to it in the shebang?

didibus05:01:18

Possibly... I think it might depend on their web server. Like if it specifically handles certain shebangs, or just delegates to the OS for it

didibus05:01:26

You should try, that be sweet if it works

didibus05:01:14

And let me know

didibus06:01:20

Babashka just added hiccups too: https://github.com/babashka/babashka/commit/c4bb42df3e1a8ebcc27fee3a8886137d2f021b6e So that be a quick way to render html with it as a CGI script

jaide06:01:15

Oooh giving it a shot now. Thanks!

jaide06:01:34

$ ./bb
./bb: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./bb)
./bb: /lib64/libc.so.6: version `GLIBC_2.15' not found (required by ./bb)
Damn. I take it there’s no way to deal with that? Hypothetically what would I need to do?

didibus06:01:54

Yes there is

didibus06:01:29

Go here: https://github.com/babashka/babashka/releases and download the one that says: babashka-0.2.7-linux-static-amd64.zip

didibus06:01:16

If you use the statically linked binary instead, it'll fix that error

jaide06:01:34

Oh shittt! Thank you

jaide06:01:24

Can run a repl with that on server so that’s progress.

jaide06:01:06

Getting a 500 error though when visiting my clj script or my perl test. I’ve never used a cgi-script so learning on the fly here 😆

jaide06:01:51

Script runs locally via shell

didibus07:01:01

Hum, the bluehost docs says that 500 error can be you have the path wrong

jaide07:01:49

I looked at that too, but that can’t be the case given I can do ./cgi-bin/test.clj from shell and it works

jaide07:01:55

And the permissions are 755

didibus07:01:39

Might just be what you return doesn't meet the CGI spec? Can you have the script spit some file somewhere? So you'd know if it ran, but returned something that didn't jive with the server?

jaide07:01:39

I think you’re right, found a perl cgi-script example and that is working now. Only notable difference is that the return value was set to 1 or maybe it’s the exit code?

#!/usr/bin/perl

print "Content-type:text/html\r\n\r\n";
print '<html>';
print '<head>';
print '<title>Hello Word - First CGI Program</title>';
print '</head>';
print '<body>';
print '<h2>Hello Word! This is my first CGI program</h2>';
print '</body>';
print '</html>';

1;

jaide07:01:02

Nevermind it’s the content-type header

didibus07:01:21

> Every CGI script must print a header line, which the server uses to build the full HTTP headers of its response. If your CGI script produces invalid headers or no headers, the web server will generate a valid response for the client -- generally a 500 Internal Server Error message.

jaide07:01:54

💥 it works 🚀 !!!!

jaide07:01:49

Thanks very much for your help

didibus07:01:19

Ok, that's pretty cool. Worthy of a blog post in my opinion. I think there's a lot of people learning and they can't afford a VPS, it be nice if they could mess with Clojure on a shared hosting like that.

didibus07:01:04

What's the Clojure code for the script that worked?

jaide07:01:34

#!/home1/<username>/public_html/clj-test/bb

(println "Content-type:text/html\r\n");
(println "<h1>Hello I'm running from Clojure</h1>")

didibus07:01:06

Cool, ya, pretty easy then, just like with Perl.

jaide07:01:24

Should probably move the executable out of the public_html folder though, but yeah pretty simple really

didibus07:01:00

Now bb does take up 22mb of your hosting though 😛

jaide07:01:34

Not seemingly a problem here ¯\(ツ)

didibus07:01:08

Ya, pretty neat. That approach is probably the easiest way to teach people how to make websites for sure haha

jaide07:01:38

Agreed! The goal for tomorrow is to get it talking to a postgres db and if that works I definitely want to write an article.

4
borkdude17:01:41

@jayzawrotny I'm using a tiny PHP CGI script on a server which redirects a request from nginx to clj-kondo and the output is directly returned to the front-end. https://clj-kondo.michielborkent.nl/ It just runs on a VPS. Not sure about blue-host, never heard of it

jaide18:01:31

Bluehost is in that category of bulk, shared hosting which tend to be significantly cheaper than VPS. The downside is there isn’t root access, or access to apache beyond .htaccess files. That’s pretty close to the setup I’m aiming for. I setup a handler so that if visiting a .clj file it would run it as a cgi script. So if I can get that targeted script to run it through a binary that understands it, that might work out.

holmdunc18:01:21

In a Clojure standalone executable script (i.e. a single file with a #!/usr/bin/env clojure shebang at the top), how can we access the path of the script itself? (like $0 in Bash). It's not in *command-line-args* (that's like $* aka $1 $2 $3 ... in Bash).

kwladyka18:01:28

#!/usr/bin/env clojure is it something official and common? Do we have “bash” scripts in Clojure heh. I didn’t hear about that.

holmdunc18:01:23

Of course it's possible. All languages that are good citizens of Unix support shebangs!

Mno18:01:56

Well if I understand correctly that header would tell the shell that this file should be passed to the clojure executable, which I've never tested.. But I have done something similar with #babashka which is used for scripts due to its very fast start times. So in a way yes?

kwladyka18:01:10

But I guess it has pretty slow cold start right?

Mno18:01:39

Normal clojure yes, babashka no (at the cost of some limitations)

R.A. Porter18:01:59

Yeah. For that use case, I’d definitely use #babashka. In that case, you can get the run script with (System/getProperty "babashka.file")

p-himik18:01:40

#! is a comment in Clojure. And yes, it tells your shell (if it understands shebangs) to execute the file using the specified interpreter. Just a single (println "hello") in such a file takes less than a second on my PC.

holmdunc18:01:54

The shebang causes e.g. /usr/local/bin/clojure to receive the the path to the script as its first argument. The question is whether it gets stored in a user accessible way

Mno18:01:08

Ohh so that's how that works.. Thanks that's a great bit of info!

👍 3
kwladyka18:01:58

babushka what a name heh 🙂

p-himik18:01:09

ba-*bash-ka :)

kwladyka18:01:24

Thanks, I was curious

holmdunc18:01:21

I know about and have used Babashka. That's besides the point, my question is about how to get this info in "real" Clojure :thinking_face:

holmdunc18:01:05

That's it.. thanks!!

holmdunc19:01:00

This kind of standalone script can even specify dependencies directly in the shebang, that will get auto-fetched if needed (`#!/usr/bin/env clojure -Sdeps {:deps{...}}}` (just don't use any whitespace inside that map)). Obviously the ever-present caveat is slow startup time.

p-himik19:01:51

If you use white spaces in the map, you should be able to just use single quotes around it.

p-himik19:01:21

Like #!/usr/bin/env clojure -Sdeps '{:deps {...}}}'

andy.fingerhut19:01:49

You can even make the beginning of the bash script a bit more involved, if you want, e.g.: https://github.com/jafingerhut/dotfiles/blob/master/bin/clj-simple-single-file-script

p-himik19:01:47

M-m-magic! :D

holmdunc19:01:33

I think quoted strings are a shell thing. Processing the shebang is a kernel/OS level thing, no?

p-himik19:01:54

Shebangs are a shell thing. Different shells might need different things.

holmdunc19:01:22

I mean the processing of the shebang. Shebang-having executables can be executed without there being any shell involved at all, as I understand it

p-himik19:01:34

Actually, I'm absolutely wrong - please disregard my comment above.

p-himik19:01:49

Yeah, man execve describes it. I have no clue what made me think #! was a shell related thing.

👍 3
holmdunc19:01:30

I remember seeing that wild Bash/Clojure hybrid approach Andy. If the straightforward shebang couldn't support deps then maybe that would become a hit!

andy.fingerhut19:01:13

Yeah, the one I linked to might very well be more fragile/magic than one would typically want to use.

Michael Thurmond19:01:13

Has anyone integrated aws cognito with a clojure server for authorization and authentication?

borkdude20:01:27

In clj-kondo, for analysis purposes, we want to have a list of all occurring keyword including their surface form, alias, etc. For some keywords it might also be handy to have something like :def true which indicates if they were used to define something in a registry (think spec and re-frame), so you can "navigate to definition". But the "keyword definition" sounds a bit weird, maybe. What would you call this?

noisesmith20:01:39

keyword registration?

noisesmith20:01:21

arguably that could also include things like watch keys used with container watchers

borkdude20:01:07

Maybe :registered-by clojure.spec.alpha works. :registered-by clojure.core/add-watch

noisesmith20:01:20

the second one might not be worth implementing, but it's a tuple of keyword / container, since the same watch key can be used elsewhere

noisesmith20:01:39

I don't know that watches are so ubiquitous in program logic that it's even making a feature there

borkdude20:01:54

probably not. we are thinking spec and re-frame as the most prominent examples

noisesmith20:01:22

in re-frame, can the same key be used in different ratoms?

noisesmith20:01:56

I know reagent lets use use multiple ratoms, but I haven't dug that deep into re-frame

phronmophobic20:01:15

re-frame has subscriptions registered to keywords. afaik, subscriptions are still only global

or20:01:22

@borkdude: thanks for clj-kondo. I can't even remember what life was like without it. Or maybe don't want to.

❤️ 17
teodorlu21:01:41

Idea, looking for criticism / feedback / questions. > Object orientation conflates the notion of "namespace" with the notion of "hierarchy". In Clojure, not so much. Namespace qualified keywords are not the same as nesting in maps. In javascript, hierarchy is often used for namespacing.

teodorlu21:01:45

I've been thinking about Rich's The Language of the System for a long time. I'm still not sure what he meant. But I feel like I'm getting closer. https://www.youtube.com/watch?v=ROor6_NGIWU

noisesmith21:01:45

that feels a bit off to me, but I'm not sure why yet

3
teodorlu21:01:09

@noisesmith I'd be curious to hear why it feels off. Apologies for not having a more precise question.

teodorlu21:01:31

I've often felt a weird, unsettling feeling when I encounter a library API that's a hierarchy / nested list. Examples include #malli and #hiccup. Both allow you to put arbitrary nesting in your input data. But if you allow arbitrary nesting in your input, how do you provide extensibility?

noisesmith21:01:08

in clojure a namespace as an object, is a first class scope you can reflect on, namespaced symbols are prefixed so they can be specific to one of those first class scopes for lookup, namespaced keywords can be correlated to namespaces for convenience but the namespace is an opaque token (there's no heirarchy of namespaces, as scopes or as prefixes, the relations are purely textual)

Mno21:01:22

There's still a bit of hierarchy in namespaces..right? foo.bar and foo.baz are sibling namespace (common parent foo), and foo.bar.bing is would be a grandchild.. Right?

noisesmith21:01:34

they are not siblings in any meaningful way

noisesmith21:01:58

the coincidence / alignment helps human readers, but has no meaning in terms of lookup or binding

noisesmith21:01:05

you could just use foobar and foobaz as names, without the dots, without changing the meaning of your program

noisesmith21:01:32

(unless you are using gen-class, in that case you need at least two segments)

noisesmith21:01:55

or you could make them foo.bar and baz.quux

Mno21:01:55

Ah ok. So there's no functional hierarchy, it's just there mostly because of folders and some Java inheritance?

noisesmith21:01:06

the folders are just for lookup

noisesmith21:01:12

there's no effect on inheritence

noisesmith21:01:51

(except for the weird rules for classes with a single segment, solved by having at least two segments if using gen-class)

Mno21:01:55

(I meant because of the gen class thing, not becuase of type inheritance)

noisesmith21:01:20

yeah, there's no semantic relation, it's just lookup coordinates

Mno21:01:21

Haha cool thanks for the info, very interesting

noisesmith21:01:28

but this seems pretty far off from the initial question

😄 6
teodorlu21:01:51

I'd say it's tangential!

teodorlu21:01:52

> in clojure a namespace as an object, is a first class scope you can reflect on, namespaced symbols are prefixed so they can be specific to one of those first class scopes for lookup, namespaced keywords can be correlated to namespaces for convenience but the namespace is an opaque token (there's no heirarchy of namespaces, as scopes or as prefixes, the relations are purely textual) @noisesmith not sure if I understand precisely what you mean. I guess the vagueness of my non-question might be to blame. I agree with all you say in terms of Clojure semantics. I guess a more question-like formulation is: > is there a tangible description of the principle behind providing namespace qualified keywords? Not sure that helps 🙂

noisesmith21:01:33

OK - what I was attempting to clarify there (and didn't quite achieve yet), is that a namespace means at least two things in clojure

3
noisesmith21:01:44

1. it's a literal object you can reflect on and resolve symbols in 2. it's the part of a symbol that controls which namespace it is looked up in 3. it's the part of a keyword that plays an analogous but different role as the namespace of a symbol

👍 3
3
thinking-face 3
teodorlu21:01:39

> There's still a bit of hierarchy in namespaces..right? foo.bar and foo.baz are sibling namespace (common parent foo), and foo.bar.bing is would be a grandchild.. Right? @UGFL22X0Q Right! I'd say it's a bit like a hierarchy. I like to think of my keys as organized in terms of a hirerarchy, that helps me think about it. But still, there's this stragely non-hierarchical part. I mean, why use namespace qualified keys when we could just nest? Why {:person/name "mno"} over {:person {:name "mno"}} ? I don't have a good answer for that. But I'm looking for one.

noisesmith21:01:53

'clojure.core/+ is resolved by the evaluator to the token under + in ns clojure.core

noisesmith21:01:00

that's a namespacing operation

teodorlu21:01:26

@noisesmith good point about actually looking at the primitive operations you have on a namespaced keyword / namespaced symbol vs for example nested maps ✔️

noisesmith21:01:27

@U3X7174KS that "sibling" relationship is only recognized by humans and has no effect on the compiler

✔️ 6
teodorlu21:01:16

hmmm. So, @noisesmith, would you say that the function of a namespace is to map an identifier to a value?

noisesmith21:01:29

so it's closer to case 3 in my enumeration above - it's a token that helps humans categorize things, all the compiler needs is that two uses of it use the same token, which always stands for itself

✔️ 3
noisesmith21:01:48

well - that's why I made the enumeration above: the function of (1) namespace is to hold a map from symbols to vars holding values (along with other conveniences like shortands created by refer / :as)

noisesmith21:01:07

(2) is loosely related to that, (3) is loosely related to (2)

teodorlu21:01:07

mmmm, okay. I apologise for bad problem formulation here. Good enumeration: > 1. it's a literal object you can reflect on and resolve symbols in > 2. it's the part of a symbol that controls which namespace it is looked up in > 3. it's the part of a keyword that plays an analogous but different role as the namespace of a symbol I'm more interested in our mental notion of a namespace than the literal clojure object that you can reflect on, ask for public vars, etc. So I think I'm most interested in your (3).

noisesmith21:01:44

you go from "thing that can hold other things, we know which thing it is partly based on which thing holds it" (two (1) namespaces can hold the same symbol with unrelated values), to "thing that tells you where to look a symbol up, and what to look up, in one token" (a (2) namespace is a helper object to using values from (1) namespaces), to finally "thing that is qualified by a prefix to prevent collisions, but only stands for itself"

3
👍 3
noisesmith21:01:02

in that case - it's a prefix, start and end of story

noisesmith21:01:55

for human convenience we can pretend they have a hierarchy, and expect :a.a/b and :a.b/b to have some relation

seancorfield21:01:10

I feel like Stuart Halloway gave a good example on the namespaced keyword: :github/id and :twitter/id in a "person" hash map.

3
👍 3
noisesmith21:01:03

yeah, coming back to pragmatics - in js that would likely be {:person {:id ...} :twitter {:id ...}} which gets closer to the initial question I think

✔️ 3
teodorlu21:01:31

So why do we prefer the namespaced keys?

noisesmith21:01:36

but it would just as often be {twitter_id: ... :person_id ...} - that contains no less info

3
noisesmith21:01:53

in fact, repeated prefixes on symbols are a sign you want a namespace

👍 3
noisesmith21:01:03

(in the abstract sense, not the clojure impl sense)

✔️ 3
noisesmith21:01:34

namespaced keys make operations simpler, and data literals easier to read and work with

teodorlu21:01:37

I like the namespaced keys better in a sense, but I can't really say why. Perhaps it's because names exist in a domain, and we can make the domain explicit. And that "name spaces" / "domains of names" is a different notion than "contains".

seancorfield21:01:58

Because :foo_bar_baz doesn't tell you whether it is a Baz in a FooBar context or a BarBaz in a Foo context

💯 3
noisesmith21:01:21

I bet if I had more coffee I could articulate the relationship between namespaced keys vs. nested maps and normalization of data

teodorlu21:01:20

> I bet if I had more coffee I could articulate the relationship between namespaced keys vs. nested maps and normalization of data I'd happily ship you a few liters of coffee if you could solve this once and for all!

teodorlu21:01:35

> Because :foo_bar_baz doesn't tell you whether it is a Baz in a FooBar context or a BarBaz in a Foo context @seancorfield that's a very good point. Crisp separation between space/context/domain and name.

noisesmith21:01:26

{:foo {:id ...} :bar {:id ...}} has a reification of "ownership" into the tree of the data - all you need to do is select a key to get all :foo related things, while {:foo/id ... :bar/id ...} isn't reifying ownership, it's telling a human reader either where the item came from, or who cares that it is there

3
seancorfield21:01:27

And it's also clear (to me, at least) that there's no "containing" relationship here - no nesting.

✔️ 3
seancorfield21:01:36

(my slow responses are due to typing on my phone)

✔️ 3
teodorlu21:01:44

I'm curious about how you've been selecting your namespace qualified keys for jdbc.next. I feel like naïvely using ::key often gives me way more nesting than I really wanted; "Good: crisp vocabulary with little nesting, bad: throwing deeply nested keywords around".

teodorlu21:01:47

I also sense that the Clojure core team has some very good taste in designing namespaces. For instance, there's very little nesting in clojure.core. Perhaps there's something Rich just isn't telling us, saving it for the next hit once conferences open up again 😅

teodorlu21:01:07

> {:foo {:id ...} :bar {:id ...}} > has a reification of "ownership" into the tree of the data - all you > need to do is select a key to get all :foo related things, while  {:foo/id ... :bar/id ...} isn't reifying ownership, it's telling a human reader either where the item came from, or who cares that it is there (edited) @noisesmith would you say that "reification of "ownership" into the tree of the data" is bad? If so, why?

noisesmith21:01:44

I would say it's something that OO code does even when it makes code worse

3
seancorfield21:01:51

Only :next.jdbc/count comes to mind @U3X7174KS

👍 3
noisesmith21:01:21

if :foo and :bar were two things that were meaningful to combine in operations, I'd rather they be sub maps that I could easily select and work on

👍 3
noisesmith21:01:04

if they are actually segregating related data, then I'd rather they use a namespacing prefix and merge into the top level single item

👍 3
teodorlu21:01:32

> Only :next.jdbc/count comes to mind @U3X7174KS Observation: that's a single namespace (next.jdbc) and a set of names; a vocabulary under that namespace.

noisesmith21:01:47

for example - if there's no use case for combing twitter/id and user/id outside the fact that they belong to one entity communicating with the API, then there's no reason to leave any sub-maps, having both as keys in one map is an advantage

3
teodorlu21:01:48

@noisesmith got an example of when you'd choose "combination into sub maps" and when you'd choose "namespacing prefix and merge into the top level single item"?

noisesmith21:01:21

but if what I had was user/id and friend/id - those are parts of two peer objects that I could interchange and combine differently

3
teodorlu21:01:24

would the default be namespaced keywords, and combination only if it provides benefits?

noisesmith21:01:39

and it makes more sense in that case to have {:user ... :friend ...} - as each sub map contains the same data

👍 3
noisesmith21:01:04

@U3X7174KS I guess my question is "what operations are useful", once I know that I know whether to namespace or to nest. nesting is good for modular things that are meaningful to rearrange

✔️ 3
👍 3
💯 3
teodorlu21:01:19

> but if what I had was user/id and friend/id - those are parts of two peer objects that I could interchange and combine differently Yeah, that feels a bit weird to me. I'd reach towards {:me {:id ...} :friend {:id ...}} rather than using namespaced maps. But I can't really explain why.

noisesmith21:01:59

right - that's what I was trying to say, if they are meaningfully interchangible, then it's useful to leave them segregated

👍 3
teodorlu21:01:01

> @U3X7174KS > I guess my question is "what operations are useful", once I know that I > know whether to namespace or to nest. nesting is good for modular > things that are meaningful to rearrange In type land, I would say that you should keep the same shape of your data when you can, so that you can write a single function that works on all the different instances. "typeable".

seancorfield21:01:07

Nesting makes sense to me when the submap can stand alone as an entity.

👍 3
✔️ 3
💯 3
teodorlu21:01:38

Very good point about analyzing "use maps vs use namespaced keywords" by seeing how one would design a good API of operations :thumbsup:

noisesmith21:01:40

@U3X7174KS I think a big mistake in by-the-book OO code is the reflexive treatment of everything as a model or domain entity - some things are just useful groupings that simplify code

3
noisesmith21:01:03

an OO programmer would look at your {:user/id ... :twitter/id ...} and ask "what is this entity called, and what does it represent"

👍 3
teodorlu21:01:10

@noisesmith mmmm, yeah. And that's the cases where you would like to limit nesting. Because a single level of (possibly namespaced) keys is a very simple model. Merge, etc.

teodorlu21:01:38

> an OO programmer would look at your {:user/id ... :twitter/id ...} and ask "what is this entity called, and what does it represent" And we would say "don't name it, we need it just here". Gotcha.

noisesmith21:01:50

did you mean "simple model"? - or I guess, what do you mean by "single model"?

teodorlu21:01:15

Simple. Thanks ✔️

seancorfield21:01:15

OK, at the big machine now 🙂 So the other area where namespaced keywords come into next.jdbc is in result sets: the qualifier matches the table where the column came from so if you join two tables, you get a single collection of data that you've assembled, and the qualifiers indicate the table which also ensures you can mix and match data from multiple tables without any name conflicts (e.g., :person/id and :address/id when joining the person table to the address table).

👀 3
3
teodorlu21:01:54

mmm. So the domain design you work you do, in putting names (columns) into namespaces (tables), can be reused in Clojure and SQL, rather than having two different models that you have to keep in sync. If that's what you meant?

seancorfield21:01:02

For person/address, you may well want :address as a full submap in your person hash map -- but that's about whether you want to treat address as a distinct entity, rather than just for (say) a mailing label (where you want :person/first-name :person/last-name :address/street :address/state :address/zip)

3
💯 6
teodorlu21:01:39

> address as a distinct entity, rather than just for (say) a mailing label (where you want :person/first-name :person/last-name :address/street :address/state :address/zip) I really like that formulation. It looks similar to what @noisesmith said about "convenicence of grouping" and "collecting stuff together to provide shared operations over it".

seancorfield21:01:43

I think if the early examples of Clojure code had just used qualified keywords out of the gate instead of simple keywords, we wouldn't even be having this conversation.

3
teodorlu21:01:06

That leaves me curious about whether you can know in advance whether you are going to treat address as a distinct entity, and when you just want a set of properties. Do you decide on one or the other? Or do you create functions assemble-address and splice-address or similar?

seancorfield21:01:34

We've just gotten used to simple keywords and now we're starting to look at qualified keywords as new/strange/different -- or having some inherent "meaning" in and of themselves.

3
noisesmith21:01:50

that would usually be the boilerplate layer between application code and the db, but next.jdbc can cut a bunch of it out

👍 3
teodorlu21:01:54

> I think if the early examples of Clojure code had just used qualified keywords out of the gate instead of simple keywords, we wouldn't even be having this conversation. @seancorfield are good open codebases in the wild written like this that you would recommend reading for education purposes?

seancorfield21:01:23

With SQL, I could just select * from address where id = ? and get an address "entity", if I want it separately.

seancorfield21:01:14

@U3X7174KS No, there are very few applications out there in OSS land -- and libraries almost always take an inherently different style/approach.

✔️ 3
teodorlu21:01:18

Gotcha, thanks.

seancorfield21:01:47

In our production codebase, we're using qualified keywords a lot more now than when we started a decade ago (see my comment above about if Clojure had used qualified namespaces from the get-go in examples).

👍 3
teodorlu22:01:05

I remember having a look at #cljdoc, being surprised about the taste that I perceived to be put into that. But I don't remember seeing anything about namespace qualified symbols in particular.

seancorfield22:01:42

And we pick a qualifier that signifies how "unique" the scope of data needs to be. :wsbilling/* for example meaning it's associated with the World Singles "billing" subsystem.

✔️ 3
seancorfield22:01:26

But a lot of the time, data is just internal to one subsystem and we just use the DB table name as a qualifier (in our modern code -- legacy code tends to use simple keywords).

👍 3
✔️ 3
teodorlu22:01:39

And then you can document the semantics of :wsbilling/thing and provide a single source of truth for that?

seancorfield22:01:50

Via Spec, yes.

teodorlu22:01:11

Ah, of course.

teodorlu22:01:03

I'm curious about how much ... "prose", "written description about the intention of this attribute" that follows each attribute's spec documentation, then.

teodorlu22:01:25

peeking at the Spec guide

seancorfield22:01:35

We have a bunch of web-app-related middleware and logic that tends to use :ws.web as a qualifier (so we're not terribly consistent about dotted or non-dotted). But it's just enough to partition the data if it all ends up in one map.

👍 3
👀 3
seancorfield22:01:07

We don't use much "prose". We just try to use expressive names.

👍 3
seancorfield22:01:54

I'm big on docstrings, my colleague not so much. But we discuss naming issues in PR reviews, and as long as we can follow the intent of each other's code via names, that's good.

👍 3
teodorlu22:01:32

That makes sense. I guess a large problem is solved in just being consitent in the use of :wsbilling/thing, so that you can grep around and know that it's the same thing in different places.

seancorfield22:01:50

There's really very little comments in our code either.

3
seancorfield22:01:18

When I write OSS, I'm much more expansive about that: I'm writing code -- and docstrings and comments -- for a lot of readers out there.

teodorlu22:01:40

mmm. Higher fanout / read-to-write ratio?

seancorfield22:01:21

Libraries are also more general so there's a lot less "context" than application code which always has your business domain as context.

👍 3
seancorfield22:01:13

I believe Clojure 1.11 will give us a lightweight way to create namespace aliases so that should help encourage qualified keywords that no longer have to be tied to namespaces in people's minds (they don't have to be tied to namespaces today but it's a bit clunky to create an alias that has no code namespace behind it).

❤️ 3
seancorfield22:01:05

I think that will help make qualified keywords more appealing since folks can set up some alias easily and then use ::foo/bar in their code instead of :foo-is-a-long-name/bar

❤️ 3
teodorlu22:01:21

I would love for that to happen. It feels like a hack to have to construct the foo-is-a-long-name.clj file just to be able to alias the namespace.

teodorlu22:01:34

Thanks for a wonderful discussion, @noisesmith and @seancorfield. Really helpful to hear your insights. I'm signing off, talk to you later! 👋

teodorlu22:01:47

Thanks for a wonderful discussion, @noisesmith and @seancorfield. Really helpful to hear your insights. I'm signing off, talk to you later! 👋

teodorlu22:01:38

bah, I didn't mean to post to channel. Fat fingers, sorry for the spam.

seancorfield22:01:51

@U3X7174KS You don't need a source file, you can do (alias 'foo (create-ns 'foo-is-a-long-name))

💡 3
seancorfield22:01:20

From our codebase at work:

(alias 'm  (create-ns 'ws.domain.member))
(alias 'mu (create-ns 'ws.domain.member.update))
(alias 'mv (create-ns 'ws.domain.member.validation))
🙂

❤️ 3
seancorfield22:01:45

But that is the "hack" that you need a namespace in memory for the alias right now.

👍 3
katox22:01:38

What do you do about printing aliased ns in maps?

katox22:01:23

In code maps with aliased kws look tidy when printed for debugging do you just sift through that noise?

didibus05:01:54

@U0KD0N5T5 there's a short syntax:

#:foo {:bar 1, :baz 2}
;; is the same as:
{:foo/bar 1, :foo/baz 2}

didibus05:01:28

@U3X7174KS A keyword namespace can have hierarchies in Clojure, but it is up to the application to decide of that scheme (if any) for itself. Clojure itself does not have the concept of hierarchies for keyword namespaces. What that means is that you do not find something by following the trail of . inside a keyword namespace, and you also do not find something by following the namespace somewhere to find the name of the keyword inside it.

(def m
  {:foo/bar {:biz/baz 2}
   :foo/baz 3})
Here you don't navigate the map using the keyword namespaces. For example, you don't do:
(-> m :foo :bar :biz :baz)
But if it was OO, you would, you'd have the Foo object which contains bar which contains the Biz object which contains baz. And you'd do:
m.bar.baz

3
didibus05:01:51

What that means is that :foo/bar is not necessarily inside the :foo map, or any other thing, there is no concrete thing that is a :foo where :bar is found inside. You could not have had a namespace mechanism built in, but having one both instill good practice to people, and also means there's no conflict in parsing the namespace from the name for which you need to come up with a solution for clashes. There are reasons to want to use namespaces in your app. Key conflicts is one. But as an app developer, you can also leverage it for other useful things. Could be used for defining types, or context, and could even be used for hierarchy if you want.

didibus05:01:33

Now namespaces (non keyword), so like *ns* are scoped in Clojure, but not hierarchical, though you can think of a scope as almost a hierarchy of one. Its just hierarchical generally implies nesting and sub-nesting. So you do use the namespace to look up the thing for the given symbol name from somewhere, the where being the namespace itself. But if you think of where those are, they are all flat:

{'foo.bar
 '
 'foo
 'fizz.buzz
 'fizz.buzz.beer}
So even though those refer to a real place where the Var of the symbol will be found, they don't nest, so when you have fizz.buzz it is not the case that buzz is inside fizz. Its just a name for the space where the Vars of those symbols will be found, not a hierarchy of things nested in one another. So the symbols are like nested inside the namespace, but namespaces don't nest themselves inside others, they are flat, and not hierarchical once again. And finally symbol namespaces when the symbol is used as a value itself, like you'd use keywords, where that's exactly the same as for keywords.

didibus05:01:27

Ok, last thing, maybe this seems like splitting hair. But actually, it makes most sense if you start from OO, how do you do this in OO?:

{:user/id 123
 :order/id 234}
Well, you can invent your own namespacing scheme and do:
Purchase {
  user_id;
  order_id;
}
new Purchase(user_id = 123, order_id = 234)
Which is just you re-inventing an ad-hoc namespacing mechanism. Or you use OO's hierarchies:
User {
  id;
}
Order {
  id;
}
Purchase {
  user;
  order;
}
user = new User(id = 123)
order = new Order(id = 234)
new Purchase(user = user, order = order)
But now you need to get the thing by navigating the hierarchy:
purchase.user.id

💯 3
teodorlu10:01:51

> So the symbols are like nested inside the namespace, but namespaces don't nest themselves inside others, they are flat, and not hierarchical once again. @U0K064KQV good point.

teodorlu10:01:44

Very good example. "separation of concern between namespacing and hierarchies prevent reinventing namespacing in terms of hierarchies, or ad-hoc conventions for namespacing." (my interpretation) https://clojurians.slack.com/archives/C03S1KBA2/p1611466587246700?thread_ts=1611435881.204400&amp;cid=C03S1KBA2

Andy Wootton19:01:43

Thanks to all contributors to this fascinating thread. I have started to learn Clojure for an idea about moving away from trees towards graphs, so I've been a little nervous about namespaces appearing to be a hierarchy. I've found this very helpful. I think I understand now that a syntactic hierarchy doesn't have to imply a semantic hierarchy, though I'm not sure I can trust myself to always remember that 'in the heart of battle'.

💯 3
❤️ 3
katox20:01:08

@U0K064KQV that's not the point - the point was what do you do when you want to print/pprint/debug a map full of aliased definitions. Using @seancorfield's example ns:

{::m/a 1
 ::m/b 2
 ::mu/c 3
 ::mu/d 4
 ::mv/e 5
 ::mv/f 6}
=>
{:ws.domain.member/a 1,
 :ws.domain.member/b 2,
 :ws.domain.member.update/c 3,
 :ws.domain.member.update/d 4,
 :ws.domain.member.validation/e 5,
 :ws.domain.member.validation/f 6}

✔️ 3
katox20:01:49

unlike the original short syntax it is always lengthy and the lengthy namespaces are mostly noise

teodorlu14:01:46

@U0KD0N5T5 Good point. I hit this problem previously trying to "push" qualified keywords on a frontend developer. His reply was simply, "I don't like this. It's tedious, longer to write, and takes more time to eyeball". He was right, really. Which makes me kind of curious about the Juxt guys' approach to JSON (#apex (doesn't exist yet?), #crux). @U050CTFRT called Clojure "JSONScript" on a Clojure conference in Berlin. That stuck with me.

didibus22:01:10

@U0KD0N5T5 Ya, when you have a map with mixed namespaces, only one can be shortened:

#:ws.domain.member{:a 1,
                   :b 2,
                   :ws.domain.member.update/c 3,
                   :ws.domain.member.update/d 4,
                   :ws.domain.member.validation/e 5,
                   :ws.domain.member.validation/f 6}
Evan alias won't help, because you're serializing them, you don't want a contextual alias in that case. > unlike the original short syntax it is always lengthy and the lengthy namespaces are mostly noise I think this is on the app developer though. Like why you use a feature you don't benefit from? Its like if you made all your maps records for no good reasons, or if all your functions were multi-methods even if they only have a single defmethod.

didibus22:01:14

Well, alias do help in the usage of these maps in the code, since you can alias each of these to a shorter alias, but if you care about the payload size, or the verbosity of it when say logged or something like that, alias won't help

didibus22:01:18

I guess in theory, Clojure could come up with a data representation for it, maybe it will in the future. Actually, having a #:[] might be useful. Or something like that:

#:ws.domain.member {:a 1,
                    :b 2,
                    #:ws.domain.member.update [:c 3, d 4]
                    #:ws.domain.member.validation [:e 5, :f 6]}
This for example. The idea is that #: defines a repeated namespace but at the same level. The reader could be made to recognize this.

katox22:01:34

@U0K064KQV yes, that's what we did - we shortened all namespaces in maps to minimum, turned off short version map printing and replaced every single ::kw and ::aliased/kw with :short/kw. And we don't have to fix all tooling all of a sudden.

didibus22:01:36

Ya, I've actually just stop using namespaces in my actual data, unless there's a need for it, like I need to protect from name conflict. But they are useful for spec and things like that.

katox22:01:28

I have no use for spec either. I don't do too many macros and for data malli is a much better fit because of coercions and swagger. But that's OT. Having consistent keys is a plus so I like namespaced keys pretty much everywhere.

didibus22:01:32

It depends very much on your use case. That's why I think people need to understand it as a feature only. Keywords can have namespaces, when they do, it doesn't mean anything special for Clojure. So its really just, if you in your app need a namespace for your keys, well you're in luck, you don't need to build your own internal namespacing scheme and all, Clojure keywords can have namespaces directly.

katox22:01:52

Aliased kws remind me very much the land of xml schemas. I doubt there is an easy solution. So to date I'd like to stop seeing aliased kws ... like completely.

noisesmith02:01:12

very literally they are the same thing: namespaces on tokens

noisesmith02:01:56

they are only going to be more pervasive now that clojure spec uses them

borkdude23:01:57

Is it possible to get a named match out of a regex somehow with core.match?

(clojure.core.match/match ["foo"] [#"f(?<a>.)o"] a)'
or do I use re-matches for this inside the pattern?