Fork me on GitHub
#beginners
<
2021-07-03
>
lw199004:07:22

Can someone help me understand why quote as a function is useful, why it exists, how often it's used?

seancorfield05:07:57

Take a look at https://cljdoc.org/d/com.github.seancorfield/honeysql/2.0.0-rc3/doc/getting-started where you can use a DSL that involves either keywords or symbols: some people like the symbol-based version because they don't have to type : everywhere -- but you need ' so that the symbol names are treated as names and not looked up for their values.

didibus07:07:27

Alright, so what is this:

"hello"
If you don't know programming, you'd tell me it's the word hello surrounded by double quotes. If you know programming you'd tell me it's a String whose value is hello But in actuality it's just a bunch of characters, they take on a meaning based on some contextual semantics. In the context of English reading, it is the word hello surrounded by double quotes. In the context of Clojure reading, it is a Unicode String whose value is hello. That's because characters surrounded by double quotes in Clojure mean that they represent a Unicode String. From this point on, lets remain in the realm of Clojure reading, so with that in mind, what is this:
:hello
It is an unqualified Keyword whose name is hello. That's because characters starting with a colon mean that they represent a Keyword. So what is this:
hello
This is a Symbol whose name is hello. That's because characters not prefixed or surrounded by anything special mean that they represent a Symbol. What is this:
("hello" "John")
This is a List with two Strings as elements. You know this because anything wrapped between open/close parenthesis represent a List in the context of reading Clojure. Now what is this:
(hello "John")
This is a List where the first element is a Symbol and the second a String. Now you might ask me, but isn't hello in this case a function? Well, no it is not, because I'm the context of reading Clojure, as we saw, this is a list of two elements, the first one is read as a Symbol, and the second as a String. But after you read source text in Clojure, you get back what you read. So after reading the text (hello "John" you now have a List of Symbol and String, this is no longer text, you hold now in memory an instance of a List object whose first element is an instance of a Symbol object and second element is an instance of a String object. Once you have an instance of a List, you can ask Clojure to evaluate this List as code. Now we are in the context of Clojure evaluation, and as you see, Clojure can evaluate a data-structure like a List, that's why people say in Clojure "code is data(structure)". Because code is what can be evaluated, and in Clojure that can be, among other things, a List data-structure. So if I were to represent this List in text once again, I would write (in the Clojure language):
(hello "John")
Now, after reading this and getting the List of Symbol and String, if I call eval on it, which Clojure will do automatically when given a source file to run, or when sending a piece of source to the REPL (where it will Read and then Eval), well what Clojure evaluation semantics will do is that they'll take the first element of any list and look it up to see if there is a function of that name and then replace it by that function, where the rest of the elements in the list will become the arguments to it. So when evaluating the List:
(hello "John")
Clojure will actually execute the hello function, now let's pretend the hello function is this:
(defn hello [name] (println "Hello " name))
That will return nil . So after evaluating our list, we get nil back. Okay, but now we have a problem, because we read a List, but as soon as Clojure evaluates it, it'll become a function (or macro) call. So what if we want the List itself? Imagine we had:
(def sum [numbers]
  (reduce + numbers))
And so I wanted to write:
(sum (1 2 3))
See the issue? When we read this, we see that we have a two element List where first element is a Symbol (sum) and second element another List. That nested List contains three Numbers. But now when Clojure evaluates this List, it'll consider sum a function, which is correct, and pass it as an argument the nested Lists, and prior to function executing, their arguments are also evaluated, so now we have a List being evaluated again, and Clojure will consider 1 to be a function and it'll call it with 2 and 3 as arguments, this will fail obviously. The problem is that since Lists represent function or macro calls when evaluated, how do you evaluate something to get a List back? And not have it be treated as a function or macro? That's where quote comes into the picture. You tell the Clojure compiler, the following form, please skip evaluating it, treat it simply for what it was read as. So now you can do:
(sum (quote (1 2 3)))
And Clojure will evaluate this where it'll consider sum a function, but this time it'll see that (1 2 3) is quoted, and so it won't evaluate it, it'll just take the List as-is and pass it to sum as the argument. It's a little meta, because of the homoiconicity, it makes it so that the textual representation for a List is the same as that for a function call, which means that you need a way to tell the compiler when it should be treated as a List or when it should be treated as a function call, and quote let's you do that, by saying don't evaluate what's quoted, treat them as-is the same as they were read.

didibus07:07:44

Now because it's annoying to wrap the List in a List calling the quote special form, Clojure also has a reader macro character for it, and so everything that is prefixed by 'is read as if it was wrapped in a List calling quote.

lw199004:07:17

The only useful example I've found so far is when you want to print an expression out, like for debugging or something like that

indy04:07:51

The single quote ' is just a macro reader that expands to quote . And these are what make "code as data" possible. Manipulating source code via macros wouldn't be possible without quoting. A lot of the usage of quoting is in macros where you're building and manipulating forms that are not going to be evaled "yet" (macroexpansion time). https://www.braveclojure.com/writing-macros/

noisesmith15:07:04

it would be possible - just very inconvenient

(defmacro tedious-infix
  [& forms]
  (cons (symbol "do")
        (for [[x op y] forms]
          (list op x y))))

noisesmith15:07:16

user=> (tedious-infix (1 + 1))
2

noisesmith15:07:43

(that's massively simplified in that it doesn't break down the x and y for subforms etc.)

dpsutton04:07:21

I think what makes "code as data" is demonstrated well with (eval (list + 1 2))

💯 3
dpsutton04:07:54

And think about what (list + 1 2) returns and how evaling a list can really do anything at all. if you're familiar with C# or Java, consider how weird it would be to eval(IEnumerable<Object>) or the like. And then recognize that (list + 1 2) is using just regular clojure functions and a regular clojure datastructure

dpsutton04:07:06

(eval (list (quote let) (vector 'a 1) (list + (quote a) 2)))

lw199004:07:05

ok so apparently atom is not a special form, and the source code for it doesn't seem to use special forms directly, but I'm guessing it uses let or var somehow if you dig deep enough, is there a simple way to see this in a repl?

indy04:07:30

It does use the new special form. An instance of the clojure.lang.Atom class will be created in this case.

seancorfield05:07:44

@U9Q7C6G4C Can you see it in the REPL? Yes, via the source function:

user=> (source atom)
(defn atom
  "Creates and returns an Atom with an initial value of x and zero or
  more options (in any order):

  :meta metadata-map

  :validator validate-fn

  If metadata-map is supplied, it will become the metadata on the
  atom. validate-fn must be nil or a side-effect-free fn of one
  argument, which will be passed the intended new state on any state
  change. If the new state is unacceptable, the validate-fn should
  return false or throw an exception."
  {:added "1.0"
   :static true}
  ([x] (new clojure.lang.Atom x))
  ([x & options] (setup-reference (atom x) options)))
nil

didibus07:07:06

I actually recommend you the read each section of the reference in order, it's pretty short, but it explains a lot of things really clearly.

noisesmith15:07:06

@U9Q7C6G4C one thing you might be confused about is that clojure is not an interpreter - value storage is not implemented on top of let and var, it's done via java viirtual machine bytecode instructions

Elias Elfarri11:07:18

I am trying to use the defroutes macro to convert this:

(defn home-routes-backup
  (routes
    (GET "/" _
      (-> "public/index.html"
          io/resource
          io/input-stream
          response
          (assoc :headers {"Content-Type" "text/html; charset=utf-8"})))
    (resources "/")))
To this:
(defroutes home-routes
  (GET "/" [] (content-type (resource-response "index.html" {:root "public"}) "text/html"))
 (resources "/"))

Elias Elfarri11:07:21

But i get a "java.lang.NullPointerException: Response map is nil" when i try to load up the localhost. Any ideas what could be the issue?? I tried to follow the tips from this thread https://stackoverflow.com/questions/7729628/serve-index-html-at-by-default-in-compojure

indy14:07:50

The second argument to it being “public” if your files are in the resources/public folder

Elias Elfarri11:07:27

These are the imports for clarity sake:

[compojure.route :refer [resources]]
[ring.util.response :refer [redirect response file-response resource-response content-type]]

bigos14:07:44

is it possible to use java source when compiling clojure jars to be used by java app? https://github.com/bigos/JavaApplication3/blob/master/src/clojure/responder/src/jac/responder.clj

bigos14:07:27

ideally I would not only have the type of the object but also call it's methods

bigos14:07:04

my build system is clj deps.edn

bigos14:07:57

looks like it can not be done

Alex Miller (Clojure team)15:07:45

the new tools.build library for deps.edn projects is nearly out and will support this

🎉 11
dpsutton15:07:34

Started watching that last night. The intro is hilarious. And whoever did the captions explaining the missing music really nailed it

Alex Miller (Clojure team)15:07:00

it was better with the music though :)

dpsutton15:07:29

i imagine. was glad to see it. looks amazing. thanks as always for your thoughtful work 🙂

Juλian (he/him)19:07:39

is there a way to set environment variables? more specific: can I have clojure load environment variables from an .env file, at least for when I use the repl?

Juλian (he/him)19:07:51

maybe I should drop the environment variable idea and use config files instead... outpace/config looks promising for that purpose

indy19:07:59

Also check out https://github.com/juxt/aero. Also hand rolling config accessors is pretty straight forward. 1. Read from edn. 2. Store it in an atom. 3. Have functions accessors that when called get the values from the atom

seancorfield20:07:58

You cannot set environment variables once a process has started (at all). You can set system properties in the JVM however, so properties are a bit more flexible than environment variables in that respect. But using an external config file is probably a better approach.

didibus20:07:58

The environment variables as the name implies are managed by the "environment". So what you need to do is set them in your environment before launching your REPL, and then you can read them from the REPL.

didibus20:07:24

If you use lein to start your REPL, I've used: https://github.com/athos/lein-with-env-vars before to have lein define some environment variables to set.

didibus21:07:39

You can also have a look at: https://github.com/cdimascio/dotenv-java which attempts to replicate Ruby's dotenv. But like Sean said, JVM has immutable-ish env variables, so you can't set them once the program is running. The dotenv-java bypasses this by having you use its own API to get env variables. What I'd recommend though is to use System/getProperty to get the your .env config, and use dotenv-java with its systemProperties option. In general, Java libs that look for env variables have logic to fallback to properties as well (or vice versa).

didibus21:07:34

Now, I personally strongly disagree with the 12 factor app idea of using env variables for config. In my experience, it creates a mess. I recommend going with configuration for your app. And what I do is that I have my configuration be defined per-environment. Now the only environment variable I have is the name of the environment I'm in. I then use that to load the right config. Imagine: default.edn johnny-desktop.edn bob-desktop.edn build-server.edn ci.edn beta.edn prod-eu.edn prod-na.edn And then an env variable: APP_ENVIRONMENT=beta

didibus21:07:04

That way, configuration is still in code, you have it versioned controlled, can roll it back, know exactly what the config was at a point in time, etc

didibus21:07:39

Here's a nice library that implements something similar to what I'm recommending: https://github.com/levand/immuconf

noisesmith15:07:14

btw not being able to manage env vars at runtime is a portability issue, and via the right OS system calls it can be done, you just give up on the one very popular OS where this can't be done

noisesmith15:07:45

also the 12 factor thing of providing config via environment makes stealing credentials very easy one linux (the OS almost everyone actually uses in prod) - just slurp /proc/$PID/environ and you get the whole thing or better yet, the shortcut /proc/self/environ since the kernel knows your PID

👍 2
didibus16:07:09

Even if you change the env vars through the OS it won't reflect in Java, and System.getenv will return only what was there at startup, because Java caches the environment on startup I believe. I saw some ways to change the JVM env cache, but it's not an official API, so depending on your JVM and its version it might not work. I think other langs, when you execute setenv it doesn't actually set the environment vars of the parent, but simply changes the one of the current process, that bit I'm not sure why Java doesn't allow it.

noisesmith18:07:47

it's exactly the "set the vars of the current process" bit that isn't portable though

noisesmith18:07:12

if you need a child to have a different env, ProcessBuilder supports that, but that happens in a full new process not the current vm

indy15:07:17

@U051SS2EU Trying to get some more insights here. What are the other options to store credentials apart from storing it in a file that is accessible to the process or as env vars? The files could also be slurped right?

indy15:07:33

(we use a mix of files and env vars to store creds)

noisesmith15:07:53

right, files can be slurped, the industry standard right now AFAIK is having a secure credential providing service that uses SSL to provide creds on demand

noisesmith15:07:36

that way you can eg. have centralized auditing of which services are getting which creds

noisesmith15:07:49

but between a file and an env, the env is always in the same place, an attacker would have to figure out which file it's in :D

indy15:07:20

Hmm, yeah I think AWS secrets has such an API

noisesmith15:07:34

but you can slurp files that aren't on classpath too - we said the file was accessible but didn't say it was in the jar

indy15:07:01

Yeah I was wrong and hence retracted the comment, oopsie 🙂

noisesmith15:07:52

security is usually (always?) imperfect and often it's a question of putting just enough barrier or complexity in the way so an attack doesn't finally succeed

indy15:07:21

Right 👍:skin-tone-4:

noisesmith15:07:44

my main point at the start was process env is an extremely low barrier (it might be relying on the "data at rest" vs. "data in use" / "data in transit" distinction that is mostly bullshit invented for legal reasons)

noisesmith15:07:34

(modern operating system and hardware tech tends to blur that at rest / in use distinction, and I seriously doubt "attacker accesses your physical device" is a higher probability than "attacker injects code into process" in a cloud setting)

noisesmith15:07:27

if someone has access to your physical machine and you don't, putting data in env vars is less than a bandaid haha

didibus16:07:44

I'm not sure, but I think env can be found without permissions, like any user on the machine maybe can find them. While you can restrict your file to only allow read permission by the same user as your process is running under

2
didibus16:07:04

About the "not portable way", what I'm saying is other languages modify their own environment variables as part of their runtime, not the OS machinery. Java could have done that as well.

noisesmith16:07:35

the proc/pid file system thing is a file, and uses the same enforcement you would have for a file

2
noisesmith16:07:18

if it's not the OS environment variable you are changing, then why call it an environment variable? that's why we also have system properties

didibus16:07:37

Ok, then I'm not sure I consider it less secure. Once someone has ssh access with user permissions that are the same as your app, your screwed already

noisesmith16:07:08

it's a key value store in a predictable place - that removes a few hurdles

noisesmith16:07:31

my point is more that it isn't more secure, and doesn't really justify jumping through hoops to use

didibus16:07:43

Ah sure sure

didibus16:07:38

About the "point", well it's a matter of opinion, but clearly in a practical use-case kind of way, most people ended up implementing logic that goes: (or (System/getenv "foo") (System/getProperty "foo")) And a lot of people than realized, how can I mock "foo" in some test? And then everyone eventually encounters a library that maybe only relies on env variables, but you don't use them, you use a configuration and would like to be able to from your config inject the env vars of that lib, etc.

noisesmith16:07:44

ah right - the reliance on env (as in the OS facility) gets over-concretized

noisesmith16:07:52

so you see the question as "how do I test / elaborate a system that relies on env" while I saw it as "how do I change this thing that has a specific meaning to the OS"

didibus16:07:02

I think they could have done something like getenv has a counterpart setenv, which is the union of the OS environment + what was set in the process itself. And then have another getOSenv API that returns only what is in the OS environment.

noisesmith16:07:09

it's not like many people fork+exec the jvm after all (the main reason to change your env vars)

didibus16:07:50

Right, and I'm pretty sure other languages don't change the OS environment of the process either, cause like you said, that's not portable and not all OS eben support it. But they do what I say, in that if inside your program you change the current environment, it's changing the runtime values of it (not the OS one), but in practice you don't see the difference, because any call in your language to getenv will return the values. And I think some even make it that if you do start a child process it'll copy those over to it as well. So as the programmer you're fooled in a way, as if it did change the environment vars, even though it didn't.

rickheere19:07:44

Some ClojureScript

(defn get-prices-and-balances
  [client return]
  (let [prices-c (chan 1 (map #(assoc {} :prices %)))
        balances-c (chan 1 (map #(assoc {} :balances %)))
        combined-c (merge [prices-c balances-c])]
    (b/get-all-prices client prices-c)
    (b/get-balances client balances-c)
    (go-loop [prices nil
              balances nil]
      (if (some nil? [prices balances])
        (let [result (<! combined-c)]
          (cond
            (:prices result) (recur (:prices result) balances)
            (:balances result) (recur prices (:balances result))))
        (>! return
            (assoc {} :prices prices :balances balances))))))
The question I have is. Is there another way to fire off 2 async calls (get data from 2 resources on the internet in this case) and manage the data returned. I know I can do better in this code with putting the values on a map, then taking them from the map and putting it in the recur and all the way at the end put it on a map again. The thing is also, both the b/get-all-prices and b/get-balances functions internally return promises that I put on core.async chancels but now I'm starting to get the idea it would be better to just leave those as promises and use something like (<p! (.all js/Promise #js [b/get-all-prices b/get-balances])) and I get the same effect (parallel calles). I also made something with pipeline-async in the past but ended up with quite some code as well. I have the idea I'm missing something.

hiredman19:07:48

Or use into instead of merge

hiredman19:07:03

Or don't combine the channels at all and loop around alt

dpsutton19:07:47

what benefit do you get from all of this rather than

(defn ps-and-bs [client]
  (go
    (let [p (b/get-all-prices client)
          b (b/get-balances client)]
      {:prices (<! p-c )
       :balances (<! b-c)})))
Return a channel rather than taking in a channel. you can pipe from one to the other though if you want. No adding things into maps and then taking out and then adding back into a map

👌 3
2
rickheere19:07:41

This is gold, I did miss that a go block returns a channel with the result, and yes I can just wait for both to resolve. This is exactly what I wanted. I just thought I needed all that other code to get this effect. Thanks a bunch!!

dpsutton19:07:37

Yeah just fire the events before waiting on them achieves what you want

rickheere19:07:25

I read, I think in a book form Alex Miller it would be more flexible to accept channels instead of creating and returning them.

rickheere19:07:12

Thats why I did that. But then again, this is not some library but just my own code so yeah, this is easier anyway

rickheere19:07:04

Thanks for the reply as well @hiredman