Fork me on GitHub
#beginners
<
2016-12-09
>
bcbradley05:12:56

Can functions be invoked at the top level in clojure? I have this situation where I want to generate the definitions for a bunch of functions from some data in an array of maps. I'd like to have a top level function call that iterates over the array and builds and binds one function per map using def. I don't know if clojure supports this level of dynamicism though.

Shantanu Kumar06:12:30

@bcbradley I think technically you can do that using alter-var-root, but I’m curious what use case is motivating you to do such a thing.

bcbradley06:12:03

@kumarshantanu I'm making a clojure library for https://www.khronos.org/registry/spir-v/specs/1.1/SPIRV.html by leveraging https://github.com/clojurewerkz/buffy but I have found it easiest / least error prone to codify the api as something resembling a table, and then convert the tabulated data into functions and documentation programmatically

bcbradley06:12:28

for instance, OpSourceContinued looks like https://www.khronos.org/registry/spir-v/specs/1.1/SPIRV.html#OpSourceContinued looks like

{:name "source-continued" :code 2 :operands [{:name "continued-source" :type :literal-string}]}

Shantanu Kumar06:12:13

If you just provide a factory fn, which given a table would create and return the fns, would that not suffice? If the fns are created at runtime, the user can create vars from those, probably also instrumenting those if required etc.

bcbradley06:12:37

hrm, is there a performance penalty with that?

bcbradley06:12:45

this is supposed to be a fairly low level lib

Shantanu Kumar06:12:00

I don’t think there’s any perf penalty in that approach

Shantanu Kumar06:12:19

in fact, if you define a var and don’t enable direct-linking then you may incur a perf penalty

bcbradley06:12:42

seems like my options are to either make the table visible to the library user

bcbradley06:12:55

or use the table to generate a library and then let the library be visible to the user

bcbradley06:12:19

the indirection is either visible or invisible

bcbradley06:12:23

i'm not sure how I feel about it

Shantanu Kumar06:12:31

Would the table ever need to be manipulated, visualized, stored in DB etc? Imagine the use cases a user may want to use it in. If the answer is yes, then it needs to be a first class concept in your API.

bcbradley06:12:56

would you want to have such use cases?

bcbradley06:12:01

i mean personally

bcbradley06:12:39

the api isn't going to change that much, it may have some things added to it

bcbradley06:12:49

so the table wouldn't change that much

Shantanu Kumar06:12:17

Actually, I don’t know anything about graphical shaders, so can’t comment personally, but I’d suggest that you consider the various use cases a user may have, then take a call.

bcbradley06:12:26

i do feel like it is pretty nice to be able to view the salient details of the entire api as a table

hippo06:12:43

hey guys, how can i write line by line collection of strings into the file?

roelofw07:12:24

@hippo and did you get it work ?

hippo07:12:03

@roelof yep, thank you

rustam.gilaztdinov08:12:58

Hey guys, which is the best practices in case you lazy iterate on collection and you should check that rows not duplicates?

roelofw08:12:06

@rustam.gilaztdinov I would do it like this (

roelofw08:12:05

( = col (set(col)))

roelofw08:12:38

so I make a set of it and compare it to the orginal. if there are no duplicates both are the same

roelofw08:12:22

and the return value is true

rustam.gilaztdinov08:12:53

Yes, i do the same, but i don’t won't to force my collection into set Think about distinct, may be

rustam.gilaztdinov08:12:31

And this may be solution

(defn distinct-by
  "Returns elements of xs which return unique
   values according to f. If multiple elements of xs return the same
   value under f, the first is returned"
  [f xs]
  (let [s (atom #{})]
    (for [x xs
          :let [id (f x)]
          :when (not (contains? <@U06QXASV8> id))]
      (do (swap! s conj id)
          x))))

roelofw08:12:43

could also work

roelofw08:12:11

@rustam.gilaztdinov then you have to make a function to for comparision

rustam.gilaztdinov08:12:14

Check this out

(let [coll (repeatedly 1000000 #(do
                                 {:id (rand-int 500000)
                                  :a (rand-int 100000)
                                  :b (rand-int 100000)}))]
  (println "distinct-by"
           (with-out-str
             (time
               (plumbing.core/distinct-by :id coll))))
  (println "group-by"
           (with-out-str
             (time
               (->> coll (group-by :id) (vals) (map first))))))

distinct-by "Elapsed time: 0.054141 msecs"

group-by "Elapsed time: 1113.419474 msecs"

roelofw08:12:29

That is a lot of difference

agile_geek10:12:46

@rustam.gilaztdinov yeah but you are not comparing the same things there. vals forces the entire lazy sequence to realise after you've already walked it with group-by

agile_geek10:12:13

Of course if you really want performance you can step into the world of mutable data structures (yeuch!) but I would confine them to the scope of you fn not globally

agile_geek10:12:30

This is exactly what @weavejester has done in his implementation https://weavejester.github.io/medley/medley.core.html#var-distinct-by using volatile!

dominicm15:12:44

@bcbradley I think you're definitely looking for macros

dominicm15:12:51

I've seen a few libraries do what you're looking for. It's perfectly acceptable to use macros to automate calls to def

dominicm15:12:24

https://github.com/SevereOverfl0w/bukkure/blob/7fd1cb6dc875f137698dee78deea149d31e4d2e5/src/bukkure/config.clj#L12-L32 this is similar to what you want. I don't know if it's a best practice the way it's written here. I can't recall any of the other libs that do this though!

dominicm15:12:38

I need to revisit that library, I want to do something fun with it at some point.

roelofw16:12:18

Is it right that if I refractor all the client/get out only do-both-in-parallel is not pure in the api-get.clj file ?

seancorfield16:12:14

I’d have to see the latest version of your code but that sounds likely, from what I remember of your code.

seancorfield16:12:42

Are you keeping an up-to-date copy of the code up on GitHub now?

roelofw16:12:56

This is the lastest one before the testing questions

roelofw16:12:14

so yes, the latest version can be found on github

seancorfield16:12:22

OK, so you have client/get in three functions and the do…parallel calls two of them, so everything is non-pure there.

roelofw16:12:42

yes, that correct

roelofw16:12:15

we talked in the clojure channel that I can refractor the client/get to the home route

roelofw16:12:36

I can do the same I think for the rest of the functions

seancorfield16:12:44

Given that both read-data-painting and read-image-url are dependent on their id argument to call client/get, you can’t do much about that.

dpsutton16:12:20

and i like that those keep the client/get call inside of them because the function is type (id -> data about the thing with that id)

roelofw16:12:51

oke, so there I have to mock things up with with-redefs

dpsutton16:12:01

to test those, yes

roelofw16:12:36

oke, I will try that and hope I get it working

roelofw16:12:52

both thanks for the help and patience with me

scott.haleen16:12:00

Is there a higher order function or idiomatic way to convert a sequence of maps to a map based on a key fn? Or is it just reduce ex. (reduce #(assoc %1 (:id %2) %2) {} [{:id 1} {:id 2}]) ;;=> {1 {:id 1}, 2 {:id 2}}

roelofw16:12:39

hmm, I did something not right.

roelofw16:12:56

I change the read-numbers to this :

(defn read-numbers
  "Reads the ids of the paintings"
  [response]
  (->> :body
       :artObjects
       (map :objectNumber))) 

roelofw16:12:20

and the home.route to this :

(defn home-page []
  (let [url ""
        options {:as :json :query-params {:key (env :key) :format "json" :type "schilderij" :toppieces "True"}}]
        (layout/render
          "home.html" {:paintings (-> (client/get url options)
                                      api/read-numbers
                                      api/do-both-in-parallel)}))) 

roelofw16:12:36

but somehow no output at all on the screen

roelofw16:12:22

solved , I did change :body to (:body response)

roelofw16:12:39

now I can finally writing test for this function

roelofw16:12:16

and the first fail

roelofw16:12:30

I have this test :

(deftest test-app
  (testing "get numbers"
   (is (= [ "1234" "abcd"] (api/read-numbers [ {:objectNumber "1234"} {:objectNumber "abcd"} ])))))  

roelofw16:12:59

and I see this output :

AIL in (test-app) (api-get.clj:9)
get numbers
expected: (= ["1234" "abcd"] (api/read-numbers [{:objectNumber "1234"} {:objectNumber "abcd"}]))
  actual: (not (= ["1234" "abcd"] ()))  

roelofw16:12:32

time for dinner

roelofw16:12:38

found it but ran into another issue : Im testing now with

(read-numbers [{:body {:artObjects {:objectNumber "1234"} {:objectNumber "abcd"}}} ])  

roelofw16:12:34

but in repl I see these error messages :

RuntimeException Map literal must contain an even number of forms  clojure.lang.Util.runtimeException (Util.java:221)
RuntimeException Unmatched delimiter: }  clojure.lang.Util.runtimeException (Util.java:221)
RuntimeException Unmatched delimiter: ]  clojure.lang.Util.runtimeException (Util.java:221)
RuntimeException Unmatched delimiter: )  clojure.lang.Util.runtimeException (Util.java:221)  

dpsutton16:12:06

that's a syntax error

dpsutton16:12:09

fix your syntax

dpsutton16:12:23

art objects is a map with "two" maps as its values

dpsutton16:12:25

that's not possible

roelofw16:12:00

fixed but something is still wrong. See this repl :

(read-numbers [{:body {:artObjects [{:objectNumber "1234"} {:objectNumber "abcd"}]}} ])
=> ()  

dpsutton16:12:32

well, you've found the reason for testing. fix your function

dpsutton16:12:39

and make sure your input data is what you expect

roelofw16:12:43

the function looks like this :

(defn read-numbers
  "Reads the ids of the paintings"
  [response]
  (->> (:body response)
       :artObjects
       (map :objectNumber)))  

roelofw17:12:27

I think there is still something wrong with my test data

dpsutton17:12:00

you're calling read-numbers on a vector, not a map

roelofw17:12:04

Changed it : (read-numbers [{:body {:artObjects ({:objectNumber "1234"} {:objectNumber "abcd"})}} ])

roelofw17:12:12

but still empty output

dpsutton17:12:17

you're still calling read-numbers on a vector

dpsutton17:12:30

(read-numbers [

dpsutton17:12:36

that opening brace means vector

dpsutton17:12:42

and that is not what your function handles

roelofw17:12:57

oke, changed it again : (read-numbers ({:body {:artObjects {{:objectNumber "1234"} {:objectNumber "abcd"}}}} ))

roelofw17:12:09

and now arity error message 😞

roelofw17:12:17

this is not my day

dpsutton17:12:25

don't put a container around your map!

dpsutton17:12:29

pass the map directly

dpsutton17:12:50

(read-numbers {:body {:artObjects {{:objectNumber "1234"} {:objectNumber "abcd"}}}} )

dpsutton17:12:09

and what are the extra braces around the object number stuff

dpsutton17:12:46

(read-numbers {:body {:artObjects [{:objectNumber "1234"} {:objectNumber "abcd"}}]}})

dpsutton17:12:37

you are calling read-numbers on a map

dpsutton17:12:49

body is a map

dpsutton17:12:59

artbojects is the key of that map whose value is an array of two maps

roelofw17:12:33

Thanks, test is a success

mss17:12:49

newbie question re core async – how should I think the size of my buffers? do they want to roughly correspond to processors available? memory available? something else? does it depend on workload?

Drew Verlee18:12:59

i fel like i ask this every other day, but whats the best way to do this? (??? [[1 2] [3 4] [5 6]]) => [[1 3 5] [2 4 5]]

rauh18:12:44

@drewverlee apply map vector -> transpose

Drew Verlee18:12:28

yep, thats it. but i cant grok why. i dont think i understand what apply does

jswart18:12:13

(apply + [1 2 3]) is like (+ 1 2 3)

Drew Verlee18:12:53

ok, i understand apply. I think its actual some behavior or map i wasn’t aware of/ wasn’t using: Returns a lazy sequence consisting of the result of applying f to the set of first items of each coll

jswart18:12:05

ah yes, but it makes sense now right?

Drew Verlee18:12:10

thanks jswart!

jswart18:12:23

haha, I think the credit goes to rauh

Drew Verlee18:12:35

in the future i might be able to ask a barliman. https://www.youtube.com/watch?v=er_lLvkklsk

jswart18:12:16

will byrd is one of the smartest and also funniest people I ever had the pleasure to meet

Drew Verlee18:12:31

I’m going to make it to the next conj, clojure job or not 🙂

Drew Verlee18:12:49

is there version of take with a predicate? like “only take if predicate"

Alex Miller (Clojure team)18:12:25

clojure.core/take-while
([pred] [pred coll])
  Returns a lazy sequence of successive items from coll while
  (pred item) returns true. pred must be free of side-effects.
  Returns a transducer when no collection is provided.

tokenshift18:12:43

Or filter, if you want to skip elements that don’t match the predicate, rather than stop takeing entirely at the first failure.

Drew Verlee18:12:26

right, i suppose i’m looking to skip elements i dont match, not stop when i reach one i dont need.

tokenshift19:12:56

(Though I’d use (remove nil? …) instead of (filter (complement nil?) …) in the latter case

Drew Verlee19:12:02

@agile_geek. More specially, i’m not sure how to grow a collection to a certain point. Grow a collection until it meets a predicate (coll count is greater then 10). I can see using while to make this work, but (maybe incorrectly) i feel while isn’t the right idea.

agile_geek19:12:59

@roelofw I get the impression you're struggling a little with syntax. Here is a little reminder that might help: A list is a datastructure in Clojure that is accessible only in sequence from the first element i.e to get to third element you have to read the first and second element. A list is written inside parenthesis (a b c) Clojure's compiler will interpret the first thing in a list as a function and will try to evaluate it using the subsequent things in the list as arguments e.g. (fn arg1 arg2). This is why when you put your map inside parens you got an arity error as a map is a fn that takes the key to use to look up a value but you just had the map inside the parens so you were missing the key argument. If you want to write a list as a literal and not have the first element evaluated as a function then use a quote e.g. '(1 2 3) is a list of the integers 1,2 and 3. A vector is an indexed (associative) data structure so you can access an element using it's index. It's written using brackets (square) like so; [1 2 3]. A vector is also used in the context of a function definition to represent an argument or in a let, loop, etc. to bind a value to a symbol (a name) e.g. (defn add-3[x] (+ x 3)) or (let [a 1 name "Chris"] .....) A map is an associative data structure that consists of key-value pairs and is written using braces e.g. {:a 1 "b" 2} I know you know most of this but if you are struggling to work out why something is giving arity errors or an error mentioning Ifn (the interface for a function in Clojure) it's often a syntax problem i.e you have parens in the wrong place, so remember the syntax of Clojure is summed up by my friend @krisajenkins as: The first thing in a list is a function. The first thing in a list is a function. and the first thing in a list is a function Hope that helps?

agile_geek19:12:50

@drewverlee what do you mean by grow? You can write a function that produces a lazy sequence of things infinitely and then take only the number you need. E.g. for first 10 natural numbers: (take 10 (iterate inc 1))

roelofw19:12:39

@agile_geek thanks, I struggeling because I learned a lot of things and mixed up things now

agile_geek19:12:47

so you could def natural numbers and take 10 like this:

(def natural-numbers (iterate inc 1))

(take 10 natural-numbers)

roelofw19:12:53

IM loosing the big picture sometimes

agile_geek19:12:15

@roelofw no problem it's part of learning. We all do it.

kyle_schmidt19:12:32

trying to figure out why this isn't working:

(defn get-one-from-table [table id]
  (jdbc/query mysql-db ["select * from ? where id = ?" table id]))

kyle_schmidt19:12:44

seems to insert strings within the query string:

"select * from 'account' where id = '1'"

donaldball19:12:39

The jdbc prepared statement interpreter assumes that ? params refer to query literal values, not, um, ddl symbols?

agile_geek19:12:55

@kyle_schmidt I don't think you can substitute the table name using a placeholder like that

donaldball19:12:00

I seem to recall the syntax for mysql is backticks if you need to escape the table name

donaldball19:12:37

You’ll have to do your own sanification if you can’t trust the table name of course, sadly

donaldball19:12:46

If you’re going to be doing query composition widely, you might want to look at using e.g. honeysql, which handles cases like this nicely

kyle_schmidt19:12:00

awesome thank you! I took a look at honeysql and seems to get at the general formatting that I need because I do this for most tables in my database

donaldball19:12:04

The other lib people like to use atop clojure.java.jdbc is I think hugsql these days. It eschews query composition (largely) and advocates resource files of queries.

agile_geek19:12:44

Never tried but you might be able to construct the query with the table using str or clojure.string/join like so:

(defn get-one-from-table [table id]
  (jdbc/query mysql-db [(str "select * from " table " where id = ?" id]))

agile_geek19:12:20

Also you could pass it to prepared-statement and then call the resulting statement.

kyle_schmidt20:12:16

also just curious because I've never used a build tool before (coming from Python) but what is the Clojure way to connect to a database? Is that something I should configure in the application itself or within project.clj?

agile_geek20:12:32

If you get really stuck I'm sure @seancorfield would help out. He's a nice guy and one of the lead contributors to clojure.java.jdbc

kyle_schmidt20:12:48

I realize I'm probably opening up a can of worms but any resources around this topic are greatly appreciated

agile_geek20:12:30

@kyle_schmidt usually you would either construct the connection or connection pool in the code using parameters that you pick up from an edn file or more often environemnt variables. If you use environment variables you might want to look at environ which lets you provide defaults in either your project.clj or, even better, a separate profiles.clj that you keep out of version control system if you have passwords etc. in it.

agile_geek20:12:57

You can parse an old fashioned properties file too.

gdeer8120:12:23

@kyle_schmidt I think the Luminus framework has good examples

gdeer8120:12:54

I think it uses yesql now

dpsutton20:12:17

i've been using properties files. I check in sample ones and let the person make a real one from the sql.properties.config version

gdeer8120:12:19

but it starts you off with an H2 database for developing and then when you're going to certification or prod you run your ddl and migration if there is data you want to put into cert and prod

dpsutton20:12:41

if you have a better build system than us, you can have it replace a checked in version with the actual production connection strings, secrets, etc

dpsutton20:12:48

which is what we do for our .NET applications

kyle_schmidt20:12:24

cool! thank you for the help!

seancorfield22:12:54

@kyle_schmidt Happy to help — just got back from lunch — and I can try to answer any java.jdbc questions!

seancorfield22:12:22

There’s also the (community-maintained) documentation for java.jdbc here: http://clojure-doc.org/articles/ecosystem/java_jdbc/home.html

seancorfield22:12:38

And I’ll +1 @agile_geek suggestion to use a connection pool (there are examples in the above docs for two such libraries!) and build that at application startup and pass that around for java.jdbc to get connections from.

seancorfield22:12:14

The SQL DSL I usually point people at, to use on top of java.jdbc, is HoneySQL https://github.com/jkk/honeysql — I really like that it lets you compose SQL fragments, which is very helpful if you’re dealing with complex, dynamically constructed queries.