Fork me on GitHub
#beginners
<
2016-11-29
>
jgh00:11:12

Am I able to pass a constant created earlier in a let statement to a function used later in the let statement? For example, I'm using compojure and taking some arguments like so:

(defn ^:private register [req]
    (let [username (get (:params req) :username)
          password (get (:params req) :password)
          email (get (:params req) :email)
          entry-count (d/q '[:find (count ?e) . :where [?e :user/email email]] db)
         ]
...
The entry-count query works properly if I hard-code an existing e-mail address, but does not work if I attempt to use the email constant I'm creating with let. Within the let statement the email parameter contains the expected data, but accessing entry-count returns nil.

jgh00:11:29

Or, perhaps, do I need to do something different to pass a parameter into a datalog query

jgh00:11:12

Also why does everyone who makes sample code for clojure insist on using out-of-context snippets with single letter names for everything, it's kinda annoying and would make getting into clojure a lot easier if everything wasn't so cryptic

jgh00:11:09

ok, i think i see where i'm going wrong..need to pass it more like

(d/q '[:find (count ?e) . :in $ ?email :where [?e :user/email ?email]] db email)

roelofw07:11:57

maybe a stupid beginners question. I have this code :

(map
      (->> client/get "" {:as :json} )
           :body
           :artObjects           
      )
    id_list ) 

roelofw07:11:58

how can I make it work so the xxx in the client/get line gets replaced by the current item out of the map ?

andrii_u08:11:11

you should give to map an anonymous function which will take one argument and pass it to format function

(map
 #(->> client/get (format “/…/%s?key=whatever” %)
 …)
 id-l)
and AFAICT there is something wrong with braces in your snippet

roelofw13:11:39

@andrii_u and the ... is the part of the https://www.rijksmuseum ?

andrii_u13:11:18

its whole url part before xxx

roelofw13:11:37

another question : can I somehow print the what is the output of client/get .

roelofw13:11:53

Can I just ise a (println around it ?

sb13:11:10

not a best code I know, but maybe helpful..

agile_geek14:11:08

@roelofw BTW your println code didn't work because you are printing the anonymous function not the result of calling the anonymous fn

roelofw14:11:55

oke, but chancing it to '(println (# ...... )) ` does also not work

roelofw14:11:14

I would like to check if the url is right

dpsutton14:11:41

make a function that formats the url

dpsutton14:11:38

(defn [id] (format """ id))

dpsutton14:11:45

then use that where you need the url

roelofw14:11:31

oke, so first take the format function as a seperate function and then use it on a second function that reads from the external api ?

dpsutton14:11:42

make a function called id->url

dpsutton14:11:58

yeah. make small composable bits

roelofw14:11:16

oke, I have to think if I use the map right. The function is now only returning the input

roelofw14:11:14

What I want is map over a [] , make the url and do something with the output of the url

roelofw14:11:22

and now it does not do it 😞

roelofw14:11:35

back to the drawing board

sb14:11:09

ring-curl?

roelofw14:11:54

I will look, I was hoping that clj-http could do the job

roelofw14:11:12

I think I cannot use a map function in a ->> somehow

roelofw14:11:27

but I will look at ring-curl

agile_geek14:11:06

@roelofw your parens are in the wrong place

agile_geek14:11:41

your call to map is not taking id_list as an argument

roelofw14:11:16

I have noticed that

agile_geek14:11:17

and you are returning id_list at the end unchanged

roelofw14:11:42

I try to figure out where the parentheses must be then

agile_geek14:11:59

(defn your-fn [args] (map <fn to call> id_list))

roelofw14:11:59

But I find it confusing that after the client-get must be the :body part

roelofw14:11:19

At this moment I have no clue where id-list must be placed

roelofw14:11:16

I try to mimick this part :

roelofw14:11:00

but then the ->> must be in a map

agile_geek14:11:35

Put all the code so far in some thing like pastebin or https://www.refheap.com/ and post the link here

dpsutton14:11:33

remember what each form must look like

dpsutton14:11:39

(map fn coll)

roelofw14:11:56

the first function makes the [] which will read by the second function later on

dpsutton14:11:47

i'm not sure what you're talking about

roelofw14:11:43

The first function read-numbers reads a json file from a external api and takes out ids in a []

roelofw14:11:55

so the output could be [1 , 10, 20]

haywood14:11:08

@roelofw do you use parinfer?

dpsutton14:11:23

^ yeah. don't ever type close parens 🙂

agile_geek14:11:25

or smartparens or paredit

dpsutton14:11:31

some kind of structure

agile_geek14:11:48

Your second fn isn't using the first

roelofw14:11:52

the second function will take that as a input and uses the numbers to make a different url which reads more info.

roelofw14:11:09

@agile_geek : at this moment, no

roelofw14:11:10

I try to make the second function work before I would use it on the output of the first function with some dummy data

agile_geek14:11:16

Here is the bit of the second fn that includes map

(defn read-daata_painting
  "Reads the title, description, date , collection, colors and url of a image"
   [id_list]
    (map
      #(->> client/get (format "" {:as :json} % ))
             :body
     )

agile_geek14:11:30

there is no id_list to map over

agile_geek14:11:38

you are not mapping over anything

roelofw14:11:48

therefore the line : read-data-painting [1] as dummy data

agile_geek14:11:48

in fact you're retruning a transducer

haywood14:11:29

^ ( the map function partially applied, in a sense)

roelofw14:11:33

a what, never heard of a transducer

haywood14:11:22

what are you trying to do here that your read_the_numbers function doesn't do

roelofw14:11:29

oke, what I try to do it map over at this moment as input [1] , make a url of it and read it and take the body part out of it

haywood14:11:03

maybe instead of using the thread last macro, you should just write it out

haywood14:11:17

(map (fn [id] ...) id_list)

haywood14:11:27

because your code is really confusing

roelofw14:11:34

read_the_numbers works fine. That one gives as output : (nl-SK-A-1718 nl-SK-C-1368 nl-SK-A-2963 nl-SK-C-2 nl-SK-C-149 nl-SK-A-3841 nl-SK-A-1115 nl-SK-C-216 nl-SK-C-211 nl-SK-A-799)

roelofw14:11:01

oke, I will try again to make this work

roelofw14:11:12

maybe I had the wrong approach here

haywood14:11:24

I think you know what you need to do, you're just getting tangled up in syntax

roelofw14:11:32

@haywood : yes. Im sure im tangled up

haywood14:11:44

use a let block to define each step

agile_geek14:11:47

Yeah that thread last macro ->> is probably trying to take the results of calling client/get with no arguments and threading that to the format call that also takes no args!

haywood14:11:52

clojure
(map (fn [id]
       (let [url (str "" id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
             response (client/get url {:as :json})]))
  list_of_ids)

haywood14:11:02

also try and install parinfer as soon as possible

haywood14:11:22

it manages the parens for you, you just have to line things up via indentation

roelofw14:11:34

I use cursive which has parinfer

haywood14:11:41

I know it has paredit, I don't think it uses parinfer

roelofw14:11:05

Thanks, I see now as output (nil) so more puzzeling to do

haywood14:11:25

cool, good work

roelofw14:11:43

can I now print the url with (println url) instead of the resource line ?

haywood14:11:21

yea url is just a string

haywood14:11:51

you can println in a let block by _ (println url)

haywood14:11:02

so it's a side effect not bound to anything

roelofw14:11:46

oke, the url is well , I could check it on a browser

agile_geek14:11:53

So if you changed the code above to return response you should see something

roelofw14:11:27

Now finding out how I can get the artObjects and after that more data then I need

agile_geek14:11:38

(map (fn [id]
       (let [url (str "" id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
             response (client/get url {:as :json})]
         response))
  list_of_ids)

roelofw14:11:53

yes, I can do (body (:body response) in the let binding 🙂

agile_geek14:11:59

(map (fn [id]
       (let [url (str "" id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
             response (client/get url {:as :json})
             body (:body response)
             art-objects (:artObjects body)]
         art-objects))
  list_of_ids)

agile_geek15:11:17

that's one step at a time

roelofw15:11:30

everybody thanks

roelofw15:11:30

last question : can this be converted to a threading macro just like the first ?

dpsutton15:11:54

almost definitely

dpsutton15:11:58

play around with it and figure it out

agile_geek15:11:09

Basically you would need to do the bit that formats the url and get's the response in one call then thread that through :body and :artObjects

roelofw15:11:42

I think I have a idea. I think the let part can be replaced with the threading macro

agile_geek15:11:17

you couldn't do the url threaded into the get using ->> as the get takes the url as the first argument not the last

agile_geek15:11:18

well you can ... if you create a fn that takes one argument for the get then that single arg is both first and last

roelofw15:11:52

@agile_geek oke, back to another idea

agile_geek15:11:11

@roelofw something like this:

(map (fn [id]
       (-->
             (str "" id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
             #(client/get % {:as :json})
             :body
             :artObjects))
  list_of_ids)

roelofw15:11:23

That is what I was thinking 🙂

agile_geek15:11:24

and in fact you could use the thread first macro too ->

agile_geek15:11:25

(map (fn [id]
       (->
             (str "" id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
             (client/get {:as :json})
             :body
             :artObjects))
  list_of_ids)

roelofw15:11:47

Thanks, Can I also do something like this under the :artObjects part description = :description to get the description

roelofw15:11:08

and store it in a variable for a moment

agile_geek15:11:10

If :description is a key inside the :artObjects map

agile_geek15:11:30

you would have to use a let to store it locally

agile_geek15:11:54

you can't do variable assignment outside of a let of a def

roelofw15:11:31

hmm, What I wanted is to store everything in a map of maps

roelofw15:11:19

so I have to make a let on data which I use in the body

agile_geek16:11:04

(map (fn [id]
       (let [art-objects (->
                                       (str "" id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
                                       (client/get {:as :json})
                                       :body
                                       :artObjects)
               description (:description art-objects)
               another-thing (:another-thing art-objects)]
        {:description description :another-thing another-thing}))
  list_of_ids)

agile_geek16:11:29

or you could use assoc's threaded together to build a map

agile_geek16:11:49

this will produce a sequence of maps not a map of maps

agile_geek16:11:46

if you wanted a map of maps you would need to use somehting like id as the key of each map for each object and use reduce not map to build your outer map

roelofw16:11:28

oke, I thought this would be nice : '( ( 1 "my painting" "My first made painting") ( 2 "rubbisch" "rubbisch made by me ") )

roelofw16:11:27

hmm, code stills outputs nill , so I cannot check the contents of the map function 😞

roelofw16:11:32

time for a break

agile_geek16:11:14

I'm typing my code directly into slack so my parens might get unbalanced!

roelofw16:11:26

I solved that already

roelofw16:11:56

@agile_geek but does the description part not be a part of the macro ?

agile_geek16:11:22

it can be but then you will get only the description

agile_geek16:11:01

and you've lost the art objects so you can't go back into that to get anything else that's in art objects

agile_geek16:11:20

I think you want other stuff?

agile_geek16:11:43

i.e. art objects has multiple key-values you want?

roelofw16:11:19

yes, that is correct

roelofw16:11:00

and I have to make another function which also has to be a part of the maps of maps or whatever the output will be

agile_geek16:11:29

so you need to hang on to art objects and use it more than once, hence I put it in a let

dpsutton16:11:46

can you post the entirety of the code you're working on now?

agile_geek16:11:02

or you could use juxt to destructure all your values at once but that's a bit advanced

roelofw16:11:32

@dpsutton here or on another paste ?

dpsutton16:11:56

just post the code here. we're trying to diagnose what i think are just arguments out of place without seeing the code

roelofw16:11:17

code :

(ns proefopdracht.core
  (require [clj-http.client :as client])
  (require [cheshire.core :refer :all]))

(defn read_numbers
  "Reads the ids of the paintings"
  []
  (->> (client/get "" {:as :json})
       :body
       :artObjects
       (map :id)))


(defn read-data_painting
  "Reads the title, description, date , collection, colors and url of a image"
   [id_list]
  (map (fn [id]
         (let [art-objects
               (-> (str "" id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
                   (client/get {:as :json})
                   :body
                   :artObjects)
               description (:description art-objects)]))  id_list))


(println (read-data_painting ["SK-A-1718" ]))  

roelofw16:11:11

which produces (nil) as output where I expect to see : ( (SK-A-1718 "my painting))

roelofw16:11:20

as a example

agile_geek16:11:45

you're not returning anything from the fn called by map

agile_geek16:11:23

so after the ] in the line with description on it you need to return the description

agile_geek16:11:51

also check those parens balance

roelofw16:11:37

oke, so I can do something like : ( ( id description))

roelofw16:11:44

I will do that

agile_geek16:11:04

[id description]

agile_geek16:11:27

or '(id description) for a list

agile_geek16:11:36

note the quote

agile_geek16:11:10

I would use a vector

roelofw16:11:39

and with more a vector of vectors ?

dpsutton16:11:28

also, since you are having trouble with seeing what your functions are returning, don't use anonymous functions

agile_geek16:11:29

You will get a lazy sequence from the map fn so it will produce a sequence of vectors, one vector in the sequence for each artObject

dpsutton16:11:36

name them, try them, ensure they are correct before you compose

roelofw16:11:54

thanks, I have to figure out why description is nil

roelofw17:11:45

Very last question today : Can I later on add something to the map

dpsutton17:11:06

you know you can

dpsutton17:11:11

maps are extensible with assoc

dpsutton17:11:19

they are just objects with whatever keys you want

roelofw17:11:24

I have to write a function which can add the image url to every map

dpsutton17:11:38

you literally made one with the stuff you needed to begin with

roelofw17:11:20

@dpsutton : how do you mean that

roelofw17:11:28

I have now a keyword which is under 2 : so first I have to find the keyword dating and within it I have to find the keyword year . Can I do something like this : (date :dating :year)

roelofw17:11:55

found it. I can use (:year (:dating art-objects))

agile_geek17:11:02

More idiomatically you can call (get-in art-objects [:dating :year])

roelofw18:11:13

Thanks , changed it

roelofw18:11:34

I takes now some 4 seconds to read 1 painting

agile_geek18:11:36

It's making an http call to retrieve data that will be slow

roelofw18:11:16

yes , and it does in sequantally

agile_geek18:11:23

And you're making that call for every od

roelofw18:11:37

first all the numbers, then for the data

roelofw18:11:40

Could I make it work that I take the numbers out of the first request and then place them in a sort of quee so that the data is downloaded in parallel

agile_geek18:11:00

If the first call returns the art objects with their IDs and description you can just make one call

roelofw18:11:23

unfortanly it does not.

agile_geek18:11:07

I'd need to know more about what the http api was to determine best solution

roelofw18:11:41

I can give you the same readme file of the api that I use

roelofw18:11:59

then you have the same info as I have

agile_geek18:11:17

I'm not at a computer atm (on the bus!) but you can post a link to the problem here. I wouldn't upload the whole file to slack

agile_geek18:11:46

I can look at it tonight if I get time

roelofw18:11:04

no problem. Here is the link : http://rijksmuseum.github.io/

roelofw18:11:27

have a save ride back home

roelofw18:11:16

i think I will be around for some 2 - 3 hours. then bed-time here

agile_geek18:11:39

:sleeping_accommodation:

roelofw18:11:15

then its around 10 o clock in the evening here

roelofw18:11:44

maybe later I can make it work as my first web app in for example luminus or just like so many people look for my own parts like ring

roelofw19:11:06

@agile_geek do you also need the code I have so far ?

agile_geek19:11:08

Could do - just paste it to refheap

agile_geek20:11:17

Looking at those APIs I can't see an obvious way to minimise the number of calls.

roelofw20:11:14

oke,. here is my code so far: https://www.refheap.com/124020

agile_geek20:11:09

Also be aware that map is lazy so if you called this in a program rather than from a repl it may do nothing. The repl forces the sequence from map to be realised. To force it to realise you may need to wrap the top map call in a doall

tokenshift20:11:38

If you have to make a separate call for each ID, the simplest way to speed it up might be pmap instead of map; still makes all the calls, but does them in parallel.

tokenshift20:11:03

agile_geek’s note about the result being lazy still holds.

agile_geek20:11:56

@tokenshift true. Although I usually use core.aync pipelines to parallelise this stuff in real projects

roelofw20:11:02

oke, the code that I have, give the output as expected

roelofw20:11:10

I have tested pmap but it seems not be make much difference

agile_geek20:11:28

@roelofw well done. Try pmap where map is in your read-data_paintings fn

roelofw20:11:55

I did , looks that it does not make much difference.

roelofw20:11:06

when I count the time myself

agile_geek20:11:22

Btw it's idiomatic to use hyphens - between words in fn names not underscores _

roelofw20:11:26

I have not found out how I can time it with cursive

mlev20:11:45

I am receiving a Clojure Multimethod error in one of the Clojure Brave and True examples. I've posted the question on Stack Overflow in case others have run into a similar question. If anyone has any ideas about how I can fix my code, I'd certainly appreciate it: http://stackoverflow.com/questions/40875042/clojure-multimethod-error-no-method-in-multimethod-for-dispatch-value-null

roelofw20:11:25

with map : "Elapsed time: 2045.2662 msecs" with pmap : "Elapsed time: 689.110806 msecs"

roelofw20:11:42

so it makes a lot of difference

agile_geek20:11:12

All with the addition of a p 😄

roelofw20:11:59

maybe I can read up with core.async or can look if clj=http can do jobs async

roelofw20:11:09

and look if there is more to win

agile_geek20:11:04

Core.async is reasonably advanced.

roelofw20:11:33

oke, maybe not the best solution yet as Im find myself still a beginner

agile_geek20:11:02

I suspect what u have is about as good as you'll get for now. Http calls are slow.

agile_geek20:11:41

It's fetching data across internet. A faster connection makes most difference

roelofw20:11:31

oke, I found this example :

(client/get ""
            {:async? true}
            ;; respond callback
            (fn [response] (println "response is:" response))
            ;; raise callback
            (fn [exception] (println "exception message is: " (.getMessage exception))))  

agile_geek20:11:02

That's not going to speed up your problem

agile_geek20:11:37

It is useful if you want to do something else in the same thread while waiting for the response

agile_geek20:11:24

But of you need the response it has to return from the server and that's dependent on the speed of network and the server

roelofw20:11:45

that is right

roelofw20:11:06

I tested it and cam back to more then 2000 ms

roelofw20:11:11

so back to pmap

agile_geek20:11:48

pmap is parallelising some of the calls so best you can do. I.e. calls several at same time in different threads

agile_geek20:11:27

The core.async pipelines I mentioned is just a more flexible and scalable version of same thing.

agile_geek20:11:08

It allows tuning of number of parallel threads and composing transducers which are easy to test in isolation if you've a very complex pipeline of transformations. But you don't need the extra complexity of core.async right now

agile_geek20:11:28

Any way got to go

roelofw20:11:03

maybe if im more experience with clojure I can try

roelofw20:11:06

@seancorfield I have this so far with a lot of help. Does this look good : https://www.refheap.com/124020

roelofw20:11:44

tomorrow I try to add that a image url is read and added to the vector

roelofw20:11:51

ans maybe later adding some sort of pagination. This is some 470 pages long if I take all the images

seancorfield21:11:09

@roelofw: I'll take a look when I'm back at my computer. For now I'd just note that _ in names is not idiomatic - use - instead.

roelofw21:11:53

oke, I can change that

seancorfield21:11:28

@roelofw: is there a new URL? I still see _ in names at the other URL.

roelofw21:11:12

oke, I will make a new paste , moment

seancorfield21:11:17

id_list, also it should be :require in ns (you'll get an error on Clojure 1.9 for require). More later once I see it on a bigger screen

roelofw21:11:07

id_list schould be id-list?

roelofw21:11:35

I tried :require but I see then a error

roelofw22:11:21

time for sleep for me now. it's 23:00 hour here

roelofw22:11:33

thanks for the remarks

roelofw22:11:38

oke, it worked fine.

roelofw22:11:49

New code :

(ns proefopdracht.core
  (:require [clj-http.client :as client])
  )

(defn read-numbers
  "Reads the ids of the paintings"
  []
  (->> (client/get "" {:as :json})
       :body
       :artObjects
       (map :id)))


(defn read-data-painting
  "Reads the title, description, date , collection, colors and url of a image"
  [id-list]
  (pmap (fn [id]
         (let [art-objects (-> (str "" id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
                               (client/get {:as :json} )
                               :body
                               :artObject)
               description (:description art-objects)
               date        (get-in art-objects [:dating :year])
               collectie   (first (:objectCollection art-objects))
               colors      (:colors art-objects)
               ]
           {:id  id :description description :date date :collectie collectie :colors colors }  ))  id-list))


(time(println (read-data-painting (read-numbers))))