This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-11-29
Channels
- # arachne (3)
- # bangalore-clj (6)
- # beginners (246)
- # boot (57)
- # business (1)
- # capetown (1)
- # cider (2)
- # clara (1)
- # cljsjs (36)
- # cljsrn (1)
- # clojure (150)
- # clojure-austin (4)
- # clojure-brasil (7)
- # clojure-china (2)
- # clojure-france (3)
- # clojure-greece (3)
- # clojure-japan (3)
- # clojure-russia (38)
- # clojure-spec (25)
- # clojure-uk (25)
- # clojurescript (320)
- # clojurex (1)
- # cursive (38)
- # datascript (48)
- # datomic (23)
- # emacs (29)
- # events (1)
- # funcool (2)
- # hoplon (64)
- # jobs (3)
- # luminus (10)
- # off-topic (26)
- # om (27)
- # om-next (1)
- # onyx (1)
- # parinfer (38)
- # perun (5)
- # planck (19)
- # re-frame (38)
- # reagent (19)
- # remote-jobs (1)
- # rum (2)
- # schema (2)
- # spacemacs (1)
- # specter (8)
- # test-check (10)
- # vim (7)
- # yada (14)
Am I able to pass a constant created earlier in a let statement to a function used later in the let statement? For example, I'm using compojure and taking some arguments like so:
(defn ^:private register [req]
(let [username (get (:params req) :username)
password (get (:params req) :password)
email (get (:params req) :email)
entry-count (d/q '[:find (count ?e) . :where [?e :user/email email]] db)
]
...
The entry-count
query works properly if I hard-code an existing e-mail address, but does not work if I attempt to use the email
constant I'm creating with let
. Within the let
statement the email
parameter contains the expected data, but accessing entry-count
returns nil.Or, perhaps, do I need to do something different to pass a parameter into a datalog query
Also why does everyone who makes sample code for clojure insist on using out-of-context snippets with single letter names for everything, it's kinda annoying and would make getting into clojure a lot easier if everything wasn't so cryptic
ok, i think i see where i'm going wrong..need to pass it more like
(d/q '[:find (count ?e) . :in $ ?email :where [?e :user/email ?email]] db email)
maybe a stupid beginners question. I have this code :
(map
(->> client/get "" {:as :json} )
:body
:artObjects
)
id_list )
how can I make it work so the xxx in the client/get line gets replaced by the current item out of the map ?
you should give to map an anonymous function which will take one argument and pass it to format
function
(map
#(->> client/get (format “/…/%s?key=whatever” %)
…)
id-l)
and AFAICT there is something wrong with braces in your snippet@andrii_u and the ... is the part of the https://www.rijksmuseum ?
@roelofw BTW your println
code didn't work because you are printing the anonymous function not the result of calling the anonymous fn
oke, so first take the format function as a seperate function and then use it on a second function that reads from the external api ?
oke, I have to think if I use the map right. The function is now only returning the input
What I want is map over a [] , make the url and do something with the output of the url
@roelofw your parens are in the wrong place
your call to map
is not taking id_list
as an argument
and you are returning id_list
at the end unchanged
(defn your-fn [args] (map <fn to call> id_list))
@agile_geek I know that part
Put all the code so far in some thing like pastebin or https://www.refheap.com/ and post the link here
here you are : https://www.refheap.com/124016
The first function read-numbers reads a json file from a external api and takes out ids in a []
or smartparens or paredit
Your second fn isn't using the first
the second function will take that as a input and uses the numbers to make a different url which reads more info.
@agile_geek : at this moment, no
I try to make the second function work before I would use it on the output of the first function with some dummy data
Here is the bit of the second fn that includes map
(defn read-daata_painting
"Reads the title, description, date , collection, colors and url of a image"
[id_list]
(map
#(->> client/get (format "" {:as :json} % ))
:body
)
there is no id_list
to map over
you are not mapping over anything
in fact you're retruning a transducer
oke, what I try to do it map over at this moment as input [1] , make a url of it and read it and take the body part out of it
read_the_numbers works fine. That one gives as output : (nl-SK-A-1718 nl-SK-C-1368 nl-SK-A-2963 nl-SK-C-2 nl-SK-C-149 nl-SK-A-3841 nl-SK-A-1115 nl-SK-C-216 nl-SK-C-211 nl-SK-A-799)
Yeah that thread last macro ->>
is probably trying to take the results of calling client/get
with no arguments and threading that to the format
call that also takes no args!
clojure
(map (fn [id]
(let [url (str " " id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
response (client/get url {:as :json})]))
list_of_ids)
So if you changed the code above to return response you should see something
(map (fn [id]
(let [url (str " " id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
response (client/get url {:as :json})]
response))
list_of_ids)
(map (fn [id]
(let [url (str " " id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
response (client/get url {:as :json})
body (:body response)
art-objects (:artObjects body)]
art-objects))
list_of_ids)
that's one step at a time
Basically you would need to do the bit that formats the url and get's the response in one call then thread that through :body
and :artObjects
you couldn't do the url threaded into the get using ->>
as the get takes the url as the first argument not the last
well you can ... if you create a fn that takes one argument for the get then that single arg is both first and last
@agile_geek oke, back to another idea
@roelofw something like this:
(map (fn [id]
(-->
(str " " id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
#(client/get % {:as :json})
:body
:artObjects))
list_of_ids)
and in fact you could use the thread first macro too ->
(map (fn [id]
(->
(str " " id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
(client/get {:as :json})
:body
:artObjects))
list_of_ids)
Thanks, Can I also do something like this under the :artObjects part description = :description
to get the description
If :description
is a key inside the :artObjects
map
you would have to use a let to store it locally
you can't do variable assignment outside of a let of a def
(map (fn [id]
(let [art-objects (->
(str " " id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
(client/get {:as :json})
:body
:artObjects)
description (:description art-objects)
another-thing (:another-thing art-objects)]
{:description description :another-thing another-thing}))
list_of_ids)
or you could use assoc's threaded together to build a map
this will produce a sequence of maps not a map of maps
if you wanted a map of maps you would need to use somehting like id as the key of each map for each object and use reduce
not map
to build your outer map
oke, I thought this would be nice : '( ( 1 "my painting" "My first made painting") ( 2 "rubbisch" "rubbisch made by me ") )
I'm typing my code directly into slack so my parens might get unbalanced!
@agile_geek but does the description part not be a part of the macro ?
it can be but then you will get only the description
and you've lost the art objects so you can't go back into that to get anything else that's in art objects
I think you want other stuff?
i.e. art objects has multiple key-values you want?
and I have to make another function which also has to be a part of the maps of maps or whatever the output will be
so you need to hang on to art objects and use it more than once, hence I put it in a let
or you could use juxt
to destructure all your values at once but that's a bit advanced
just post the code here. we're trying to diagnose what i think are just arguments out of place without seeing the code
code :
(ns proefopdracht.core
(require [clj-http.client :as client])
(require [cheshire.core :refer :all]))
(defn read_numbers
"Reads the ids of the paintings"
[]
(->> (client/get "" {:as :json})
:body
:artObjects
(map :id)))
(defn read-data_painting
"Reads the title, description, date , collection, colors and url of a image"
[id_list]
(map (fn [id]
(let [art-objects
(-> (str " " id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
(client/get {:as :json})
:body
:artObjects)
description (:description art-objects)])) id_list))
(println (read-data_painting ["SK-A-1718" ]))
you're not returning anything from the fn called by map
so after the ]
in the line with description on it you need to return the description
also check those parens balance
[id description]
in a vector
or '(id description)
for a list
note the quote
I would use a vector
also, since you are having trouble with seeing what your functions are returning, don't use anonymous functions
You will get a lazy sequence from the map
fn so it will produce a sequence of vectors, one vector in the sequence for each artObject
@dpsutton @agile_geek both many many thanks
I have now a keyword which is under 2 : so first I have to find the keyword dating and within it I have to find the keyword year . Can I do something like this : (date :dating :year)
More idiomatically you can call (get-in art-objects [:dating :year])
It's making an http call to retrieve data that will be slow
And you're making that call for every od
Could I make it work that I take the numbers out of the first request and then place them in a sort of quee so that the data is downloaded in parallel
If the first call returns the art objects with their IDs and description you can just make one call
I'd need to know more about what the http api was to determine best solution
I'm not at a computer atm (on the bus!) but you can post a link to the problem here. I wouldn't upload the whole file to slack
I can look at it tonight if I get time
no problem. Here is the link : http://rijksmuseum.github.io/
:sleeping_accommodation:
maybe later I can make it work as my first web app in for example luminus or just like so many people look for my own parts like ring
@agile_geek do you also need the code I have so far ?
Could do - just paste it to refheap
Looking at those APIs I can't see an obvious way to minimise the number of calls.
oke,. here is my code so far: https://www.refheap.com/124020
Also be aware that map
is lazy so if you called this in a program rather than from a repl it may do nothing. The repl forces the sequence from map
to be realised. To force it to realise you may need to wrap the top map
call in a doall
If you have to make a separate call for each ID, the simplest way to speed it up might be pmap instead of map; still makes all the calls, but does them in parallel.
agile_geek’s note about the result being lazy still holds.
@tokenshift true. Although I usually use core.aync pipelines to parallelise this stuff in real projects
@roelofw well done. Try pmap where map is in your read-data_paintings
fn
Btw it's idiomatic to use hyphens - between words in fn names not underscores _
I am receiving a Clojure Multimethod error in one of the Clojure Brave and True examples. I've posted the question on Stack Overflow in case others have run into a similar question. If anyone has any ideas about how I can fix my code, I'd certainly appreciate it: http://stackoverflow.com/questions/40875042/clojure-multimethod-error-no-method-in-multimethod-for-dispatch-value-null
with map : "Elapsed time: 2045.2662 msecs" with pmap : "Elapsed time: 689.110806 msecs"
All with the addition of a p
😄
Core.async is reasonably advanced.
I suspect what u have is about as good as you'll get for now. Http calls are slow.
It's fetching data across internet. A faster connection makes most difference
oke, I found this example :
(client/get ""
{:async? true}
;; respond callback
(fn [response] (println "response is:" response))
;; raise callback
(fn [exception] (println "exception message is: " (.getMessage exception))))
That's not going to speed up your problem
It is useful if you want to do something else in the same thread while waiting for the response
But of you need the response it has to return from the server and that's dependent on the speed of network and the server
pmap is parallelising some of the calls so best you can do. I.e. calls several at same time in different threads
The core.async pipelines I mentioned is just a more flexible and scalable version of same thing.
It allows tuning of number of parallel threads and composing transducers which are easy to test in isolation if you've a very complex pipeline of transformations. But you don't need the extra complexity of core.async right now
Any way got to go
@agile_geek thanks
@seancorfield I have this so far with a lot of help. Does this look good : https://www.refheap.com/124020
ans maybe later adding some sort of pagination. This is some 470 pages long if I take all the images
@roelofw: I'll take a look when I'm back at my computer. For now I'd just note that _ in names is not idiomatic - use - instead.
@seancorfield is changed
@roelofw: is there a new URL? I still see _ in names at the other URL.
id_list
, also it should be :require
in ns (you'll get an error on Clojure 1.9 for require
). More later once I see it on a bigger screen
New code :
(ns proefopdracht.core
(:require [clj-http.client :as client])
)
(defn read-numbers
"Reads the ids of the paintings"
[]
(->> (client/get "" {:as :json})
:body
:artObjects
(map :id)))
(defn read-data-painting
"Reads the title, description, date , collection, colors and url of a image"
[id-list]
(pmap (fn [id]
(let [art-objects (-> (str " " id "?key=14OGzuak&format=json&type=schilderij&toppieces=True")
(client/get {:as :json} )
:body
:artObject)
description (:description art-objects)
date (get-in art-objects [:dating :year])
collectie (first (:objectCollection art-objects))
colors (:colors art-objects)
]
{:id id :description description :date date :collectie collectie :colors colors } )) id-list))
(time(println (read-data-painting (read-numbers))))