Fork me on GitHub
#beginners
<
2016-12-30
>
seancorfield00:12:41

Given that you’re looping until you have a vector of Boolean with 35,651,584 entries in it, I’m not too surprised you’re hitting the default JVM memory limit 🙂

seancorfield00:12:42

(esp. with all the concat, rseq, and partition operations going on 🙂 )

tosh00:12:59

naive q: would a string of that size use more or less or similar amounts of ram?

tosh00:12:15

with the 4 gig memory flag part2 takes about 5minutes to complete for me on my 2yo 12” macbook

tosh00:12:31

whereas iirc the python version by peter norvig just ran a few seconds

seancorfield00:12:48

A 35MB string vs a vector of Boolean with 35MB elements in it? The latter is going to be bigger than 35MB.

seancorfield00:12:06

(that would be my naïve answer — I haven’t verified that!)

roelof07:12:32

I get this error message : IllegalArgumentException Key must be integer clojure.lang.APersistentVector.invoke (APersistentVector.java:292) on this repl

(s/explain ::artobject-objectNumber [{:artObjects{:objectNumber "sk-c-5"}}]) 
with these specs :
;; the specs files for input validation

(def id-regex #"[a-z]{2}-[a-z]-\d{1,2}")

(s/def ::id-type (s/and string? #(re-matches id-regex %)))

(s/def ::objectNumber ::id-type)

(s/def ::objectNumber-list (s/coll-of
                             (s/keys :req-un [::objectNumber])
                             :kind vector?))

(s/def ::artobject-objectNumber (s/coll-of
                                  (::objectNumber-list :req-un [::artObjects])
                                  :kind vector?)) 

roelof07:12:41

what went wrong this time

seako08:12:57

@roelof you might want to try (s/exercise ::artobject-objectNumber) and compare what’s generated with your expectation

seako08:12:24

hm, that’s too bad. (s/exercise) wasn’t helpful since in this case since it wasn’t able to generate data satisfying the specs.

seako08:12:22

maybe this will be helpful

;; this spec matches a map containing the key :artObject
(s/def ::art-object (s/keys :req-un [::artObject]))

 (s/explain ::art-object {:artObject {}})
;;=> Success!

;; this spec matches a collection of ::art-object 
(s/def ::list-of-art-objects (s/coll-of ::art-object :kind vector?))

(s/explain ::list-of-art-objects [{:artObject {}}])
;;=> Success!

seako08:12:29

and here’s how you could combine specs together to create a nested spec

;; a spec that matches a map with the key :objectNumber
(s/def ::object-number (s/keys :req-un [::objectNumber]))

;;  a spec that matches a map containing the key :artObject that maps keywords to values that match the ::object-number spec

(s/def ::nested-spec (s/and (s/keys :req-un [::artObject]) (s/map-of keyword? ::object-number)))

(s/explain ::nested-spec {:artObject {:objectNumber "sk-c-5"}})
;;=> Success!

seako08:12:25

and here’s how you could combine the nested spec with a collection spec

;; here we don’t specify that the collection must be a vector - it just needs to be a collection of values that satisfy the ::nested-spec
(s/def ::list-of-nested (s/coll-of ::nested-spec))

;; so it will accept a vector
(s/explain ::list-of-nested [{:artObject {:objectNumber "sk-c-5"}}])
;;=> Success!

;; and it will also accept a list
(s/explain ::list-of-nested '({:artObject {:objectNumber "sk-c-5"}}))
;;=> Success!

roelof08:12:45

@seako here it do not work :

(s/explain ::objectNumber-list [{:artObject {:objectNumber "sk-c-5"}}])
Exception Unable to resolve spec: :paintings2.api-get/object-number  clojure.spec/reg-resolve! (spec.clj:68)  

roelof08:12:24

with this code :

(def id-regex #"[a-z]{2}-[a-z]-\d{1,2}")

(s/def ::id-type (s/and string? #(re-matches id-regex %)))

(s/def ::objectNumber ::id-type)

(s/def ::objectNumber-list (s/coll-of
                             (s/and (s/keys :req-un [::artObject]) (s/map-of keyword? ::object-number))
                             :kind vector?)) 

seako09:12:13

oh, that’s because i defined a new spec called ::object-number that is not in that code

roelof09:12:56

Thanks, now it works fine :

=> #object[clojure.lang.Namespace 0x1c44c16 "paintings2.api-get"]
(s/explain ::objectNumber-list [{:artObject {:objectNumber "sk-c-5"}}])
Success!
=> nil
(s/explain ::objectNumber-list [{:artObject {:objectNumber "sk-c"}}])
In: [0 :artObject 1 :objectNumber] val: "sk-c" fails spec: :paintings2.api-get/id-type at: [1 :objectNumber] predicate: (re-matches id-regex %)
=> nil 

roelof09:12:20

now I can do the same I think when I want to check also for artObjects as the first one

roelof09:12:38

I want to make this a success : (s/explain ::objectNumber-list [{:body{:artObjects{:objectNumber "sk-c-5"}}}])

seako09:12:04

i would recommend instead defining a new spec that matches a collection of values that match your specs

seako09:12:04

the general strategy is going to be similar - write specs for the values nested inside the structure and then combine them together with the specs for collections and maps

roelof09:12:31

yep, the same way as you did

roelof11:12:15

still one problem I cannot fix

roelof11:12:45

I have this in repl : (s/explain ::objectNumber-list2 [{:body{:artObject {:objectNumber "sk-c-5"}}}])

roelof11:12:11

with these specs :

(def id-regex #"[a-z]{2}-[a-z]-\d{1,2}")

(s/def ::id-type (s/and string? #(re-matches id-regex %)))

(s/def ::objectNumber ::id-type)

(s/def ::objectNumber-check (s/keys :req-un [::objectNumber]))

(s/def ::objectNumber-list (s/coll-of
                             (s/and (s/keys :req-un [::artObject])  (s/map-of keyword? ::objectNumber-check))
                             :kind vector?))

(s/def ::objectNumber-list2 (s/coll-of
                             (s/and (s/keys :req-un [::body])  (s/map-of keyword? ::objectNumber-list))
                             :kind vector?))  

roelof11:12:22

now the specs fails with this message :

In: [0 :body 1] val: {:artObject {:objectNumber "sk-c-5"}} fails spec: :paintings2.api-get/objectNumber-list at: [1] predicate: vector?
=> nil   

roelof11:12:35

so I deleted the kind vector? part but then it fails with this :

In: [0 :body 1 0] val: [:artObject {:objectNumber "sk-c-5"}] fails spec: :paintings2.api-get/objectNumber-list at: [1] predicate: map? 

seako11:12:31

the problem is in how (s/map-of keyword? ::objectNumber-list) is being used

roelof11:12:57

oke, then I have to study your example better

roelof11:12:36

I thought it would work because of this : (s/and (s/keys :req-un [::artObject]) (s/map-of keyword? ::objectNumber-check))

seako11:12:47

also, i’m not sure if ::objectNumber-check behaves the way you want

seako11:12:59

i think you are beginning to see the light

seako11:12:15

do you need/want that s/coll-of?

seako11:12:31

that’s wrapped around (s/and (s/keys :req-un [::artObject]) (s/map-of keyword?

roelof11:12:02

yes, that behaves well, I checked that before I added the check for :body

roelof11:12:06

See here :

(s/explain ::objectNumber-list [{:artObject {:objectNumber "sk-c-5"}}])
Success!
=> nil  

seako11:12:05

(s/map-of keyword? ::objectNumber-list) expects a map that looks like {:any-keyword [{:artObject {:objectNumber “sk-c-5”}}]

roelof11:12:23

yep, and any-keyword is here :body

seako11:12:12

but your data doesn’t have the shape you’ve written a spec for

roelof11:12:28

chips. I see the problem . it looks like this : [:any-keyword { ...... } ]

roelof11:12:47

so it not a map-of

seako11:12:05

the outer part is a coll-of the inner part is a map of

roelof11:12:45

@seako you mean I have to change map-of with coll-of ??

seako11:12:02

in

(s/def ::objectNumber-list2 (s/coll-of
                             (s/and (s/keys :req-un [::body])  (s/map-of keyword? ::objectNumber-list))
                             :kind vector?))  

roelof11:12:08

nope , I cannot do this : (s/coll-of keyword? ::objectNumber-list))

seako11:12:37

replace ::objectNumber-list with (s/and (s/keys :req-un [::artObject]) (s/map-of keyword? ::objectNumber-check))

seako11:12:15

(s/def ::objectNumber-list2 (s/coll-of
                             (s/and (s/keys :req-un [::body])  (s/and (s/keys :req-un [::artObject])  (s/map-of keyword? ::objectNumber-check))
                             :kind vector?))  

seako11:12:25

this spec matches a collection of nested maps

seako11:12:39

the spec you wrote matches a collection of maps containing collections of maps

roelof11:12:25

Thanks, it worked well

(s/explain ::objectNumber-list [{:body{:artObject {:objectNumber "sk-c-5"}}}])
Success!
=> nil
(s/explain ::objectNumber-list [{:body{:artObject {:objectNumber "sk-c"}}}])
In: [0 :body 1 :artObject 1 :objectNumber] val: "sk-c" fails spec: :paintings2.api-get/id-type at: [1 1 :objectNumber] predicate: (re-matches id-regex %)
=> nil  
`

roelof11:12:01

and here :

(s/explain ::objectNumber-list [{:artObject {:objectNumber "sk-c-5"}}])
In: [0] val: {:artObject {:objectNumber "sk-c-5"}} fails spec: :paintings2.api-get/objectNumber-list predicate: (contains? % :body)
=> nil 

roelof11:12:15

im a happy man now

roelof11:12:19

thanks for the help

seako11:12:38

i’m happy you’re happy! glad to help.

roelof11:12:41

another problem . I have this code :

(defn read-numbers
  "Reads the ids of the paintings"
  [response]
  (if (s/conform ::objectNumber-list response)
    (->> (:body response)
         :artObjects
         (map :objectNumber))))
 

roelof11:12:10

but when I do (read-numbers [{:artObject {:objectNumber "sk-c-5"}}])

roelof11:12:20

i see a empty output

roelof11:12:25

a, I see the problem : ` (s/conform ::objectNumber-list [{:body{:artObject {:objectNumber "sk-c-5"}}}]) => [{:body {:artObject {:objectNumber "sk-c-5"}}}] `

roelof11:12:15

so I have to find a way something happens when invalid

roelof11:12:27

first lunch

roelof12:12:30

(s/explain ::objectNumber-list [{:body{:artObjects {:objectNumber "sk-c-5"}}}])
Success!
=> nil 

roelof12:12:50

(read-numbers [{:body{:artObjects {:objectNumber "sk-c-5"}}}])
=> ()  

roelof12:12:12

with this code :

(defn read-numbers
  "Reads the ids of the paintings"
  [response]
  (let [response-checked (s/conform ::objectNumber-list response)]
  (if (s/invalid? response-checked)
    (layout/render "error.html" {:message "something was wrong with reading the ids"})
    (->> (:body response)
         :artObjects
         (map :objectNumber)))))

roelof12:12:31

wierd, (:body response) does not have any output

seako12:12:30

what is the value of response?

seako12:12:47

(i’m asking rhetorically)

roelof12:12:34

I thought something like this [{:body{:artObjects {:objectNumber "sk-c-5"}}}]

roelof12:12:46

but im checking now to be sure

henriklundahl12:12:38

Doesn't it look like this instead: {:body {:artObjects [{:objectNumber "sk-c-5"}]}}?

roelof12:12:56

@henriklundahl yep, you are right :

{:status 200,
 :headers {"Date" "Fri, 30 Dec 2016 12:34:44 GMT",
           "Cache-Control" "no-store, must-revalidate, no-cache",
           "Pragma" "no-cache",
           "Content-Type" "application/json; charset=utf-8",
           "Access-Control-Allow-Origin" "*",
           "X-Frame-Options" "DENY",
           "Content-Length" "12707",
           "Connection" "close"},
 :body {:elapsedMilliseconds 43,
        :count 4746,
        :artObjects [{:   

roelof12:12:08

so I have to rewrite my specs

roelof12:12:42

its not a map or a vector but a {}

henriklundahl12:12:31

You should only need to change ::artObjects and the top one.

roelof12:12:43

sorry, I do not understand , @henriklundahl

henriklundahl12:12:25

Ah, I thought you had an ::artObjects spec.

henriklundahl12:12:38

Can you write all the specs you have currently?

roelof12:12:23

@henriklundahl here you have :

`(def id-regex #"[a-z]{2}-[a-z]-\d{1,2}")

(s/def ::id-type (s/and string? #(re-matches id-regex %)))

(s/def ::objectNumber ::id-type)

(s/def ::objectNumber-check (s/keys :req-un [::objectNumber]))

(s/def ::objectNumber-list (s/coll-of
                             (s/and (s/keys :req-un [::body]) (s/map-of keyword?
                                                                        (s/and (s/keys :req-un [::artObjects])
                                                                               (s/map-of keyword? ::objectNumber-check))))
                             )) 

henriklundahl12:12:00

::artObjects is mentioned in ::objectNumber-list, but I can't the definition of it?

henriklundahl12:12:57

I think ::objectNumber-check should be renamed ::artObject.

roelof12:12:26

correct, when I do [ { ...... } ] it all works well

roelof12:12:45

Its no problem renaming the function

roelof12:12:49

I will do that

henriklundahl12:12:01

And then the spec for ::artObjects should be (s/def ::artObjects (s/coll-of ::artObject)).

henriklundahl12:12:33

And then you add a spec for ::body: (s/def ::body (s/keys :req-un [::artObjects])).

henriklundahl12:12:25

Finally, add a spec for ::response: (s/def ::response (s/keys :req-un [::body])).

henriklundahl12:12:50

Then you validate against the ::reponse spec.

roelof12:12:17

oke, so the objectNumber-list can be deleted ?

henriklundahl12:12:31

I believe so, yes.

roelof12:12:38

Still the same output :

(s/explain ::response {:body{:artObjects {:objectNumber "sk-c-5"}}})
In: [:body :artObjects 0] val: [:objectNumber "sk-c-5"] fails spec: :paintings2.api-get/artObject at: [:body :artObjects] predicate: map?
=> nil  

henriklundahl12:12:10

Don't forget to change the :artObjects value to a vector.

roelof12:12:55

Hmm, why, that is a response from a external api

roelof12:12:03

that I try to validate

roelof12:12:44

a, I see, in the orginal response artObject is a vector

roelof13:12:34

yes, success

roelof13:12:39

(s/explain ::response {:body{:artObjects [{:objectNumber "sk-c-5"}]}}) 
Success! 

henriklundahl13:12:13

::id-type could be inlined into ::objectNumber, so you would have the specs ::objectNumber, ::artObject, ::artObjects, ::body and ::reponse.

roelof13:12:13

and the function is also doing what I suppose to do

roelof13:12:50

@henriklundahl you mean it could be this

(s/def ::objectNumber (s/and string? #(re-matches id-regex %)) ) 

henriklundahl13:12:34

And you'll end up with five very simple specs.

roelof13:12:23

6 if you count the regex with it

henriklundahl13:12:59

That's not a spec. 🙂

roelof13:12:12

oke, you win again

roelof13:12:46

thanks, finnaly made my first input validation spec working

henriklundahl13:12:03

That's great. Good luck!

pwalsh14:12:04

Hey all. I’m a fairly experienced developer, fluent in Python and JavaScript. I’ve been reading up on Clojure for some time, and played with it in small doses via Overtone, and just some REPL learning sessions. However, for me the only way to really learn is to build a project, so I think I’ll port a Python library I know well to Clojure. I think the areas I need most initial guidance on are conventions around (i) grouping functions that operate on mutable data structures (something that I’d use classes for in Python), and (ii) library structure in general. Would this be an appropriate list to ask questions around that?

sveri14:12:54

@pwalsh Yes, this is the right channel for beginner questions. My first comment. There are hardly any mutable data structures in clojure. Most of them are immutable and you will create a data flow that creates new data structures via structural sharing mostly.

mingp15:12:10

@pwalsh Which library were you thinking of porting? I'd agree that the default mindset in Clojure is functions which accept and return immutable data, but of course it can be situational depending on what it is you're writing and what you want from it in terms of behavior/performance.

roelof15:12:37

another spec question . I have these specs :

(s/def ::objectNumber (s/and string? #(re-matches id-regex %)))
(s/def ::artObject (s/keys :req-un [::objectNumber]))
(s/def ::artObjects (s/coll-of ::artObject))
(s/def ::body (s/keys :req-un [::artObjects]))
(s/def ::response (s/keys :req-un [::body]))
(s/def ::objectNumbers (s/coll-of ::objectNumber))  

roelof15:12:11

and I do this in repl : (s/explain ::objectNumbers ["SK-C-5"] )

roelof15:12:50

but then I see this output : In: [0] val: "SK-C-5" fails spec: :paintings2.api-get/objectNumber predicate: (re-matches id-regex %)

roelof15:12:06

but why does this fail on the regex.

roelof15:12:37

that one is 2 characters - one character - one of two numbers

sveri15:12:02

@roelof Its hard to tell without knowing what id-regex is

roelof15:12:05

sorry , I thought I had copied that one to

roelof15:12:16

here it is (def id-regex #"[a-z]{2}-[a-z]-\d{1,2}")

sveri15:12:45

there are only regex matchers that will help you

sveri15:12:59

My guess is that [a-z] only covers lowercase characters

roelof15:12:00

I try to validate this : ` ["sk-c-5", "sk-c-6"]

roelof15:12:39

@sveri , chips , you are right

roelof15:12:47

I have to update my regex

tosh16:12:10

@pwalsh I also come from python, curious to follow your progress, please keep us posted here

roelof17:12:42

I have made some input validation

roelof17:12:07

but now when I run my web application I see this error :

An application error occurred on the server. The current custom error settings for this application prevent the details of the application error from being viewed remotely (for security reasons). It could, however, be viewed by browsers running on the local server machine.\r\n 

roelof17:12:47

before I forget , Im working at Windows 10

petr.mensik19:12:32

Hell guys, I came across an interesting issue. I am using migratus for creating DB schemas so I have init.sql file in src/migrations directory. However when I do lein ring uberwar file is not included in the war file (checking with jar tvf target/veit-is.war | fgrep init.sq). When I include this path in project.clj via :resource-paths ["src/migrations"], the file is there but directly in the WEB-INF/classes/init.sql folder and not in migrations directory. Any ideas what is happening there?

petr.mensik19:12:07

Btw I have a very similar project with the same setup and it's working there, init.sql is correctly in WEB-INF/classes/migrations directory.

seancorfield19:12:45

Are you sure :resource-paths in the working project is src/migrations or just src?

seancorfield19:12:23

Resources will be copied into classes using their path relative to the resource paths.

petr.mensik19:12:37

Actually there is none and it's still working

petr.mensik19:12:46

src/migrations is included by defafult

roelof19:12:15

@seancorfield is there a way I can see what id is when I do lein run in this code :

(defn read-image-data [id] (client/get
                             (str "" id "/tiles")
                             {:as :json, :query-params {:format "json", :key (env :key)}})) 

roelof19:12:49

that one or this one :

(defn read-image-data [id] (client/get
                             (str "" id "/tiles")
                             {:as :json, :query-params {:format "json", :key (env :key)}}))

(defn read-image-url
  "Reads the image-url"
  [response]
  (let [art-objects (-> response
                        :body
                        :levels)
        url (filter #(= (:name %) "z4") art-objects)
        tiles (:tiles (first url))
        image (get-in tiles [0 :url])
        ]
    {:tiles image}))
 

roelof19:12:00

are reponsible for a 400 error

kdivringi19:12:38

(defmacro infix2 
 "Takes a list like (1 + 2 * 3 - 2) and transforms to appropriate clojure with op order" 
  [infixed] 
  (reduce 
    (fn [prev [op term]] 
      (if (= op *)
        (concat (butlast prev) [* (last prev) term])
        (concat prev [(op term)])))
    (list + (first infixed))
    (partition 2 (rest infixed))))

kdivringi19:12:54

Hi all, can anyone tell my why I get a nullpointer exception here?

kdivringi19:12:44

I've tried running portions of it in the repl and I can't figure it out

kdivringi19:12:08

Has anyone used partition like that with reduce to work on two elements at a time?

roelof20:12:12

Very wierd . when I change it to

defn read-image-data [id]
  (let [id-checked (s/conform ::objectNumbers id)]
     (if (s/invalid? id-checked)
       (layout/render "error.html" {:message "something was wrong with reading the ids"})
       (client/get
         (str "" id "/tiles")
         {:as :json, :query-params {:format "json", :key (env :key)}}))))  

roelof20:12:32

everythings works well except there are no images schown

kdivringi20:12:15

Ohhh, I suppose the anonymous function is evaluating my arguments? That most be it

kdivringi20:12:26

Are there anonymous macros?

roelof20:12:03

I work on clojure with ISS web server. But schould adding specs and input validation not interfer when I do lein run

roelof20:12:38

Something is changed in my code since adding spec so the site is broken

roelof20:12:36

oke, found a lead

roelof20:12:57

there is something wrong with the read-data function

roelof20:12:21

but why do I see here a explanation here :

;; the specs files for input validation

(def id-regex #"[A-Z]{2}-[A-Z]-\d{1,2}")

; spec whci defines how the objectNumber must look like
(s/def ::objectNumber (s/and string? #(re-matches id-regex %)))

; specs for validatimg input to find rhe ids
(s/def ::artObject (s/keys :req-un [::objectNumber]))
(s/def ::artObjects (s/coll-of ::artObject))
(s/def ::body (s/keys :req-un [::artObjects]))
(s/def ::response (s/keys :req-un [::body]))

; spec to validate the id
(s/def ::objectNumbers (s/coll-of ::objectNumber))

; core functions

(defn read-numbers
  "Reads the ids of the paintings"
  [response]
  (layout/render "error.html" {:message  (s/explain ::response response)})) 

seancorfield22:12:56

@petr.mensik That’s because src is on the classpath — so src/migrations/init.sql is copied over as migrations/init.sql — so remove :resource-paths and you should see the correct behavior.