Morning

Morning @dominicm and indeed EVERYONE! I have a chest infection (minor, acute, just got to wait it out). I am SO HAPPY!

:lemsip:

on slack, there are no limits

moin moin

månmån

morning

get well soon @maleghast

Morning folks.

Morning :wind_blowing_face:

Bit blustery this morning

morning - I too end up mixing kebab-case and snake_case throughout my code, much to my own annoyance (I must have a word with myself)

I think ideally I’d have wrapped all my db calls in a treewalk -> camel-snake-kebab thingy

(and all the api bits that interoperate with other lesser languages)

i mostly stopped using :keys and switched to long-hand destructuring, so (let [{con-foo-bar :foo_bar :as con} conversation] ...)

…argh! but :keys is sooo convenient

maybe that’s the best way

it turned out, in my codebase, that quite often i will destructure a few different objects in the same fn and that it was really convenient to put a prefix on the bound names anyway, so :keys wasn't so convenient anymore

I’ve certainly had one or two frustrating moments with typing the wrong one - especially once IntelliJ gets both variants into it’s indices and suggests them both

destructuring is quite an extensive mini-lang though - so I’ve tried to keep it simple, but I think I probably should be leaning more on that

i tend to prefer destructuring at the start of a block over keyword access and get-in - the intent is more explicit, the code using simpler names easier to read, and there's only one place (the destructure) you can make a typo without the compiler calling you out

morning all!

Hello All - thanks @otfrom

Right, so, different enlive question… Anyone know a good way of stripping out all the pointless whitespace in a web-page before using enlive to turn the text into a list of nested maps..?

The stone age web-pages I am scraping have whitespace that really f**ks things up in them.

(I could create regexes for all the different combinations of spaces, tabs and “\n”, I realise that , but if there were a way to tell enlive / Clojure to ignore the whitespace, that I have not yet been able to fathom, that would be so much more awesome and less cumbersome…

:thinking_face:

I'm not sure i understand

i thought enlive works by using the id tags and things in the html

https://github.com/cgrand/enlive#selectors

to get rid of whitespace you could use https://clojuredocs.org/clojure.string/trim

@guy - It does, but if you have HTML that has no #ids and .classes you rely on pure DOM through enlive and then if there is whitespace that gets interpreted as DOM elements and appears inside the data structure that gets created and effectively (seems to to me anyway) screws up the ability to do things like:

[:body [:table {:tr [:td]]]]

which should__ get you all the <td>s in the first <table>, which is what I need, but for some reason it doesn’t and when I inspect the data structure(s) there are lots of entries inside the nested maps for things like ” \n” that enlive detects but does not really know what to do with.

I am fairly sure that if the <td>s I wanted all had a class on them, I would be able to get them really easily. As it is I am doing a lot of hoop jumping to use a bit of enlive’s clever selector stuff along with more traditional data manipulation(s) and the result looks fragile to me, and certainly not very re-usable or configurable.

e.g.

(defn get-link-hrefs
  [html-snippet acc]
  (reduce
   (fn [acc subcoll]
     (let [href (:href (:attrs (first (:content subcoll))))]
      (if
        (not (nil? href))
        (conj acc href)
        acc)))
   acc
   html-snippet))

(defn get-variable-by-country-list
  []
  (-> @(http/get root-url)
    :body
    bs/to-string
    html/html-snippet
    first
    :content
    (html/select [:body [:table]])
    first
    :content
    rest
    (html/select [:tr [:td]])
    (get-link-hrefs '())))

This ^^ works, but I should__ be able to do:

(defn get-link-hrefs
  [html-snippet acc]
  (reduce
   (fn [acc subcoll]
     (let [href (:href (:attrs (first (:content subcoll))))]
      (if
        (not (nil? href))
        (conj acc href)
        acc)))
   acc
   html-snippet))

(defn get-variable-by-country-list
  []
  (-> @(http/get root-url)
    :body
    bs/to-string
    html/html-snippet
    first
    :content
    (html/select [:body [:table [:tr [:td]]]])
    (get-link-hrefs '())))

(some of the <td>s don’t have an <a> inside them, hence the reducing fn)

I should really be able to do this:

(-> @(http/get root-url)
    :body
    bs/to-string
    html/html-snippet
    first
    :content
    (html/select [:body [:table [:tr [:td [:a (attrs? :href)]]]]]))

but that REALLY doesn’t work… 😞

I've never known enlive to break for whitespace...

So I have a clunky solution, but also a reasonably informed theory that if I could get the HTTP response :body to have no whitespace before I pass it to html/html-snippet then it might work better, and be less clunky…

can u give me a sample of the data

the HTML?

sure

PM

is fine

https://crudata.uea.ac.uk/cru/data/hrg/cru_ts_4.01/crucy.1709191757.v4.01/countries/

oops, sorry, pasted before I read the “PM” bit…

nah thats fine

i thought u might just stick in ` thats all xD

I am using aleph to make the request, in case that’s relevant…

are u missing a middle element?

cos that has tbody

urs goes table, tr, td

When I tried to reference the tbody element in my selector I got an empty list…

kk

But I am prepared to try it again…

nah its cool man

let me try it at lunch break

when i used it, it was pretty simple just specificing the keywords in a vector xD

ok back in abit

just to check, when I read enlive readme again, it looked like the correct selector was [:html :> :body :> :table :> :tbody] , no?

@dominicm - I saw that, but I also saw the nested vectors thing…

Let me try that^^

Well, this->

(-> @(http/get root-url)
    :body
    bs/to-string
    html/html-snippet
    (html/select [:body :> :table :> :tbody :> :tr :> :td :> :a]))

as does this:

(-> @(http/get root-url)
    :body
    bs/to-string
    html/html-snippet
    (html/select [:body [:table [:tbody [:tr [:td [:a [:attrs [:href]]]]]]]]))

I also tried the “:>” approach all the way down to :attrs “:>” :href - still an empty list

(quotes to prevent emojii swapping)

Based on the selector I can get out of Chrome Dev Tools, I would expect either to work… When I looked at the contents of the data structure that was being created by enlive using html-snippet, I noticed that there were these weird maps inside the list(s) that did not have the same structure as everything else and were clearly trying to express whitespace.

That’s when I wondered if this would work better with the whitespace cleared out first, based on similar things having similar effects in other languages / environments, when parsing XHTML and XML in my dim and distant past.

Although, that may be bollocks, as this:

(-> @(http/get root-url)
    :body
    bs/to-string
    html/html-snippet
    (html/select [:body :> :table :> :tr :> :td :> :a]))

what does [:body 😆 :table 😆 :tbody]

give u

@guy see my last

[:body :> :table :> :tbody]

kk

kk so u know how 😆 works right?

':>'

Not sure how to get the hrefs out though as this:

(-> @(http/get root-url)
    :body
    bs/to-string
    html/html-snippet
    (html/select [:body :> :table :> :tr :> :td :> :a :> :attrs :> :href]))

@guy, no

ok so as far as i understand it, it trys to match anywhere in the document with it

so try

(html/select [ :table :> :tr :> :td :> :a :> :attrs :> :href]))

because it will search for table

then work from that

Yeah, that is an empty list.

😂

It will work up to “🅰️” I reckon…

kk

Yeah, I don’t need :body you are quite right

yeah i did some xml stuff with enlive and xpath

it works in a similar way i believe

I can just reduce the snippet of all the <a> tags if I have to - would be nice to get all the href with enlive syntax though

i guess it depends on whats in the html doc

do u really wanna just get all the a tags?

what if u get the wrong ones

No, I want all the <a> tags inside the table

yeah

I know that I want all of them

u could try just :td :a

:td :> :a 

I will, hang on…

and see what u get

can i see what that returns?

Sure, hang on…

:thumbsup:

How do I get an in-buffer evaluation result into the clipboard..?

mmm

could just do a screen shot

:x

okie dokie…

im not sure what ide ur using sorry

Emacs + CIDER

can't you just select it?

i seem to

kk

kewl

that looks good

so u have all the maps in it

what did u want out?

({:tag :a, :attrs {:href cld}, :content (cld)} {:tag :a, :attrs {:href dtr}, :content (dtr)} {:tag :a, :attrs {:href frs}, :content (frs)} {:tag :a, :attrs {:href pet}, :content (pet)} {:tag :a, :attrs {:href pre}, :content (pre)} {:tag :a, :attrs {:href tmn}, :content (tmn)} {:tag :a, :attrs {:href tmp}, :content (tmp)} {:tag :a, :attrs {:href tmx}, :content (tmx)} {:tag :a, :attrs {:href vap}, :content (vap)} {:tag :a, :attrs {:href wet}, :content (wet)})

the :content

or the :href

well, in this case either would do as they are the same, but I thought it better to get the :href

you should be able to do :attrs

let me check the docs

ok

thanks very much 🙂

ok so i guess

i cant see it in the docs but

i guess u cant select the attrs out as its not the :tag

so each map is made up of :tag, :atts and :content

so i think the selection vector takes the tags

so u might need to just map over those maps doing (map #(-> % :attrs :href) maps)

like the docs say

So [:p (attr? :lang)] is going to match any elements with a lang attribute inside a :p element. On the other hand, [[:p (attr? :lang)]] is going to match any p with a lang attribute.

so to me it seems like they dont use attrs as a tag but i might be wrong

Yeah, that was my reading of it, hey ho 😞

Thanks anyway

but try

(attr? :href)

find attributes with :href in

(html/select [ :table :> :tr :> :td :> :a :> (attr? :href)]))

something like that

give that a go

cos ur saying the last step should be

attribute with :href inside

just like they did

(attr? :lang)

https://github.com/cgrand/enlive#selectors

this is what im reading

I get a horrible “can’t resolve symbol” error when I do that

But, I do have a really great function now that you helped me to build, @guy so thanks very much for that 🙂

(defn get-link-hrefs
  [html-snippet acc]
  (reduce
   (fn [acc subcoll]
     (let [href (:href (:attrs subcoll))]
      (if
        (not (nil? href))
        (conj acc href)
        acc)))
   acc
   html-snippet))

(defn get-cru-document-links
  [url]
  (-> @(http/get url)
    :body
    bs/to-string
    html/html-snippet
    (html/select [:td :> :a])
    (get-link-hrefs '())
    (->>
     (map #(str url %)))))

This ^^ works perfectly 🙂

:thumbsup:

sorry i couldnt help more

Thanks, really no apology needed - you were loads of help 🙂

🙂

https://www.postgresql.org/about/news/1786/

PostgreSQL 10 released ☝️

Oooh!

almost a good database for the cloud

still the best RDBMS

and the only proper one of two (Oracle being the other)

what makes it bad for cloud?

doesn't handle node failure well (as sort of trailed in their changelog)

it is great if you can keep a box from falling over

(would be happy to be proved wrong about multi-node resilience of postgres as it is a great db for relational/time series/gis)

“gis”?

why is datomic not an RDBMS?

actually i have a more interesting question for you all. when using spec, if you want to spec out functions, is it preferable to use s/fdef or to put an s/valid? check in the :pre and :post of your functions? I feel like fdef is preferable, but i don't like writing two defs for every fn

I've been thinking about the same thing all day

fdef is useful for tests as well

whereas pre and post dont give you much test wise

but from what i heard, you wouldnt want instrument'-ing' on in production (performance wise)

so then pre and post might be better

😞

meh, depends what your performance profile is like

😄

for my use case i'll probably never have enough going on for it to be a problem

then i would argue fdef is better

it adds extra information to docs, it helps out with tests and production issues

that is my gut feeling, but i hate writing two forms for each fn

you write two fn's but less stuff in tests, is my gut feeling

especially if you add generators too the fdef's too

yeah that may be true

but then again, ive only started using it too

well the specs all have generators

sure but for specific things

is what i mean sorry

@peterwestmacott geographical information system

like that email generator you wrote

you could write the fdef but then get the spec in code if you want to assert explicitly

(s/assert
   (:ret (s/get-spec `this-fn))
   body)

yeah sure

i think fdef is a winner

in case anybody is interested i wrote specs and gens for emails and urls yesterday https://gist.github.com/conan/2edca210999b96ad26d38c1ee96dfe40

@conan "foo@bar" is a valid email too 😉

@conan That's a very restrictive spec for email. Here's a much more accurate regex for emails (not claiming it's perfect):

(def email-regex
  "Sophisticated regex for validating an email address."
  (re-pattern
   (str "(([^<>()\\[\\]\\\\.,;:\\s@\"]+(\\.[^<>()\\[\\]\\\\.,;:\\s@\"]+)*)|"
        "(\".+\"))@((\\[[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\])|"
        "(([a-zA-Z\\-0-9]+\\.)+[a-zA-Z]{2,}))")))

We use test.chucks regex generator to produce sample test emails from that.

@sundarj I don't think foo@bar is a valid email address -- unless you meant "foo@bar"@quux.com (which certainly is valid)?

<input type="email"> in HTML5 accepts it - not every domain has a tld (though most do)

it doesn't seem that restrictive to require an @and a .

maybe the tld isn't required, in which case only an @ would be necessary, but i've never seen an email without one nor had an error in a system i've validated in this way. i'm happy to wait until i do to add support for it, as i think the benefit of requiring everyone to type it in correctly outweighs the loss of support for users with email addresses that do not have tlds

that's fair

but removing the requirement for a . is the only way i can see to make it less restrictive, if i remove the requirement for an @ as well then i'm just validating that it's a string

i believe Sean meant 'permissive'

oh yes, it is very permissive

but my generator only makes alphanumerics, go figure

=D

the url one is more useful, as i couldn't find a good existing one. there's a uri generator in clojure.spec, but it's just this:

(fmap #(java.net.URI/create (str "http://" % ".com")) (uuid))

that's going to make only http://<uuid>.com variations

not a wide set of urls

they clearly did that because they don't have access to the wonderful cemerick/url

anyway, hopefully it'll be helpful, i'll be using it in production soon and it's taught me all about spec well enough

I meant restrictive because you only allow alphanumeric characters.

Oh no, I misread. You only generate alphanumeric. Got it.

yep

to be honest i couldn't be bothered to generate all allowed internet characters

it'd be easy to update that bit if it matters to you

We like the fact our spec is symmetric -- it generates what it accepts and vice versa.

In particular, the generation of such wild addresses is a good test for other parts of the system to make sure they don't bake in incorrect assumptions about the structure of an email address.

@sundarj Can you point me at any domain names that do not have a TLD? I am surprised that is legal (despite HTML5's validator accepting it).

Hmmm… Anyone got any recommendations on how to turn a list of lists, where the inner lists are pairs of strings, into a map?

I realise that if it was a seq of vector pairs this would just work with (into {} pairs)

(although it might need to be keyword / string in each pair..?

(into {} (map vec) pairs)?

I will try, thanks @seancorfield

Brilliant!

Thanks @seancorfield

I'm slowly beginning to internalize the use of transducers... 😐

Did I use a transducer by doing (into [] (map vec) pairs) ..?

As in (map vec) would return a transducer

?

Yup!

Wowsers 🙂

Thanks @sundarj