Fork me on GitHub
#clojure-uk
<
2017-07-10
>
yogidevbear06:07:37

Nice and nice

seancorfield06:07:58

And at this point we're almost finished with Season 2...

thomas07:07:34

morning 😼 mogge

dominicm07:07:27

🌞 Goodbye UK

yogidevbear09:07:11

Where are you off to, @dominicm?

dominicm09:07:52

@yogidevbear gran canaria :)

agile_geek09:07:31

I've literally spent 6-7 hours over the last week filling out prescreening forms for a role I don't really want to take but is best I've got on offer ATM and they've just sent me more paperwork to complete. So I'm in a really foul mood! :thunder_cloud_and_rain: 😠 😖

yogidevbear09:07:30

For hols or work?

thomas09:07:46

@agile_geek sounds horrible, good luck though.

thomas09:07:38

or "sterkte" as we would say in Dutch. (Which translates as strength)

yogidevbear09:07:43

thomas: I won't start typing in Afrikaans to you 😉

yogidevbear09:07:05

Apparently it's like children's Dutch 😂

agile_geek09:07:47

I've another Dutch friend who says Flemish is the equivalent of a west country bumpkin speaking English.

thomas09:07:13

@yogidevbear I can kinda understand Afrikaans, though there might be a few words I don't know.

thomas09:07:47

and yes, Flemish is different in its own way as well. But still (somehow) qualifies as Dutch

dominicm09:07:24

@yogidevbear holiday. Then back in time for euroclj

chrjs10:07:50

Anyone else headed to euroclj? I’ll be about from Wednesday afternoon/evening and looking for friendly faces.

yogidevbear10:07:41

chrjs: I wish I was

otfrom10:07:37

I'm not going and if I'm lucky my face won't be there either

thomas11:07:03

I'll be there. Arriving Wednesday afternoon.

thomas11:07:35

@otfrom we'll let you know bruce

yogidevbear11:07:39

I think he'll figure it out if he looks in the mirror and sees something is missing

chrjs11:07:19

I’ll be sure to alert @otfrom to the presence of his face.

chrjs11:07:20

@U052852ES, I guess you’ll probably be the only lego person present. Easy to identify, but probably hard to spot.

thomas11:07:27

@U4E5W80P7 yes... I am only about 4 cm tall... so easy to miss 😉

dominicm16:07:31

I'm there from Tuesday :)

maleghast11:07:58

I spent a lot of the weekend working with Clojure and I have to say that once again I am reminded of how much fun coding can be when you have powerful / interesting tools at hand…

maleghast11:07:03

Morning Everyone!

maleghast12:07:41

As an idle question, while I am waiting for an email to send (yes my connection is that__ slow), has anyone else had a fair bit of trouble re-using functions that were put together to build maps for adding data to Datomic to use for inserting data into more “traditional” databases..? I mean with specific reference to how maps are not “order guaranteed” so you can’t just call

(values coll)
on them and be sure that the data is in the right order for the query..?

maleghast12:07:31

I ended up writing a little function that reduces over a vector of keys so that I could build a vector of values in the right order… Does this sound like an idiomatic thing to do, or is there a built-in that I am missing that would have solved my issue for me..? (in my defence I was coding without the safety net of an active internet connection, therefore clojuredocs and stack overflow were unavailable to me)

maleghast12:07:41

TLDR; Can one order a map based on a vector of keys without reducing over said vector to produce an ordered collection of values?

maleghast12:07:49

Side note… My functions for preparing data to push into Datomic make use of

(-> ...)
and
(update ...)
quite a lot in order to turn string-based dates into instances and to turn string representations of numbers into yer actual numbers… Is this something that I could__ be using schema or indeed clojure.spec to do in a more elegant fashion..? Example of current situation:
(vector
     (-> (reduce
         (fn [acc vecpair]
          (conj acc vecpair))
         {}
         (concat
          (zipmap (:gsod-measurement-keys ingest-cfg) (map gsod-replace-zero-placeholders (map gsod-trim-flag gsod-rest)))
          (zipmap (:gsod-frshtt-keys ingest-cfg) (gsod-frshtt-to-standard-length gsod-frshtt-value))))
        (update :gsod-measurement/date #(instant/read-instant-timestamp (noaa-date-to-rfc3339 %)))
        (update :gsod-measurement/tempaverage #(Double/parseDouble %))
        (update :gsod-measurement/tempaveragenumments #(Double/parseDouble %))
        (update :gsod-measurement/dewpoint #(Double/parseDouble %))
        (update :gsod-measurement/dewpointnumments #(Double/parseDouble %))
        (update :gsod-measurement/sealevelpressure #(Double/parseDouble %))
        (update :gsod-measurement/sealevelpressurenumments #(Double/parseDouble %))
        (update :gsod-measurement/stationpressure #(Double/parseDouble %))
        (update :gsod-measurement/stationpressurenumments #(Double/parseDouble %))
        (update :gsod-measurement/visibility #(Double/parseDouble %))
        (update :gsod-measurement/visibilitynumments #(Double/parseDouble %))
        (update :gsod-measurement/windspeedaverage #(Double/parseDouble %))
        (update :gsod-measurement/windspeednumments #(Double/parseDouble %))
        (update :gsod-measurement/windspeedmax #(Double/parseDouble %))
        (update :gsod-measurement/gustspeedmax #(Double/parseDouble %))
        (update :gsod-measurement/tempmax #(Double/parseDouble %))
        (update :gsod-measurement/tempmin #(Double/parseDouble %))
        (update :gsod-measurement/precipitation #(Double/parseDouble %))
        (update :gsod-measurement/snowdepth #(Double/parseDouble %))))))

agile_geek13:07:22

Depending on what you're doing you can also look at array-map (`hash-map` is just the default associative data structure there are other options but you need to create them explicitly)

maleghast13:07:21

@agile_geek - I had no idea that there was a sorted version of map… facepalm I will have a look at array-map as well…

agile_geek13:07:20

there are similar variants for other data structures - sorted-set is very useful, especially if diff'ing structures where dups don't matter

maleghast13:07:50

There are clearly far more varied and “interesting” options than I was aware of…

agile_geek13:07:02

You could use spec or schema to conform or coerce your data BTW. Or you could map a generic fn over each k-v pair which takes a mapping of key and conversion fn and applies the fn to the appropriate key's value.

maleghast13:07:18

Oooh, I like the idea of that… Although I always assume that map means I have to touch each value / kv pair… How does one tell the map which ones to act on and which ones to ignore?

agile_geek13:07:57

You can just write your fn to only convert the ones that you've told it too in your conversion mapping and pass the other k-v pairs thru untouched (basically an if based on whether the key exists in your conversion mapping)

maleghast13:07:00

Ah I see - yeah I can wrap my head around that…

agile_geek13:07:49

If you close over the conversion mapping (using a high order 'factory' fn that takes your conversion mapping and returns a function that does the conversion) you can easily test the concept and make it generic and reusable.

maleghast13:07:56

I’ve gone and looked at sorted-map / sorted-set… They are interesting, but they are “sorted” on the keys, where I need to coerce an arbitrary order, so unless I create my DB columns in alphabetic order they are not going to be what I am looking for…

maleghast13:07:14

@agile_geek - Thanks for that, I am not going to try it right now__ as I have stuff I need to get done, and the above situation while unwieldy does work, but I am going to put this, as an approach, into my refactor notes… 🙂

agile_geek13:07:23

Why do you need to align positionally on the database? Do you not have column names you can map to?

agile_geek13:07:33

I may have missed some context here BTW

agile_geek13:07:33

If you can build the relational query using explicit column names you could use the same kind of conversion to map datomic keywords to col names?

maleghast13:07:42

@agile_geek - I am using yesql, so my INSERT query is in the queries.sql file thus:

-- name: populate-gsod-measurements!
-- INSERTS GSOD measurement data into the database
INSERT into gsod_measurements
(stationid, wbanid, mdate, tempaverage, tempaveragenumments, dewpoint, dewpointnumments,
sealevelpressure, sealevelpressurenumments,stationpressure, stationpressurenumments,
visibility, visibilitynumments, windspeedaverage, windspeednumments, windspeedmax,
gustspeedmax, tempmax, tempmin, precipitation, snowdepth, fog, rain, snow, hail,
thunder, tornado)
VALUES
(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)

maleghast13:07:41

As such I need the values to be in the right order when I execute:

(sql/populate-gsod-measurements! {:? (ordered-values-vector-from-map (:gsod-canonical-db-columns ingest-cfg) (first (gsod-prepare-measurements-data row)))}))))

agile_geek13:07:11

So you could either specify the columns in alphabetical order in the insert and use sorted-map

agile_geek13:07:14

or you can reconstruct your map entry using reduce-kv using a conversion function that builds the entry in the correct order but the sorted-map option is a lot easier

maleghast13:07:22

So,

(gsod-prepare-measurements-data row ...)
creates a map of attribute names and values for datomic, and because it’s longer than 8-10 entries and built using (zipmap …) Clojure “chooses” hashmap and so the order is arbitrary

maleghast13:07:19

Yeah, I went with option 2:

(defn ordered-values-vector-from-map
  [ordered-headings unordered-map]
  (reduce
   (fn [output hdg]
     (conj output (hdg unordered-map)))
   []
   ordered-headings))

maleghast13:07:04

except that it would appear that I created my own reduce-kv…

agile_geek13:07:24

but you have control over the order that the col names appear in the insert? You can either dynamically construct the insert in the same order that hashmap uses or you can sort the map into alphabetical order and hard code the insert in that order?

maleghast13:07:40

@agile_geek - I could, but not and carry on using yesql, at least not “as intended” as the library is designed to completely separate SQL queries from Clojure, and the queries themselves can’t be dynamically created - at least not that I have ever seen…

agile_geek13:07:19

But you can write your Yesql insert with the column names in whatever order you want

agile_geek13:07:39

so if you write them in alphabetical order and use sorted-map?

maleghast13:07:49

It’s also worth bearing in mind that I am re-using a function that builds datomic inserts / transactions, so the column-names are not quite__ the same 😉

maleghast13:07:22

@agile_geek - Yes I could, but having them in table-order makes the DB and Queries a little more self-documenting 😉

maleghast13:07:57

I guess that I am happy with the solution I have, just wondering if there is a way to impose order on a map using a vector of keys that are in a specific order…

agile_geek13:07:55

Not sure why that matters. I don't normally care (or know) what order columns are physically stored in inside the RDBMS. Even if I specified them in an order most RDBMS won't guarantee they're physically stored that way.

glenjamin13:07:01

i thought yesql supported named params, or is that engine specific?

maleghast13:07:18

It matters to me / my brain @agile_geek

maleghast13:07:55

@glenjamin It does, but doing an INSERT on that basis is painful compared to using bound vars…

glenjamin13:07:55

i always liked that MySQL allows INSERT INTO table SET a=1, b=2 ..., and the nodejs client let you auto-expand a map into that format

maleghast13:07:11

Very useful for WHERE clauses

agile_geek13:07:21

If it really matters use reduce taking a conversion fn that has a mapping of column names - conversion fn's in the required order and reorders the key value pairs.

glenjamin13:07:29

@maleghast but surely that slight pain in writing the query is better than the pain of messing with sorted maps and having to repeat the ordered list in the code?

maleghast13:07:55

@agile_geek - That’s what I did (see code above)

glenjamin13:07:57

or maybe for simple inserts like that, you’d be better off generating the query instead of writing it

maleghast13:07:14

@glenjamin - I am not repeating the list, I re-use it out of config…

glenjamin13:07:34

but it’s written in the .sql file?

glenjamin13:07:56

to make that available as a vector to sort your map, it must be repeated or parsed out of SQL?

maleghast13:07:15

Don’t get me wrong, these are all helpful / valid points, but what I was asking is “is there a way to coerce an arbitrary order on a map using an ordered collection of keys?”

glenjamin13:07:36

yeah, sorted-map-by

glenjamin13:07:05

on something like #(- (nth %1) (nth %2))

maleghast13:07:36

@glenjamin - No, in this case I have a config file that has the Datomic attributes in a vector as the function was originally just building a map to make a Datomic transaction. The table was “built” with the same “column” order so I can reduce over the vector of attribute names to get an ordered collection of the values in the right order

glenjamin13:07:24

unless i’m missing something, that still requires the SQL file and the config file to contain the same list of columns in the same order?

glenjamin13:07:38

but i guess the point is that the config file has to exist anyway?

maleghast14:07:54

@glenjamin - you are missing something 🙂 Here is the query:

INSERT into gsod_measurements
(stationid, wbanid, mdate, tempaverage, tempaveragenumments, dewpoint, dewpointnumments,
sealevelpressure, sealevelpressurenumments,stationpressure, stationpressurenumments,
visibility, visibilitynumments, windspeedaverage, windspeednumments, windspeedmax,
gustspeedmax, tempmax, tempmin, precipitation, snowdepth, fog, rain, snow, hail,
thunder, tornado)
VALUES
(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
Here is the configured list (well vector) of Datomic attributes:
[:gsod-measurement/stationid
                             :gsod-measurement/wbanid
                             :gsod-measurement/date
                             :gsod-measurement/tempaverage
                             :gsod-measurement/tempaveragenumments
                             :gsod-measurement/dewpoint
                             :gsod-measurement/dewpointnumments
                             :gsod-measurement/sealevelpressure
                             :gsod-measurement/sealevelpressurenumments
                             :gsod-measurement/stationpressure
                             :gsod-measurement/stationpressurenumments
                             :gsod-measurement/visibility
                             :gsod-measurement/visibilitynumments
                             :gsod-measurement/windspeedaverage
                             :gsod-measurement/windspeednumments
                             :gsod-measurement/windspeedmax
                             :gsod-measurement/gustspeedmax
                             :gsod-measurement/tempmax
                             :gsod-measurement/tempmin
                             :gsod-measurement/precipitation
                             :gsod-measurement/snowdepth
                             :gsod-measurement/fog
                             :gsod-measurement/rain
                             :gsod-measurement/snow
                             :gsod-measurement/hail
                             :gsod-measurement/thunder
                             :gsod-measurement/tornado]

maleghast14:07:16

(I realise that the “column” list is the same apart from the repetition of “:gsod-measurement/” but that’s to help my brain, not out of any kind of necessity)

glenjamin14:07:23

so the list is currently repeated twice, if you used named params it’d be 3 times. If you generated the SQL from the datomic list you could get it down to only being in one place

maleghast14:07:19

However I don’t know of any way to use yesql and dynamically generate queries, so I would have two SQL implementations if I did the latter.

maleghast14:07:40

(I am not saying it’s not possible, just that I don’t know how to do it 🙂 )

glenjamin14:07:22

i suppose you could do it statically

glenjamin14:07:36

build the SQL files via a task

maleghast14:07:47

(and I want to have my other queries “managed” via yesql - I like the way it separates SQL from Clojure and yet at the same time allows me to “use” the queries in Clojure in a very Clojuric manner)

maleghast14:07:42

@glenjamin - Yeah that would work. I do think that it would be better to be able to impose an arbitrary order on a map, say using an ordered collection of keys, when it is useful to be able to do so.

glenjamin14:07:00

oh, a record

glenjamin14:07:07

records guarantee that iirc

maleghast14:07:25

Now there’s an interesting idea…

maleghast14:07:05

If I can use a record to transact data into Datomic AND a record does preserve order of properties then that would be a winner, in this case…

maleghast14:07:11

Instead of building a map of attribute names -> values I could define a record and push the values into it and then they would come out in the correct order for the SQL query as well.

agile_geek14:07:59

Record might work but array-map maintains insertion order of keys too iirc

maleghast14:07:20

@agile_geek - I think you are right, but I am using zipmap to create my map(s) and Clojure “decides” for you which implementation to use - if there are more than 8-10 kv pairs it “chooses” hash-map (according to clojuredocs, I’ve not done my own testing)

agile_geek14:07:35

Yeah you would have to convert. Record is probably the way to go or a function to reorder which is what you would have to do to use array-map anyway in your case.

agile_geek14:07:37

(def maps
  [{:a 1 :c 2 :d 3 :e 4 :f 5}
   {:a 11 :c 12 :d 13 :e 14 :f 15}
   {:a 21 :c 22 :d 23 :e 24 :f 25}])

(defrecord OrderedColumns
    [c d e a f])

(map map->OrderedColumns maps)

;; => (#try_clojure.core.OrderedColumns{:c 2, :d 3, :e 4, :a 1, :f 5} #try_clojure.core.OrderedColumns{:c 12, :d 13, :e 14, :a 11, :f 15} #try_clojure.core.OrderedColumns{:c 22, :d 23, :e 24, :a 21, :f 25})

maleghast14:07:21

@agile_geek - I LIKE that 🙂

agile_geek14:07:36

If you want to do conversion too you could create your own record constructor function that takes a map entry and creates a record but also converts values using a lookup conversion map as I mention ☝️

maleghast14:07:20

Right-oh 🙂

maleghast14:07:00

As previously stated, I will stick with the method I have right now__ as it works, but I will be coming back to this code in the coming weeks, specifically looking to improve it, and I am going to try this approach out as an alternative to my little reduce function, which is fine while I have an ordered list of keys available to me, but there may be times when I don’t…

maleghast14:07:15

(I have saved your example to my code notebook 😉 )

agile_geek14:07:34

@maleghast just for completeness...

(def maps
  [{:a "1" :c 2 :d 3 :e 4 :f 5}
   {:a "11" :c 12 :d 13 :e 14 :f 15}
   {:a "21" :c 22 :d 23 :e 24 :f 25}])

(defrecord OrderedColumns
    [c d e a f])

(defn create-ordered-columns
  [conversion-mapping]
  (fn [x]
    (let [convertor (fn [m k v]
                      (if-let [conversion-fn (k conversion-mapping)]
                        (assoc m k (conversion-fn v))
                        (assoc m k v)))]
      (map->OrderedColumns (reduce-kv convertor {} x)))))

(def conv-mapping
  {:a #(Double/parseDouble %)})

(map (create-ordered-columns conv-mapping) maps)

;; => (#try_clojure.core.OrderedColumns{:c 2, :d 3, :e 4, :a 1.0, :f 5} #try_clojure.core.OrderedColumns{:c 12, :d 13, :e 14, :a 11.0, :f 15} #try_clojure.core.OrderedColumns{:c 22, :d 23, :e 24, :a 21.0, :f 25})

maleghast14:07:28

Oh this is very cool 🙂 Thanks!

maleghast14:07:58

I need to re-visit the bits of the code that are turning all numbers into Doubles and switch out Long (datomic doesn’t have Int) for the non floating-number attributes

maleghast14:07:40

When I do that I will look at this ^^ as a refactor, so that I can put the conv-mapping into config (I’m using Aero for config, which is very covenient), and this will eliminate the need for the reduce I am doing to put the values in order for SQL usage as the map(s) will be in order when I call (vals ordered-map) 🙂

maleghast14:07:49

Nice one - really, thanks so much 🙂

agile_geek14:07:01

And here's the same thing using array-map instead of Record...

(defn convert-using-array-map
  [col-conversion-mapping]
  (fn [x]
    (apply array-map (mapcat (fn [[k f]] [k (f (k x))]) col-conversion-mapping))))

(def col-conv-mapping
  [[:c identity]
   [:d identity]
   [:e identity]
   [:a #(Double/parseDouble %)]
   [:f identity]])

(map (convert-using-array-map col-conv-mapping) maps)
;; => ({:c 2, :d 3, :e 4, :a 1.0, :f 5} {:c 12, :d 13, :e 14, :a 11.0, :f 15} {:c 22, :d 23, :e 24, :a 21.0, :f 25})

agile_geek15:07:42

Except can I replace > — even Java with > — even COBOL

yogidevbear15:07:24

Anyone know if there is a new version of Clojure Applied coming out? Seem to remember an ebook beta version being mentioned about a week ago

maleghast15:07:50

@agile_geek - That version using array-map is EVEN NICER!

agile_geek15:07:18

maleghast: You're welcome

agile_geek15:07:50

@yogidevbear I thought it had been out for ages...I read it in prerelease well over a year ago.

yogidevbear15:07:51

https://pragprog.com/book/vmclojeco/clojure-applied - Is that the same version you're referring to?

agile_geek15:07:59

In fact it must have been 2 years ago as I used some of the material around core.async pipelines in June 2015 for a bit of code for Mastodon C that @otfrom ended up rewriting!

yogidevbear15:07:05

Coolio 👍 It's next on my list to buy so didn't want to buy an out-of-date version if a newer one was in the pipelines

agile_geek15:07:33

I haven't heard about Alex and Ben writing a new version and Alex seems to have been busy with the latest version of https://pragprog.com/book/shcloj3/programming-clojure-third-edition

yogidevbear15:07:54

That's probably what I'm thinking of

yogidevbear15:07:05

Yeah, that's the one

agile_geek15:07:56

I think the major revisions to that book are around Spec (from the TOC)

agile_geek15:07:09

I must admit I am beginning to suspect @alexmiller is either an AI construct or he just never sleeps.

yogidevbear15:07:51

I think it comes down to having kids or going through a similar life shocking experience 😉

yogidevbear15:07:12

I don't sleep much anymore and tend to do a lot of work from morning until late at night

yogidevbear15:07:30

(Not that this is necessarily a good thing, mind you)

agile_geek15:07:41

I have kids and I used to sleep whenever I could...never had energy for side projects!

agile_geek15:07:19

Just saw this story...http://www.bbc.co.uk/news/uk-england-london-40552451 That's just half a mile or so from where I was working at style

maleghast15:07:24

I hear you @agile_geek - parenthood has led to a severe downturn in my overall productivity

maleghast15:07:39

Oh no… 😞

agile_geek15:07:14

Sad for the stall holders but I can't see any mention of casualties which is good

maleghast15:07:09

Yeah, that does sound good compared to what might have been…

maleghast15:07:24

…but that’s going to be the end of that market, I would expect.

agile_geek16:07:59

I hope not.... it was really cool

seancorfield16:07:13

According to that story it only affected one building and the market was open again after the fire.

yogidevbear16:07:57

@seancorfield is a good example of little sleep and lots of side projects. And he has 50 cat children 😉

seancorfield17:07:00

There are only five cats in my "cat-free" office! 😸