moin moin

morning

I'm heading down to London for the next 7 days

I’m never going to complain about Clojure or Java ever again. After working with Swift it’s ready to go out of the window, every release had breaking changes.

watched this video yesterday and loved it https://www.youtube.com/watch?v=epT1xgxSpFU&index=13&list=PLZdCLR02grLqCR8EKuil8NrffZuTl0kwL&t=107s

and now, I apparently understand monoids

allegedly

maybe

Added it to my list of video's to watch, thank you @otfrom

now you only need a vid on the category of endofunctors and you'll achieve monad nirvana @otfrom

@otfrom these videos are pretty good: https://purelyfunctional.tv/courses/category-theory/

Alreet?

Morning!

Morning All 🙂

Have any of you people had good results with OpenStreetMap and ClojureScript, and if so which libraries have you gone for..? I am using React through Reagent at the moment, so react-leaflet seems like a good idea, but I would be interested in alternatives if y’all have any recommendations..?

i've done lots with cljs and leaflet @maleghast - and leaflet works fine with OSM i believe

at the time i used om-old, and our own leaflet component... react-leaflet looks like a good idea

although it wasn't hard to use with plain om, so it shouldn't be hard with plain reagent

Splendid! Thanks @mccraigmccraig 🙂 I am a bit new to all this front end stuff - will Leaflet “play nicely” with GoogleMaps if we get a paid subscription “down the line”

?

i did an experiment with using google-maps tiles with leaflet - that worked fine - not sure about licensing though. we ended up using mapbox for our tiles - leaflet is their default UI and it was easily customisable for themes etc

we also did another experiment with ordnance survey for tiles - that was fine too

if you end up having to do lost of geospatial transformation @maleghast i recommend PostGIS, and if you end up with a lot of queries which need to do full-table scans every time the UI updates a control then elasticsearch is blindingly fast for such spatial queries

Oooh, this is all very good to know - thanks… I am putting together a PostGIS database at the moment, not that I have the slightest idea how to use it, but I have been asked to build one…

ES is a great technology for so many things - I had a sense that it was good at GeoSpatial, good to have that confirmed “from the sharp end” so to speak.

Were you putting GIS data into ES from PostGIS - using ES like a big Geospatial Cache - or did you just use ES on its own?

no, i did exactly that - do all the geospatial munging in PostGIS then upload to ES to serve

this snippet may be useful to you https://gist.github.com/mccraigmccraig/cdbd74e2a869ce5c46f36f5fa10a61dc

it imports ordnance survey shapefiles into PostGIS and does the co-ordinate system mapping from OS national grid (EPSG:27700) to WGS.84 (EPSG:4326) which is the system all the web maps use

and make sure you are using a recent postgresql @maleghast

Thanks very much 🙂 We’re running on 9.6.x I think…

(AWS RDS current PostgreSQL version)

that will be fine... there was an earlier version 9.0 or 9.1 or something which had some errors in the parameters in the PostGIS coordinate-system database, which messed your life up good

Ah ok then - dodged that bullet phew

I have a creeping sense that I am going to have to learn a LOT MORE about geospatial stuff before I am done…

All feels very new and intimidating at the moment.

you will have to learn a bunch of new terminology... but most of the painful stuff is nicely packaged by PostGIS and ES

That’s good to know 🙂

Morning.

I need to import the GADM data into PostGIS in the next couple of days - I am still not clear on how to use__ it once it’s in there however… I mean once it’s imported how does one write a query to get the boundary for say Lancashire? More to the point how does that data come out, i.e. what format is it in etc.? I am asking rhetorically - I am not looking for you to answer the questions at all, I promise. I am just trying to illustrate my level of ignorance 🙂 If you have any good recommendations for learning resources, however, that would be greatly appreciated.

import the data first - find out the EPSG # for it's coordinate system and plug it in to a script like the one i gave you above. each shapefile will import to a single postgresql table

once you have the data in postgis then you can just select from tables... the geospatial data is in custom formats, but there are sql fns to work with them and indexes which support them, so it's sql all the way down

generally there will be a bunch of metadata columns and a geospatial column in a geospatial table... it's been an age though and i don't even have postgresql installed on my current machine, so my knowledge is kinda hazy

@maleghast this is a project that uses openlayers, haven't tried leaflet .

https://github.com/cdpjenkins/iss-weather-apis

@mccraigmccraig - Thanks for the overview; it’s encouraging me to hold the view that it’s not so scary as I’ve persuaded myself that it might be.

@thomas - Thanks I will take a look at that 🙂

it's actually great fun @maleghast

😄

@mccraigmccraig pushed at your or thought of by you as reasonable things?

@otfrom Does your wording imply your thoughts on the matter? 😉

@carr0t I like streams of events. I'm a fan

Heh, fair enough. I thought from your phrasing you were suggesting otherwise

We use Kafka for our event streams

Connecting many Clojure microservices

An interesting thread by Jay Krepps on Twitter last night on why Kafka was on 0.x for so long.

Heh. The excuse I was given was "We hadn't done anything important enough to go to 1.x". Now they've done exactly-once delivery (apparently)

What was the thread consensus?

And the tooling on streams, KSQL that kinda thing. Made sense.

The 0.x was immaterial to me, it was solid enough to use.

Likewise

We're still on 0.10.2 at the moment. Do need to go to 1.0 at some point but no rush

@otfrom thought of by me as reasonable things... not kafka/onyx style streams though - that would be all heavyweight - manifold style streams

turns out that with a few stream primitives, sort-merge, outer-join, inner-join, cross-streams, you can do all sorts of fun stuff very fast with sorted streams of records from cassandra

Ooh. That sounds awesome

it's great when you need to do small-to-moderate-sized joins in cassandra - think 10ks of records, maybe 100ks - for large joins use spark already - but a lot of my current problems seem to fit into that category, e.g. expand a list of all the subscribers to a timeline or match import records against existing db records

The stuff you work on sounds like a lot of fun @mccraigmccraig

I might just be weird though

Not that those two are necessarily mutually exclusive 🙃

ha, well i mostly find it fun too, although the last release was a bit brutal

Are you able to join streams without using c* for the lookups?

@otfrom if you get a stream of records from c* from a partition sorted by the join-key then you can efficiently get a lot of records with very few queries - joining is then just doing the merge part of a sort-merge join

Sounds like I have some reading to do

😀

example - i get 3 streams from cassandra, all sorted by participant-id, conversation-participants, conversation-participant-reaction-counts and conversation-participant-notification-counts ... joining these 3 streams is just a matter of repeatedly considering the records at the head of the 3 streams, and selecting those (1 or more) with the least participant-id to merge and put on to the output stream

as long as your c* client supports streams (alia does) then there are very few queries (you can adjust it, but iirc the default page-size is 5000 in alia), the joins happen purely in memory and it's all non-blocking

you can probably do something very similar with the core.async interface in alia too