Fork me on GitHub
#clojure-uk
<
2017-07-11
>
mccraigmccraig06:07:35

how many šŸ˜ŗ in your non-šŸ˜ŗ-free office tho @seancorfield ?

yogidevbear08:07:06

morning indeed, Mr. van der Veen

yogidevbear08:07:39

(= "must" ā˜•)

conan10:07:35

@agile_geek wow that's scary stuff

otfrom11:07:36

anyone here travel to London regularly? We've got some people who will need to do that so just wondering what good options are compared to hotels (for something like 2-3 nights per week 1-4 weeks/mo)

maleghast12:07:36

@otfrom - My experience is that the way to do this as cost-effectively as possible is to do ā€œPrivate Roomā€ on AirBnB Failing that, booking far in advance with Ibis or Travellodge can work out very reasonableā€¦

maleghast12:07:13

I am actively considering purchasing a modest studio / one bed and AirBnB-ing it when I am not in it, but we will have to see how things goā€¦

otfrom12:07:46

@maleghast that last bit doesn't sound unreasonable. Side bets for startups?

maleghast12:07:23

Indeedā€¦

maleghast12:07:47

PoO - A while back someone on here (I think that it was @mccraigmccraig or @seancorfield but I am not 100% certain that it was either of them) recommended a Clojure library to me for the purpose of talking to ElasticSearchā€¦ Is that person here? Was it Spandex? The examples only seem to be for making searches - does anyone know if itā€™s feature complete / capable of shoving data into ElasticSearch, or if I need to do that should I go for Elastich instead?

mccraigmccraig12:07:58

@maleghast i haven't used spandex personally ( my project is using a hacked version of older ES client, also by @mpenet ), but my recollection is that it doesn't care what the queries you send are - it just deals with the transport and generic response handling

maleghast12:07:31

Okā€¦ So I can POST and it will add documents to the Index in question..?

mccraigmccraig12:07:36

there is an example of a delete request on the README tho - https://github.com/mpenet/spandex#bulk-requests-scheduling which would support my supposition

maleghast12:07:41

(or is it PUT?)

maleghast12:07:13

Yeah, I saw thatā€¦ OK, thatā€™s pretty much decided me then

mccraigmccraig12:07:24

haha, it's been too long since i looked at ES, i couldn't tell you any api details anymore šŸ™‚

mccraigmccraig12:07:57

you should be able to play with it in a repl and confirm that it will do the requests you are wanting to do

mccraigmccraig12:07:55

ES requests are all just plain json data structures though, so i don't think there is any fundamental reason why it shouldn't be able to

maleghast12:07:48

(I have only had very limited access to / exposure to ES in the past, but I need to store reasonably large amounts of document based data and be able to perform fast searches on the stored data, so it appears to be a good fit for my use-case šŸ˜‰ )

mccraigmccraig12:07:15

@maleghast ES is brilliant for surprisingly fast queries over medium to large data... a useful tip is to remember to set mappings and to use doc_values wherever you can ( https://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.html ) unless you are fully intending all of your dataset to fit into cluster memory

mccraigmccraig12:07:24

with doc_values ES will use memory-mapped off-heap storage which can lead to very modest RAM usage

maleghast12:07:51

This is particularly interesting information - thanks šŸ™‚

mccraigmccraig12:07:06

oh, huh - looks like they made doc_values the default now šŸ™‚

mccraigmccraig12:07:44

so you don't need to remember!

maleghast12:07:54

Yeah, you only need to remember to turn them off, if you are sure you donā€™t need them, which is really rather excellent.

mccraigmccraig12:07:15

are you storing time-series data @maleghast ?

maleghast12:07:51

Nope, well not in those termsā€¦ I have access to the data for a sliding window and I want to have access to it after the sliding window, so I am going to store it after accessing it (the proprietor of the data in question is aware and has given permission), so time is a factor, but itā€™s not a time series in truth.

mccraigmccraig12:07:53

ok, doesn't matter then - but if you were it makes sense to split your data across different indexes, bucketed by time period - because ES query allows you to query multiple indexes and combine results, and it's much easier to remove an index than it is to remove data from an index

maleghast12:07:43

I had some inkling of this - I have a feeling itā€™s a recommended approach on ELK stack stuff that I did a couple of years ago - makes queries more efficient too IIRC?

mccraigmccraig12:07:44

yes - kibana has index-pattern stuff built in iirc so you can define index patterns like events-{YYMMDD} or something

maleghast12:07:05

*nods* that is definitely ringing bells in my head

mccraigmccraig13:07:14

maybe not helping @maleghast šŸ˜¬, but i've got lots of async code composed with a promise-monad, and i wanted to have the chains of async operations carry around some side-channel log-data to aid traceability - a writer monad would be ideal for this, so a promise-writer-monad-transformer would be great, but refactoring without the help of static-checks doesn't bear thinking about

maleghast13:07:06

I see, in that at a high-level I mostly grok what you are saying. šŸ˜‰

agile_geek20:07:43

@mccraigmccraig I think all major refactoring is more difficult/dangerous without static checks even with good test coverage. If I follow u, from what I know about monads, I agree.