Fork me on GitHub
#clojure-uk
<
2017-10-24
>
guy07:10:52

morning!

thomas07:10:17

moin moin morning

otfrom07:10:27

and emerging from the void, @paulspencerwilliams asserts "I exist"

otfrom07:10:34

that's what Yo means, right?

Rachel Westmacott08:10:44

@otfrom it sounds a bit like a group-by

tcoupland08:10:52

losing the '9' from the data seq is making me uncomfortable 🙂

dominicm08:10:07

@otfrom does this mandate the values are sorted (if I'm reading correctly?)

otfrom08:10:49

@peterwestmacott it is a bit, but I want it to work with larger sequences. The stream should already be sorted.

Rachel Westmacott08:10:09

are the time periods regular?

otfrom08:10:41

@tcoupland losing the 9 from the data is fine as we just want a subset of the data. We could give in a lazy seq for the template (which is probably how it will work) and then do a take

Rachel Westmacott08:10:56

I wonder if a specific solution would be better than a general parameterised one

dominicm08:10:12

http://clojuredocs.org/clojure.core/split-with can replace separate uses of take/drop while perhaps.

otfrom08:10:17

@peterwestmacott the periods aren't always regular (tho usually will be)

tcoupland08:10:28

i think you want a stateful transducer

otfrom08:10:51

@tcoupland I think that's where it will go in the transducer version. I've not puzzled my way through that yet

otfrom08:10:19

@dominicm I'll think about how to use split-with.

tcoupland08:10:53

(drop 1 x) -> (rest x) 🙂

danielneal08:10:08

I had a look through the cgrand lib, and if the time periods were regular you could use, https://github.com/cgrand/xforms/blob/master/src/net/cgrand/xforms.cljc#L502-L562 but I can't see anything for your use case exactly

danielneal08:10:55

also that's a bit more involved ^. Definitely not concise!

dominicm08:10:56

@otfrom I mention it only because I've not read the source of split-with, if it is naive my point is moot, if it's smart about not wasting effort then it might be valuable. It's a little more concise I suppose.

otfrom08:10:16

@dominicm concise is good

otfrom08:10:20

@danieleneal I'd looked at that and got a bit confused by the doc (esp the steps), so had a go myself. I need to make sure it deals with the empty periods properly.

danielneal08:10:02

I think the steps you'd pass just 1 for n usually

danielneal08:10:14

but looks like it wouldn't fit your needs anyway

otfrom08:10:14

empty periods at the start/middle/end were the bit that drove me to write code as I hadn't found something that had done that

otfrom08:10:51

I mostly need it to line up time series from different sources

dominicm08:10:35

I think you could do this with plain partition-by if you assume the data is sorted. Your partitioning function would be something like #(first (filter (fn [template] (<= template %)) [1 2 3]))

dominicm08:10:08

The lack of find-first in core always trips me up.

dominicm08:10:18

You won't get empty groups though, if you want that.

otfrom08:10:22

yeah, the empty groups were the bit I needed that was different

otfrom08:10:43

partition-by is where I went first

otfrom09:10:53

@tcoupland you hinted at a stateful transducer?

tcoupland10:10:40

i havn't got anything that does specifically what your looking for, got some fairly light ones that cld provide a base to work from perhaps

reborg10:10:47

@otfrom had some trouble reconstructing the thread, are you asking about partitioning of maps with timestamps by time periods (e.g. every minute)?

otfrom10:10:44

@reborg yes, but with some time periods having no data in them and some possibly having multiples

otfrom10:10:57

there is a code example further up the history I can link to

otfrom10:10:49

I need it to line up multiple time series so I can do some processing on them

reborg10:10:06

Couldn't you align to the nearest minute?

(import '[java.util Date])
(def times [{:a "a" :t (Date.)} {:a "b" :t (Date.)} {:a "c" :t (Date.)}])
(partition-by #(quot (.getTime (:t %)) 60000) times)

otfrom10:10:00

@reborg I think the problem is that I don't get the empty time periods with partition-by (tho I might have to re-check to be sure)

reborg10:10:05

I see, no that's true you don't get them

otfrom10:10:52

yeah, it is all about the padding and filling in

tcoupland11:10:34

a rethink could be, to not having the empties at this phase, create these views on all streams, and create the empties when you combine them. If that makes sense

reborg11:10:01

filling the gaps approach

(def times [20 32 49 50 99])
(take 10 
  (let [buckets (group-by #(quot % 12) times)] 
    (map #(get buckets % []) (iterate inc 0))))
;; ([] [20] [32] [] [49 50] [] [] [] [99] [])

conan12:10:18

@otfrom sorry for coming in late to this, but the guys at imin have this sort of thing I think

conan12:10:16

They have an open protocol for sending event data, which is close to my understanding of your use case

otfrom12:10:26

@reborg that is an interesting solution and would work with things out of order as long as they fit into memory

reborg12:10:39

Hopefully your events could just be a timestamp and an id during buckets construction, you can then fetch the rest on demand. But yeah, the map (as is) should fit in memory

maleghast17:10:41

I’ve had a look around and can’t find a Clojure one, so… Is there a Java library out there that anyone might know of and / or recommend for administering PostgreSQL? I need to provide functionality to copy tables from one DB to another and would like to do it without weird edge-case SQL…

maleghast17:10:45

…if possible without JDBC at all in fact, just a native connection to PostgreSQL for administrative commands only.

jasonbell19:10:15

@maleghast Without completely understanding the problem is not using Postgres create table from SQL do the trick?

create table my_new_table as select * from my_existing_table
` for example.

jasonbell19:10:43

I’m assuming at this point you could do it to another db.

jasonbell19:10:16

olddb.my_existing_table to newdb.my_new_table etc

maleghast21:10:22

@jasonbell - Yeah that kind of thing, but I want to take tables from several DBs and put them into the same DB, so I may__ need to prepend table names with something

maleghast21:10:12

@jasonbell - So I am hoping to figure out a way to talk natively to PostgreSQL instead of having to use SQL

maleghast21:10:21

I may be on a hiding to nothing…