morning

morning!

moin moin

Yo

and emerging from the void, @paulspencerwilliams asserts "I exist"

that's what Yo means, right?

any love for my code review request? https://clojurians.slack.com/archives/C064BA6G2/p1508792205000218

Absolutely. 👍

@otfrom it sounds a bit like a group-by

morning

losing the '9' from the data seq is making me uncomfortable 🙂

@otfrom does this mandate the values are sorted (if I'm reading correctly?)

@peterwestmacott it is a bit, but I want it to work with larger sequences. The stream should already be sorted.

@dominicm yes 😉

are the time periods regular?

@tcoupland losing the 9 from the data is fine as we just want a subset of the data. We could give in a lazy seq for the template (which is probably how it will work) and then do a take

I wonder if a specific solution would be better than a general parameterised one

http://clojuredocs.org/clojure.core/split-with can replace separate uses of take/drop while perhaps.

@peterwestmacott the periods aren't always regular (tho usually will be)

i think you want a stateful transducer

@tcoupland I think that's where it will go in the transducer version. I've not puzzled my way through that yet

@dominicm I'll think about how to use split-with.

(drop 1 x) -> (rest x) 🙂

@tcoupland fair point

I had a look through the cgrand lib, and if the time periods were regular you could use, https://github.com/cgrand/xforms/blob/master/src/net/cgrand/xforms.cljc#L502-L562 but I can't see anything for your use case exactly

also that's a bit more involved ^. Definitely not concise!

@otfrom I mention it only because I've not read the source of split-with, if it is naive my point is moot, if it's smart about not wasting effort then it might be valuable. It's a little more concise I suppose.

@dominicm concise is good

@danieleneal I'd looked at that and got a bit confused by the doc (esp the steps), so had a go myself. I need to make sure it deals with the empty periods properly.

I think the steps you'd pass just 1 for n usually

but looks like it wouldn't fit your needs anyway

empty periods at the start/middle/end were the bit that drove me to write code as I hadn't found something that had done that

ah I see

I mostly need it to line up time series from different sources

I think you could do this with plain partition-by if you assume the data is sorted. Your partitioning function would be something like #(first (filter (fn [template] (<= template %)) [1 2 3]))

The lack of find-first in core always trips me up.

You won't get empty groups though, if you want that.

yeah, the empty groups were the bit I needed that was different

partition-by is where I went first

Morning

Lo

Lo

@tcoupland you hinted at a stateful transducer?

Morning.

i havn't got anything that does specifically what your looking for, got some fairly light ones that cld provide a base to work from perhaps

@otfrom had some trouble reconstructing the thread, are you asking about partitioning of maps with timestamps by time periods (e.g. every minute)?

@reborg yes, but with some time periods having no data in them and some possibly having multiples

there is a code example further up the history I can link to

I need it to line up multiple time series so I can do some processing on them

Couldn't you align to the nearest minute?

(import '[java.util Date])
(def times [{:a "a" :t (Date.)} {:a "b" :t (Date.)} {:a "c" :t (Date.)}])
(partition-by #(quot (.getTime (:t %)) 60000) times)

@reborg I think the problem is that I don't get the empty time periods with partition-by (tho I might have to re-check to be sure)

I see, no that's true you don't get them

yeah, it is all about the padding and filling in

a rethink could be, to not having the empties at this phase, create these views on all streams, and create the empties when you combine them. If that makes sense

filling the gaps approach

(def times [20 32 49 50 99])
(take 10 
  (let [buckets (group-by #(quot % 12) times)] 
    (map #(get buckets % []) (iterate inc 0))))
;; ([] [20] [32] [] [49 50] [] [] [] [99] [])

@otfrom sorry for coming in late to this, but the guys at imin have this sort of thing I think

@conan imin?

Sorry these guys https://www.imin.co/

They have an open protocol for sending event data, which is close to my understanding of your use case

@reborg that is an interesting solution and would work with things out of order as long as they fit into memory

Hopefully your events could just be a timestamp and an id during buckets construction, you can then fetch the rest on demand. But yeah, the map (as is) should fit in memory

I’ve had a look around and can’t find a Clojure one, so… Is there a Java library out there that anyone might know of and / or recommend for administering PostgreSQL? I need to provide functionality to copy tables from one DB to another and would like to do it without weird edge-case SQL…

…if possible without JDBC at all in fact, just a native connection to PostgreSQL for administrative commands only.

@maleghast Without completely understanding the problem is not using Postgres create table from SQL do the trick?

create table my_new_table as select * from my_existing_table

I’m assuming at this point you could do it to another db.

olddb.my_existing_table to newdb.my_new_table etc

@jasonbell - Yeah that kind of thing, but I want to take tables from several DBs and put them into the same DB, so I may__ need to prepend table names with something

@jasonbell - So I am hoping to figure out a way to talk natively to PostgreSQL instead of having to use SQL

I may be on a hiding to nothing…