Fork me on GitHub
#clojure-uk
<
2018-10-24
>
maleghast09:10:24

Morning All 🙂

maleghast09:10:06

Quick question... Does anyone here have any experience with using Transit and Postgres's JSON stuff...? As in does anyone know how compatible they might be..? Google does not seem to be able to find me anything about Postgres, JSON AND Transit, just Postgres and JSON. This may mean no one's done anything with these three things together, ever, but that seems implausible...

practicalli-john09:10:27

@maleghast are you intending to use Transit to write to Postgres, or are you doing some data processing in Clojure first? Postgres+json seems pretty straight forward http://www.postgresqltutorial.com/postgresql-json/ although you may want to search for issues specifically for postgres and jason. Transit does a good job of keeping the JSON structure so assume there should be no issues (good tests may show otherwise though)

alexlynham10:10:12

Be careful with pg and json - the perf profile of using it for operational stuff has become a bottleneck on one of the teams I'm on. Granted I'm no expert so maybe there's a way of fixing that with up front design... which was not done by the original implementers in this case

alexlynham10:10:34

Also, morning from France, heh

danm10:10:03

Remote working from the continent?

3Jane10:10:22

> Be careful with pg and json yeah I would be worried about that (although the data guy who advocated it was trying to convince me hard that the newer versions handle it well, so… maybe version dependent?)

alexlynham10:10:28

It's very easy to trigger full scans if your data is badly modelled was what we found so it really depends on shape of data and access pattern I think?

3Jane10:10:39

I want to hear a lot more about what triggered the full scans actually 😄

3Jane10:10:03

were people attempting to filter or order based on a sub-field of the json object?

mccraigmccraig11:10:52

you can index jsonb fields to avoid that can't you ?

alexlynham11:10:36

> were people attempting to filter or order based on a sub-field of the json object? This combined with a polymorphic data shape iirc

3Jane11:10:57

yeah, that’d do it, because you can’t build a good index on it

peterwestmacott11:10:53

@mccraigmccraig yes, you can now create indices on fields within Json in postgres

maleghast12:10:21

Ah, I am using it as a document store that I can__ query deep into the structure If I have too, but not for operational purposes. I use other fields in the table to store ids, and other index-able stuff for selection queries, a kind of hybrid approach, so that I don’t have to be frightened of the downsides of a pure document database

maleghast12:10:50

But all of this is genuinely interesting and worthwhile “watch out for” stuff too, make no mistake ;-)

alexlynham10:10:35

What was your use case?

alexlynham10:10:09

(I'm with family, week away, no laptop (eek!) and sporadic internet)

🌴 8
mccraigmccraig11:10:49

no laptop -nice! i currently have to reject holiday locations which don't have decent wifi or 4g 😬

guy12:10:35

Is a holiday still a holiday if ur taking ur laptop/whatever ur connecting to the internet with :thinking_face:

maleghast12:10:21

@jr0cket - Thanks! Turns out that we can use Transit for our use-case but thanks for the answer :-)

maleghast12:10:51

(Basically we can ask Transit to make the JSON verbose before we store it)

practicalli-john12:10:56

Transit is pretty cool. I do wonder how easy it would be to store EDN in Postres natively. I assume this would need some postgres experience

maleghast12:10:03

@jr0cket - Well... You could store EDN in a TEXT field and take my approach about using other columns for values that you might want to query on.

maleghast12:10:05

But the PostgreSQL implementation of JSONB is not EDN compatible... What would be GREAT is "someone" creating a PostgreSQL extension to treat EDN as the same kind of resource as JSON when you use the JSONB field type.

maleghast12:10:28

And it's all open source, right? So we just need to find someone smart enough and then - gravy!

maleghast12:10:42

(I am not smart enough, yet...)

mccraigmccraig13:10:12

with "the PostgreSQL implementation of JSONB is not EDN compatible" dyu mean that EDN's encoding in JSON makes JSONB indexes not very useful, or that you just can't store EDN->JSON in JSONB columns for some reason @maleghast?

maleghast13:10:10

The former, as far as I can tell EDN's encoding means that the advantages of JSONB columns, like the indexing etc. will not work.

maleghast13:10:32

I think that JSONB columns would actually reject EDN that is not well-formed JSON, however

mccraigmccraig13:10:26

i would expect that - was just wondering if there were limits imposed on the JSON stored by JSONB, but you didn't mean that

maleghast13:10:55

Of course I've not tested that, but I think that the column-type validates the inbound JSON, in order to allow the deep-querying and indexing, and therefore most EDN would not pass the validation.

maleghast13:10:03

I should__ test that however...

mccraigmccraig13:10:50

oh, hold on, i'm tired and confused... i'm confusing transit (which has a json encoding) with edn (which is its own encoding) ... ignore everything i have said for the last 10 mins picard-facepalmpicard-facepalmpicard-facepalm

picard-facepalm 8
yogidevbear13:10:27

Sideline code question... How is there no hammock emoji in this slack group?

😄 8
practicalli-john13:10:03

Clojure workshops update: we are planning to run the following workshops: "Building Clojure apps with Kafka" "clojure.spec from the ground up" Thoughts welcome. https://www.meetup.com/London-Clojurians/events/254693569/

👍 4
maleghast13:10:18

Curses! Wish I could make the Kafka one!

danm13:10:33

Me too, interested about what other folk are doing with it

practicalli-john13:10:19

The workshop presenter is using Kafka for financial services. Funding Circle sound like they are doing some interesting things with Kafka too, using multiple topics as part of a data transformation pipeline.

mccraigmccraig13:10:43

i'd would have like to go, but weekends are sacrosanct family time for me atm

👍 4
danm14:10:57

Oh it's a weekend?

danm14:10:04

Yeah, I need that time to decompress from work 😉

jasonbell14:10:45

I'll be having a mini breakdown on the Sunday that my slides aren't good enough, I'm not funny enough and my ego is too big for the room and all that stuff. Just a normal Sunday. 🙂

😂 8
practicalli-john14:10:58

take friday off work... its the only option 😁

jasonbell14:10:02

And I should really wear something different this year.

practicalli-john14:10:37

you look smarter than everyone else, so no one will notice you only have the one suit