Fork me on GitHub
#off-topic
<
2023-07-14
>
octahedrion08:07:47

https://github.com/cognitect-labs/transducers-python has been archived and hasn't been touched in 9 years, eduction wasn't implemented and there are no active forks. I'm guessing nobody had time to work on it, but wasn't there any interest from the Python community ?

Robert Todea09:07:59

@U0CKDHF4L I also had a look into this project, but I don't think here is the right place to ask this question 😅

octahedrion09:07:28

where is then ?

phill09:07:03

cognitect-labs did not have outside contributors, so I suppose an interested party would fork it and take up the maintenance.

octahedrion09:07:21

yet nobody has

phill09:07:33

Your turn?!

octahedrion09:07:51

like everyone else I don't have the time

octahedrion09:07:35

(or the expertise as it's been ages since I wrote in Python)

phill10:07:49

I suppose cognitect worked it out to present transducers as not a Clojure thing but a Computer Science innovation. Especially since in the 21st century all Computer Science is Python.

octahedrion10:07:36

it's a shame it hasn't received any attention

octahedrion10:07:28

out of interest does anyone know how to implement eduction in transducers-python ?

Noah Bogart11:07:55

I also think they’re just not as necessary or helpful in python. Python has traditional for loops which are more powerful than transducers

octahedrion11:07:39

is that sarcasm ?

Iulian Bojinca12:07:50

Ruby Lazy, Java Streams, Clojure Transducers (there might be other examples too) appeared kinda at the same time. Nevertheless, the idea behind them is the same. People started doing this functional tacit style of transformations and filtering that were very costly in many cases where you had to work on a big set of data. It's a shame python hasn't done anything about this internally and you have to create your own generators to handle it, or use additional packages that probably will take a hit on many code reviews and force the programmer to take the imperative approach.

Ludger Solbach17:07:02

Java Streams and Transducers are not the same beast. Transducers are about the separation of the "filters from the pipes" and letting you specify the type of pipe afterwards. With Java Streams, your filters are coupled to your pipe (stream).

isak20:07:53

For serializing Clojure values for redis, anyone use an approach where redis is able to understand the values (like maps)? (Instead of just a clojure-only binary format). For example, I see redis supports messagepack. Maybe transit in msgpack mode would be a good approach? Anyone try this?

👀 2
isak22:07:18

It does work. I serialized this clojure map to key 'k': {"hello" "there"} Then it can be accessed in redis via LUA like this:

local value = redis.call('GET', 'k');
local jvalue = cmsgpack.unpack(value);
return jvalue['hello']
=> "there"

p-himik22:07:54

Cool! I don't know how msgpack encoding works but if there's caching just as there's one for JSON, I'd try nested maps with the same key: {:some-long-key {:some-long-key "value"}}. I think caching is also active for string keys.

isak22:07:02

For keyword keys, I think there would be a problem, because I don't know how they are serialized. Nesting with string keys works: I set this value:

{"hello" "there"
     "nested" {"deeply" true}}
Then can access it like this:
local value = redis.call('GET', 'k');
local jvalue = cmsgpack.unpack(value);
local nested = jvalue['nested'];
local deeply = nested['deeply'];

return deeply
Though a strange thing is it returned the number 1, not true

isak22:07:40

That map does roundtrip correctly with just write/read in cognitect transit msgpack, but I guess not with redis' msgpack.

p-himik22:07:39

I meant nesting with the same key - that's when caching would be used, if there's one.

p-himik22:07:03

And keywords are probably serialized as strings where the first two characters are ~:, i.e. :x would become "~:x".

p-himik22:07:42

If that's the case, regular strings that start with ~ will have another ~ at the start.

isak22:07:45

Ah right. Just tested the nesting with same key, and it does roundtrip correctly.

isak22:07:05

Interesting, I will try that now

isak22:07:23

Hah you were right, neat

p-himik22:07:28

Round-tripping is one thing, but extracting the cached data from Lua is another.

p-himik22:07:05

In any case, the format is pretty well documented, including the caching part: https://github.com/cognitect/transit-format#caching

👍 2
isak22:07:26

Ok you are right again, that breaks it

isak22:07:52

So I guess you would need a transit-lua, or redis plugin for this to work 100%

isak23:07:30

More practical may be to skip transit and use messagepack directly, and restrict usage to very simple values.