data-science 2022-08-19 | Slack Archive

Cameron Kingsbury16:08:50

@chris441 is there a nicer way to quickly go from a http://tech.ml dataset to a pandas dataframe than to pipe through mapseq-reader and pd/DataFrame? trying to improve my experience of composing python libraries and scicloj libraries together

Cameron Kingsbury16:08:49

(as well as the reverse)

chrisn12:08:21

Arrow or parquet files would be a good choice.

chrisn12:08:35

Or for the particularly masochistic types you could go to a dict of numpy arrays.

chrisn12:08:36

I recommend saving to arrow - https://techascent.github.io/tech.ml.dataset/tech.v3.libs.arrow.html

Cameron Kingsbury13:08:20

Okay cool thank you!

2022-08-19

Channels