Fork me on GitHub
#transit
<
2022-02-06
>
Alex Miller (Clojure team)00:02:16

What is serialization without i/o?

Alex Miller (Clojure team)00:02:01

The io parts are interfaces with many implementations so what are you planning to do?

dcj00:02:28

Kafka will do the I/O. Kafka gives you hooks for how you want to transform (serialize/deserialize) before/after it does I/O. So, here is example of how I create a CBOR serdes for Kafka, note that I use encode and decode functions, that is what I am thinking I want/need from Transit...

(ns cktc.cbor
  (:require
   [clj-cbor.core :as cbor]
   [jackdaw.serdes.fn :as jsfn]
   )
  (:import org.apache.kafka.common.serialization.Serde
           org.apache.kafka.common.serialization.Serdes)
  (:gen-class
   :implements [org.apache.kafka.common.serialization.Serde]
   :prefix "EdnCborSerde-"
   :name cktc.cbor.EdnCborSerde))

(set! *warn-on-reflection* true)

(defn cbor-serializer []
  (jsfn/new-serializer
   {:serialize (fn [_ _ data]
                 (cbor/encode data))}))

(defn cbor-deserializer
  []
  (jsfn/new-deserializer
   {:deserialize
    (fn [_ _ data]
      (clojure.walk/keywordize-keys (cbor/decode data)))}))

(defn cbor-serde
  "Implements an EDN SerDes (Serializer/Deserializer)
  using cbor.
  (RFC 8949 Concise Binary Object Representation)
  
  "
  []
  (Serdes/serdeFrom (cbor-serializer) (cbor-deserializer)))

(def EdnCborSerde-configure
  (constantly nil))

(defn EdnCborSerde-serializer
  [& _]
  (cbor-serializer))

(defn EdnCborSerde-deserializer
  [& _ ]
  (cbor-deserializer))


;; gen-class with AOT for the cbor_serdes.clj:

;; clj -e "(compile 'slc.cbor-serdes)"

;; make sure classes (default *compile-path*) is mkdir-ed and on classpath (e.g. deps.edn :paths ["src" "resources" "classes"]

;; restart repl with (:require ... [slc.cbor-serdes]) and (:import (slc.cbor_serdes EdnCborSerde)) (if compilation went well the import should work)

Ben Sless04:02:50

Done they just have to return byte arrays? How complicated is it to serialize to byte array or read from a byte array in transit?

dcj01:02:09

The arities of the clj-cbor encode and decode fns are interesting/instructive.... https://github.com/greglook/clj-cbor#clj-cbor

favila01:02:44

I suspect there is a way to do it because the two are separated conceptually in transits very design but you will probably need to rely on private interfaces

favila01:02:17

At least in transit-js I know it’s possible because I did this many years ago to send/receive transit to/from web workers without serializing to string or reading from string (ie using structured cloning)

favila01:02:42

I don’t remember the details but it definitely wasn’t via public interfaces