Fork me on GitHub
#clojure
<
2022-03-18
>
gavin10:03:56

Hello Friends. I want to be able to work with protobuf encoded data. Any recommendations? I have proto files, and I have a source of encoded protobufs. The mission… to read (decode) the bytes and then do something - like spit out JSON of the structure. Any help / pointers appreciated.

Conor10:03:38

It is pretty easy to reconstruct a proto-generated Java class provided you have the bytes. In Kotlin it would be val reconstituted = MyProtoType.parseFrom(theByteArrayIWantToParse), and I imagine similar in Clojure. Then you can render that as whatever type of string you like.

gavin11:03:44

Hmmm… I am not sure we are on the same page. I have, say, foo.proto file(s) for things. I have a bag of bytes provided to me as a b64 encoded string. I want to take the bag of bytes and turn them into a map with all the data therein so that I can render it as say, JSON. It is the reading of the protofiles and making them usable in Clojure that I am having the issue with.

Conor11:03:54

You will need to use protoc to generate a Java class from the proto files, then instantiate one from the bytes as above

Conor11:03:12

Or at least, that is the easiest way I know to do that

gavin13:03:07

Do you know of a library (clojure) that you have successfully used ? I am currently kicking the tires on https://medium.com/appsflyer/appsflyers-open-source-libraries-for-clojure-and-protocol-buffers-da9a87583c41

gavin13:03:01

It seemed like a really nice tool… nice write up. And I like the lein pluggin to pull schemas from another git repo or local. but I can’t get it to work.

gavin13:03:19

I think there is something wrong with the proto files I am using but now I am trying to tease that apart.

Conor13:03:30

I've never used gRPC with Clojure, just Kotlin, so I'm afraid I can't recommend any libraries. However, the principles are probably the same - the Leiningen plugin is a very similar approach to what Maven uses. If it isn't working for you, I'd suggest trying to generate stubs using just protoc to rule out a problem.

gavin18:03:05

Thanks 👍 Much appreciated. 🙂

gavin21:03:46

Psssst… this is a Clojure list, right? 😉

Conor13:03:45

Yes? Not sure what point you are making

gavin09:06:05

For some closure : I sorted my issue by incorporating com.appsflyer/lein-protodeps into my workflow and using clojusc/protobuf. The former really streamlined the process and the former allowed me to do the pb marshalling and unmarshalling programmatically. In the end, problem solved.

pez11:03:02

Can I create a closed spec? I found this https://insideclojure.org/2019/04/19/journal/ which I think tells me I can't yet?

p-himik11:03:50

With spec 1, you can do it by adding an extra predicate that just checks that there are no extra keys.

pez11:03:01

Thanks. Now doing this:

(def cmyk-colors #{:cyan :magenta :yellow :black})

(s/def ::cyan int?)
(s/def ::magenta int?)
(s/def ::yellow int?)
(s/def ::black int?)

(s/def ::blend
  (s/and (fn [m] (every? cmyk-colors (keys m)))
         (s/keys :opt-un [::cyan
                          ::magenta
                          ::yellow
                          ::black])))

(s/valid? ::blend {}) ;; => true
(s/valid? ::blend {:cyan 10}) ;; => true
(s/valid? ::blend {:cyan 10 :black 59}) ;; => true
(s/valid? ::blend {:cyan 10 :black 59 :green 2}) ;; => false
Was that how you meant?

p-himik11:03:44

Yep!

🙏 1
p-himik11:03:37

And if you really would like to avoid duplication, there's https://github.com/metosin/spec-tools

👀 1
p-himik11:03:04

Maybe it actually has closed specs - has been a minute since I used it.

lassemaatta13:03:33

perhaps not relevant (or even correct, it's been a while since I've used spec), but it might be a good idea to put the predicate function after s/keys in the s/and. That way ::blend works as a generator, in case you need that somewhere.

1
Al Z. Heymer13:03:34

When I use #"foo" I get a java.util.regex.Pattern, but is that actually a pre-compiled Regex-Pattern? If I call a function with a Pattern twice, is that the same object, or will it be reinstantiated?

p-himik13:03:17

> is that actually a pre-compiled Regex-Pattern? It is. Even if it wasn't - doesn't matter much because it would be compiled on the first usage. > If I call a function with a Pattern twice, is that the same object, or will it be reinstantiated? Not clear what you mean. Can you provide some code?

Al Z. Heymer13:03:27

(defn foo [s] (re-matches #"bar" s))
foo is called n times on different s. Is #"bar" the same object on every call (i.e. a 'global' instance)?

p-himik13:03:04

It will be compiled during reading, so no, it won't be re-created again and again. It will happen only once.

👍 1
Ferdinand Beyer13:03:14

user=> (identical? #"foo" #"foo")
false
user=> (defn get-regexp [] #"foo")
#'user/get-regexp
user=> (identical? (get-regexp) (get-regexp))
true

👍 2
Al Z. Heymer13:03:48

I see! thanks guys 🙂

Joshua Suskalo15:03:00

@zimablue maybe this is a bit late, but if you're using JVM Clojure you could just use an agent. Just use send-off and that will automatically serialize operations and use a thread pool designed for blocking IO.

Joshua Suskalo15:03:28

Although if it has to be CLJS compatible, then yeah, a go loop would be your best bet.

Joshua Suskalo15:03:51

I'll admit I'm a little curious why there aren't any agent libraries for CLJS, considering it's pretty easy to implement agents in terms of core.async

phill16:03:33

For the send-off case, yes. For send, the picture is not so clear. In any case, async is so much more flexible!

Joshua Suskalo16:03:27

Why would send be less clear? It's also designed for blocking, it just blocks on computation, not IO. They have the exact same API, personally I'd think actually doing what send-off is designed for, blocking on IO, would be harder in CLJS since usually "blocking on IO" has to be done via async stuff, so you'd need to have a way to have what you put on send-off be an async function itself.

Joshua Suskalo16:03:47

Which on its own would be incompatible with the JVM version

Joshua Suskalo16:03:26

although you might be able to manage something that works on both with like a thread first macro like

(fn [x y z]
  (-> (do stuff as a body)
      #?(:cljs (a/go))))

Nundrum16:03:10

I'm going to pull this question out to a larger audience because I'm still stuck https://clojurians.slack.com/archives/C0LK5TKKM/p1647569345863029

hiredman16:03:23

are you on linux?

yes 1
hiredman16:03:05

it isn't a public method

hiredman16:03:04

the default method scope in java is package private, meaning it can only be called by other code in the same package

hiredman16:03:43

which isn't a huge surprise because the classes you are trying to use are kind of deep in implementation details of awt, likely not intended to be part of the public api

Joshua Suskalo16:03:12

is there a particular reason you're trying to talk to xlib directly?

hiredman16:03:08

if the goal is to talk to xlib, there must be better native native bindings than going through awt

Joshua Suskalo16:03:28

there is this: https://github.com/geremih/xcljb that I spot immediately. It's pretty old but it's there. If you want to wrap xlib yourself you could use https://github.com/IGJoshua/coffi or https://github.com/cnuernber/dtype-next.

Nundrum16:03:48

Oh I totally missed that 😳

Nundrum16:03:40

Yes, the goal is to do some low level X11 stuff, but ultimately want to be able to use higher level toolkits. It's the "how do I draw on window with id X" problem.

Joshua Suskalo16:03:17

Well if you want to draw on a window with OpenGL or Vulkan you can use the excellent https://www.glfw.org/ library, which I have already wrapped in Clojure for you with https://github.com/IGJoshua/glfw-clj, or if you want to go for something more established but java-oriented there is https://www.lwjgl.org/.

Joshua Suskalo16:03:01

This doesn't solve the problem of drawing to a canvas inside a window that uses native or javafx for rendering the main UI components though.

Nundrum16:03:07

Yeah, but I'm trying to combine this with XScreensaver, which passes a window ID to the program doing the drawing. It's that handoff I can't get over.

Joshua Suskalo16:03:30

aaahh, so you need a way to, given an existing id, draw to it

🎯 1
Joshua Suskalo16:03:58

Well if you know how to do it via native code, you could use coffi or dtype-next as I linked before to wrap the native libraries you'd be using.

Joshua Suskalo16:03:15

I will admit I've never tried to draw to an existing window so I don't know offhand what libraries you'd need for that.

Nundrum16:03:32

It looks simple enough here: https://github.com/Zygo/xscreensaver/blob/master/hacks/screenhack.c#L903 After XtAppInitialize on 763, the code either proceeds with it's own window, draws on the given ID, or creates a virtual root window and draws on that. So doing the lower level calls to get that far shouldn't be too hard.

Nundrum16:03:42

The problem is I can't find any way to get Xlib/XCB/whatever's notion of a window+GC into the AWT/Java2D/GLX notion of the same. Which is why I was poking around deep AWT.

hiredman16:03:11

awt is not really an xlib binding though, it is a facade over xlib to provide the same awt api on x11 as on other guis, so it seems like you would be better off with a real xlib or xcb binding

Joshua Suskalo16:03:26

I would agree there

Joshua Suskalo16:03:34

using awt for this feels like a hack at best

Nundrum16:03:43

That loses all capability to use any other Java libs for drawing

Nundrum16:03:15

I'd love to use Clojure2D in it, so it looks like doing something with AWT is going to be a necessary first step?

Joshua Suskalo16:03:22

ah, so the goal is to somehow inject a window that awt didn't set up into awt so as to use drawing functionality designed for awt?

Joshua Suskalo16:03:03

That's pretty crazy. I would love to see how you end up doing it.

😆 1
Nundrum22:03:09

FWIW, this approach seems impossible. Java's encapsulation makes accessing the X11-specific parts impossible. It looks like the higher level libs - Java2D, Swing, JavaFX - all use AWT so those aren't a route in either. It would require a patch to AWT itself, and that's way beyond my capabilities. 😔

emccue23:03:36

not impossible - you got add-opens + reflection

Nundrum02:03:43

I tried reflection, but can't get around it

Nundrum02:03:52

I'll have to look into what add-opens are...in the morning 😉

emccue22:03:53

yeah so basically launch the jvm with a flag opening that specific module to the "ALL-UNNAMED" module and then reflection will work again

👍 1
emccue22:03:19

you should know what module its in by the error message you got when using reflection

Nundrum23:03:14

It did take a moment to figure that out, but it was easy to find in the AWT source. java.desktop/sun.awt.X11 I got to the point of setAccessible working this time. But can't quite figure out how to invoke functions yet. I'm trying XOpenDisplay first. (. xod (invoke nil (*into-array* Object [(*long* 0)]))) It tells me:

Execution error (UnsatisfiedLinkError) at sun.awt.X11.XlibWrapper/XOpenDisplay (XlibWrapper.java:-2).
'long sun.awt.X11.XlibWrapper.XOpenDisplay(long)'

Joshua Suskalo22:03:12

how descriptive

Nundrum22:03:30

Right? I made some progress a few days ago. Here is it in case you didn't see it and are curious: https://clojurians.slack.com/archives/C03S1KBA2/p1647809658682999

p-himik16:03:04

You mentioned you'd like to experiment with xephyr. Did it end up not working?

Nundrum16:03:42

I'd much rather do it all directly, in one place, and not rely on yet another component in between. So I was trying to exhaust other options.

Carlo19:03:48

In my deps.edn, I have something like:

:build {:deps {io.github.clojure/tools.build {:git/tag "v0.8.1" :git/sha "7d40500"}}
         :ns-default build}
and the command I use to invoke the repl is:
/run/current-system/sw/bin/clojure -J-Dghostwheel.enabled=true -Sdeps '{:deps {nrepl/nrepl {:mvn/version "0.9.0-beta3"} refactor-nrepl/refactor-nrepl {:mvn/version "3.0.0-alpha13"} cider/cider-nrepl {:mvn/version "0.27.2"}} :aliases {:cider/nrepl {:main-opts ["-m" "nrepl.cmdline" "--middleware" "[refactor-nrepl.middleware/wrap-refactor,cider.nrepl/cider-middleware]"]}}}' -M:cider/nrepl
how should this be changed so that I have the tools.build dependency for :build?

dpsutton19:03:13

you’ll want to change :deps to :extra-deps for jacking in, and then just add build to your cider-clojure-cli-aliases so it’s included with -M:build:cider/nrepl.

🙌 1
dpsutton19:03:09

but when building you probably do want to have :deps vs extra-deps since you can just replace with the one tools.build dep needed. So i bet the preferrable solution is to throw that tools.build coordinate into some dev profile

🙌 1
Joshua Suskalo20:03:23

using :extra-deps is fine if you use -T from the commandlime

🙌 1
Nundrum22:03:09

FWIW, this approach seems impossible. Java's encapsulation makes accessing the X11-specific parts impossible. It looks like the higher level libs - Java2D, Swing, JavaFX - all use AWT so those aren't a route in either. It would require a patch to AWT itself, and that's way beyond my capabilities. 😔