Fork me on GitHub
#clojure
<
2021-03-11
>
zendevil.eth07:03:23

I’m querying a mongodb database which is returning items of the following kind:

({:_id #object[org.bson.types.ObjectId 0x79b7fb5 "6011a0b428dacb0004d7475b"], :name nil, :uri "dzhqroksvtvveqqnxmrclebakmighdlmkccpkidefupibqifziezasljxqvjkrmxfzuydyoucxozsbqxmodpjvvuzzkxjgckrkmz.mov"} etc...)
However, the problem is that in order to send the data to the client, I have to convert the _id which I think is called a tagged literal to a string. I do this using
(map id->str data)
where id->str is
(defn id->str [item]
  (assoc item :_id (str (:_id item))))
However, the problem arises when there are multiple fields that contain tagged literals, in which case I have to do this manually. Is there a way to send this data without having to manually find and convert each tagged literal?

hiredman07:03:24

It isn't a tagged literal, that is just how prn prints objects is doesn't know a more specific way to print

hiredman07:03:41

Depending on how you serialize the data to send it to clients etc, that may be the best place to do any transformation, since serialization has to walk the whole structure anyway

zendevil.eth07:03:15

@hiredman does that mean that I have to know beforehand what fields contain ObjectId’s before doing the transformation, or is there a way to do that automatically?

zendevil.eth07:03:13

I can’t convert every field to a string either because some are numbers

hiredman07:03:47

For objects that print like #object[org.bson.types.ObjectId 0x79b7fb5 "6011a0b428dacb0004d7475b"] the format is like #object[JVM_CLASS SYSTEM_IDENTITY_HASH TO_STRING]

hiredman07:03:43

So you would walk the data structure, perhaps using something like clojure.walk, looking for instances of ObjectId

hiredman07:03:27

Or use a something in your serialization code that allows you to customize the serialization based on types

zendevil.eth07:03:19

@hiredman what about the link that you sent? I required monger.json but that didn’t work

dpsutton07:03:39

did you add cheshire?

dpsutton07:03:56

> To use it, you need to add Chshire dependency to your project

zendevil.eth07:03:03

yes that’s a dependency

dpsutton07:03:47

and did you require monger.json?

dpsutton07:03:20

(json/generate-string (mc/insert-and-return (mg/get-db conn "monger-test") "documents" {:name "John" :age 30}))
"{\"_id\":\"6049ca80360cf43286348620\",\"name\":\"John\",\"age\":30}"

dpsutton07:03:29

user> (require '[monger.core :as mg])
nil
user> (def conn (mg/connect))
#'user/conn
user> (require '[monger.collection :as mc])
nil
user> (mc/insert-and-return (mg/get-db conn "monger-test") "documents" {:name "John" :age 30})
Mar 11, 2021 1:42:34 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Opened connection [connectionId{localValue:2, serverValue:3432}] to 127.0.0.1:27017
{:_id
 #object[org.bson.types.ObjectId 0x50f01cbb "6049c9ea360cf4328634861e"],
 :name "John",
 :age 30}
user> (require 'monger.json)
nil
user> (mc/insert-and-return (mg/get-db conn "monger-test") "documents" {:name "John" :age 30})
{:_id
 #object[org.bson.types.ObjectId 0x462cd22d "6049c9f0360cf4328634861f"],
 :name "John",
 :age 30}
user> (require '[cheshire.core :as json])
nil
user> (json/generate-string *2)
"{\"_id\":\"6049c9f0360cf4328634861f\",\"name\":\"John\",\"age\":30}"
user> 

dpsutton07:03:45

some logging and other stuff omitted

zendevil.eth08:03:08

@U11BV7MTK however, this involves making changes in the client, because earlier I was sending an edn and now it’s a string.

zendevil.eth08:03:45

albeit the edn gets converted into json, which is not the case here

zendevil.eth08:03:50

so I resorted to doing: (-> data json/generate-string json/parse-string)

hiredman08:03:26

If you are sending edn I would likely use clojure.walk to transform the structure before printing. You could add a print-method for ObjectId, but that is a global thing (I try to avoid that kind of thing)

ABeltramo13:03:32

Hello everyone! Don't know where to post this, can someone point me out to a clj-commons/manifold channel? Or can I ask help over here?

p-himik13:03:14

There doesn't seem to be such a channel, so here should be fine.

ABeltramo13:03:48

Thanks, I'll write my question then

flowthing13:03:54

You could try #aleph if you don't get a response here.

ABeltramo13:03:34

Thanks! I'll try there if no reply comes here

ABeltramo13:03:55

I'm using manifold and I can't wrap my head around splitting a stream: here's an example code:

(let [pos-stream (ms/stream)
      neg-stream (ms/stream)]
  (->> value-generator 
       (ms/->source)
       (ms/map (fn [value]
                 (if (pos? value)
                   (ms/put! pos-stream value)
                   (ms/put! neg-stream value))))
       ; I need to fill streams, how should I do that? 
       (ms/consume identity)
       )
  
  (->> (ms/zip pos-stream neg-stream)
       (ms/map (fn [[pos-value neg-value]]
                 (process pos-value neg-value)))))
A simple example that should capture my problem: I have a fn that generates values and put it into a stream. I need to process that stream in pairs based on a condition, how can I accomplish that? My idea is explained in the code example: map over the stream, fill two separate stream and then zip them in order to process in pairs. The issue is that the pos and neg streams are not joined with the drain stream, I think that the code above will keep generating values forever (when would consume stop?). Is there a better way to implement this?

lilactown15:03:32

I'm not an expert in manifold, but what I would think to try first is putting all values from the generator on one stream, and then filter the one stream

ABeltramo15:03:15

Thanks for your help! I don't know if I'm doing it wrong but filtering doesn't work for me. The first filter who discards messages will not propagate to the other stream.

(defn split-stream [condition stream]
  [(ms/filter condition stream)
   (ms/filter (comp not condition) stream)])
I tried something like this but it doesn't work. I guess I could have two streams but that will mean discarding valid inputs.

ABeltramo15:03:11

You can see a full working example on #aleph a value generator can be as simple as rand

👀 3
lilactown15:03:41

when you say it doesn't work, what does that mean?

ABeltramo15:03:21

if I use two filter with a single stream and opposite conditions it loops forever.

ABeltramo15:03:36

Using two streams and consuming them I get the following warning:

WARN manifold.stream.default - excessive pending puts (> 16384), closing stream

ABeltramo15:03:47

Which makes sense since the generator is infinite

ABeltramo13:03:31

Or is this fine since you can't put more than one element in the stream until it's taken away?

borkdude13:03:33

I'm assuming the answer is no, but worth a try: Is there a trick to "disable" an interface on an object instance after the fact? E.g. (def obj (reify Interface1)) (instance? Interface1 obj) ;; true (disable! obj Interface1) (instance? Interface1 obj) ;; false

Alex Miller (Clojure team)13:03:39

No, I don’t even think that’s possible via java agent trickery

borkdude13:03:45

The problem I'm struggling with: I'm trying to support reify in a clojure interpreter that runs in an environment that doesn't allow you to define new classes. The trick I've used is to support reify with a fixed list of interfaces. A compile time function creates instances that implement all of the interfaces, but the instance checks at runtime for methods it has available provided by the caller of reify. This works well. Until... you pass those objects to core functions that have a series of instanceof checks. E.g. the objects always implement IFn so (fn? obj) always returns true, even if you didn't provide an implementation for IFn, since this is a compile time decision.

borkdude13:03:01

I'm inclined to roll back this idea, unless there is a way out.

borkdude14:03:08

The other option I've considered is to support groups of interfaces that are commonly used together and then you'll have to implement them all

borkdude14:03:51

The problem there is that if we add an interface to one of the groups, it will break existing callers.

borkdude14:03:59

I have also tried the "every subset" approach, but this blows up very quickly. For 10 classes this requires pre-generation of 1024 classes. For 100 classes it's already not feasible anymore.

nilern14:03:25

Have you looked at java.lang.reflect.Proxy?

nilern14:03:06

Probably it counts as defining a new class tho :thinking_face:

borkdude15:03:57

> Therefore all dynamic proxy classes need to be generated at native image build time. Same kind of problem, but there you compile the image yourself, so the classes are adapted to your specific use case. In the above case it's a general use case.

borkdude15:03:21

So the main problem is false positives of instanceof towards the true side and runtime exceptions that result from it with "not implemented". E.g. when a data structure doesn't explicitly implement Counted you will get an exception for (count obj), because count thinks it's a Counted instance (it is, but it is not)

nilern15:03:13

Clojure kind of relies on Java being Smalltalk in C++ clothing. When it becomes C++ with GC, you are going to have a bad time with reify etc.

parens 6
nilern15:03:10

I guess you could patch clojure to replace instanceofs with something that has enough indirection but that does not feel very promising either

borkdude15:03:24

oooh haha, I could patch the Clojure that goes into babashka's compiled artifact. Hmm, that would be a major hack.

borkdude15:03:59

but even if I would do that, I would have to patch any other library that is bundled with babashka as well, if they do manual instance checks. not going there

borkdude15:03:22

it may all not be worth supporting reify with interfaces in bb. Protocols work relatively well since they have a smaller surface area

nilern15:03:31

I wonder if these issues would just go away with Truffle. Even if they did, it would be a lot of work as well

borkdude15:03:55

There is a Java on Truffle interpreter now that may allow for these things

borkdude15:03:10

joinr did some experiments on getting the clojure compiler running. It's terribly slow though

borkdude15:03:20

But this could change in the future maybe

nilern15:03:57

> TruffleClojure: A self-optimizingAST-Interpreter for Clojure > Master’s Thesis

borkdude15:03:12

yeah, I've seen that. the source code isn't available and it's based on a very old truffle version

borkdude15:03:46

I've been thinking of writing my own, but this will be another major project and it's not certain that this will solve most of these problems

borkdude15:03:17

It has the same kind of limitation: > Java interoperability works in the native configuration but requires more setup. First, only for classes loaded in the image can be accessed.

nilern15:03:50

I assumed that making a Ruby class that extends a Java class (that is in the image) would work

borkdude15:03:43

You're assuming that they implemented Ruby classes using Java class hierarchies, which doesn't have to be the case

borkdude15:03:59

But maybe they do

nilern15:03:46

The Ruby object model has more stuff but I it would not be very useful if you could not inherit from Java classes

nilern15:03:00

I sometimes think of making a new TruffleClojure when I get ahead of myself

borkdude15:03:03

Generating Java Classes at Runtime JRuby supports converting a Ruby class to a concrete Java class using become_java!. TruffleRuby does not support this. We recommend using a proper Java interface as your interface between Java and Ruby. That's also interesting.

borkdude15:03:44

If they allow creating "reified" objects that implement interfaces at runtime though, then I want to know if that is possible at runtime in a native image.

borkdude15:03:39

I posted a question in their channel: > Does TR allow creating objects that implement Java interfaces at runtime, in a native-image? How is this achieved?

nilern15:03:48

You could also just try it

borkdude15:03:41

Hmm, where in the source do you indicate that you're implementing that interface though?

class EthylAlcoholFluidForce
  def getFluidForce(x, y, depth)
    area = Math::PI * x * y
    49.4 * area * depth
  end
end

nilern15:03:58

It's weird, why can you compile TruffleRuby into a native image but not ye olde classloaders and bytecode JIT? I think Jikes RVM was more flexible than this 20 years ago...

borkdude15:03:55

Ah what they do there: on the Java side they create something which implements the interface which then delegates to the Ruby methods

borkdude15:03:14

this is what I also do with the reify trick, only the list of interfaces each object implements is too broad

nilern15:03:52

Sort of like gen-class

nilern16:03:19

Hmm. Pure Clojure libraries might not be too bad. Any interfaces that correspond to protocols can be skipped in favor of the extend-protocol mechanism (I assume you already do this) and instance? could be easily replaced. But with Java code some craziness like patching bytecode or Java on Truffle would be required.

borkdude16:03:45

@nilern Yes, instance? can be replaced, but not for built-in libraries (without changing them) that are part of the babashka std lib

nilern16:03:21

I meant replacing the Clojure instance? in addition to the sci one

borkdude16:03:25

This still won't help for functions that call into clojure.lang.RT

borkdude16:03:55

The surface area of some of these interfaces is just too big to start doing this, I think

nilern16:03:31

I think there is no 100% solution. Like IFn does not have overloads for primitives beyond arity 4. In such situations I try to focus on making the tool useful and the limits clear.

borkdude16:03:35

I realize there is no perfect solution, but I have to choose between making things work + unexpected bugs, or not support it at all E.g. (fn? (reify ILookup (valAt [this] :foo))) ;;=> true

borkdude16:03:23

as long as the reified object has limited usage, this might be fine. I'm not entirely sure about the consequences if you actually call the above object as a function, you will get "not implemented" as a runtime error

nilern16:03:54

I think it is better to have very little support or no support at all than unexpected bugs. But I am probably biased by the static typing literature where undecidability and especially unsoundness are to be avoided like the plague.

borkdude16:03:50

Yes, I tend to agree with this

naomarik16:03:04

hey everyone, been using windows forever and just got an m1 mac so I'm not tied at home. is everyone using the x64 adoptjdk8 for datomic/shadow-cljs?

nilern17:03:07

I use OpenJDK from my package manager. I used to use https://brew.sh/ on Mac.

walterl18:03:36

Is there anything that allows you to write Clojure[Script] code directly in <script type="text/clojure"> HTML tags? Like what Brython does for Python. https://brython.info/

Richie02:04:34

Just in case you haven't already seen this. https://babashka.org/scittle/

👍 1
babashka 1
borkdude18:03:26

@clojurians-slack100 Possibly with self-hosted #clojurescript, Klipse (= self-hosted CLJS) or #sci

👍 3
nilern18:03:32

I may have seen a demo years ago. It would require self-hosted, which you usually don't want to ship

walterl18:03:14

Why not? Size?

nilern18:03:18

Yes and running the compiler on every page load would also be slow (and a bit weird)

👍 3
borkdude18:03:15

This site uses #sci https://nextjournal.github.io/clojure-mode/ which is probably the smallest bundle size

borkdude18:03:09

However, you can get better perf and bundle size if you just compile a .js file custom for your site

walterl18:03:38

Of course 🙂

andarp19:03:03

Are there any Clojure webassembly projects or plans?

blak3mill3r19:03:31

related but this is not compiling clj to wasm

blak3mill3r19:03:57

I suppose GraalVM is getting us closer to this...

richiardiandrea21:03:47

Hi all, I seem to never remember if there is way to get a spec, say

(s/def ::event
  (s/keys :req-un [::id ::payload]))
and override only the ::payload when defining a specialization of it

Alex Miller (Clojure team)21:03:49

where ... might need to be a predicate that looked inside the map

richiardiandrea22:03:05

would that be the preferred route over a multi-spec (I have just recalled I could go that route)

richiardiandrea22:03:58

this is your solution implemented

(s/def ::file-id ::model.file/id)
(s/def ::file-deleted-payload
  (s/keys :req-un [::file-id]))

(s/def ::file-deleted-event
  (s/and ::event-bus/event ::file-deleted-event-payload))

richiardiandrea18:03:30

FWIW the above would not work cause it does not have a :payload key