Fork me on GitHub
#clojure
<
2018-05-22
>
wilkerlucio01:05:01

hello, is there a way to handle unencodable values during a transit write? my case is that I'm sending a lot of arbitrary data encoded in transit, but sometimes there pieces on it that transit can't encode (like some react classes), the problem is that currently the entire encoding fails when that happens, so I wonder if there is a way to efficiently catch that transit can't encode a value and handle it by returning something else (maybe just a placeholder like "cant encode", in my case is ok to drop those values, it's more important that the rest goes though). is there a way to do that with transit?

donaldball02:05:16

You may be happiest either writing custom encoders for your unencodable types (that presumably would simply note the type) or pre-processing the data before encoding to remove such types. If the former route interests, http://blog.klipse.tech/clojure/2016/09/22/transit-clojure-2.html may be of help

wilkerlucio02:05:56

thanks @U04V4HWQ4, I understand that, the problem is about performance. the data structures I'm sending can get quite big (they are client-side fulcro app states, they can get very large), so if I walk the structure is already a bad thing for me, also the only reliable way to detect that I can't encode something is trying by trying to encode it. most of the times I don't have any bad data, so doesn't worth trying to sanitise every time, what I'm currently doing is putting the encode in a try/catch, and when it fails I encode again but running the sanitiser first. it works, but I end up having to encode everything 3+ times (considering post-walk will encode multiple times as it walks), this could be much more efficient if transit provided some hook for the user to handle cases where it can't encode something, do you think is there something like that available?

noisesmith02:05:13

why not define a custom encoder for each unencodable type? if you run into more than a few that sounds like a design problem

wilkerlucio02:05:14

@U051SS2EU it's out of my control, I'm creating a developer tool, I can't predict everything that might happen in that state, it's user controlled.

noisesmith02:05:54

Then why not document the types it accepts? You could even define a container type they can keep in the state to hold things that can't be encoded, and wire a serializer that ignores its contents

wilkerlucio02:05:00

that could be, but why should we require that? wouldn't be possible for transit to provide a escape callback for those? this would make a much better interface that doesn't require any adaptation, if I could just use the pr-str of the value instead of the original that would solve the problem

tbaldridge04:05:31

Same problem though, what if the type doesn’t support pr-str

wilkerlucio11:05:02

@U07TDTQNL then we could have a default fallback, the main point is not getting a good rep of it, is just not breaking the encode of a big value because a few items can't be encoded, makes sense?

noisesmith17:05:04

this is interesting speculation, but transit lacks any such feature

korny07:05:59

hey folks - is there an easy way to do take-while but also return the first non-matching sequence entry? Kind of a take-until?

korny07:05:13

e.g. (take-until even? [1 1 3 4 5 6]) would return (1 1 3 4) - stopping at the first even entry.

korny07:05:31

I can write it using split-with but it’s a bit ugly.

schmee07:05:23

@korny it has been proposed to add take-until to core: https://dev.clojure.org/jira/browse/CLJ-1451

schmee07:05:44

you could borrow the implementation from the patch in JIRA

korny07:05:06

handy 🙂 thanks! That also pointed me to halt-when: (defn take-until [p s] (transduce (halt-when p (fn [r h] (conj r h))) conj [] s))

borkdude09:05:09

Why do I have to provide the type hint on the var here, but not inside a let to prevent reflection?

(def dtf (DateTimeFormatter/ofPattern "yyyy-MM-dd"))
(def ^LocalDateTime now (LocalDateTime/now))
(.format now dtf)
(let [now (LocalDateTime/now)] (.format now dtf))

reborg09:05:56

@borkdude when (.format now dtf) is compiled into a class, it gets access to the Var from an instance variable to invoke with getRawRoot(), which returns an Object. The info available at that point (with no type hints) is: I have an Object, I need to call "format" on it, then use reflection. If you attach a type hint to the var (or during the call with (.format ^LocalDateTime now dtf)) the object from the var can be cast to LocalDateTime before .format call, hence no reflection. In the let block there is no var access and the compiled class has all the info it needs straight away.

borkdude09:05:00

ok, because in the let block the names are just compiled away to one direct expression?

reborg09:05:37

names are not wrapped by a Var around them so no indirection required

borkdude09:05:56

Why don’t expressions like (def now (LocalDateTime/now)) automatically add the type metadata though?

borkdude09:05:30

because Vars are mutable and might be altered to some other type?

reborg10:05:01

Yes, the Var can be redefined anytime and the object in it can change to a different type. All functions compiled at that point would have a bogus cast. You can actually see that in action comparing the error messages with and without type hinting:

user=> (def ^LocalDateTime now (LocalDateTime/now))
user=> (def f #(.format now dtf))
user=> (f)
"2018-05-22"
user=> (alter-var-root #'now (constantly "aaaa"))
user=> (f)
ClassCastException java.lang.String cannot be cast to java.time.LocalDateTime
user=> (def now (LocalDateTime/now))
user=> (def f #(.format now dtf))
user=> (f)
"2018-05-22"
user=> (alter-var-root #'now (constantly "aaaa"))
user=> (f)
IllegalArgumentException No matching method found

👍 16
skrat10:05:21

in clojure.spec, is there a way to define a map spec where unqualified key has a different spec than that of the qualified keyword? ie. say I want to have, in one ns, specs for 2 entities (maps), each having a status key, each with different spec

xtreak2910:05:00

lein removes the stack trace after some steps and says n items more. Is there a way to show the complete stack trace in lein run or other commands?

borkdude11:05:18

@reborg Another way to get the behavior I want is ^:const

borkdude11:05:31

which is actually what I intended, since this is a constant

bronsa12:05:48

beware of using ^:const for anything other than printable clojure values

reborg12:05:02

@borkdude not sure what behaviour you need, but yeah, large ^:const objects can produce huge .class files

borkdude12:05:26

@reborg I mean to prevent reflection

reborg12:05:10

can you post a snippet? It should throw if you const on something like a date

bronsa12:05:35

it’s not just about producing huge class files, they might even cause compilation to fail in certain cases

borkdude12:05:10

I might revert my last commit then. Where can I read more about this?

bronsa12:05:34

I don’t think there’s any doc on :const

borkdude12:05:11

so only use :const for “small” things like numbers, strings? why not date?

reborg12:05:06

date could be ok, but for it throws for me if I define it const, because there is no print-dup override for printing it. So wondering why it's working for you

borkdude12:05:03

@reborg are you using java.time.LocalDateTime?

reborg12:05:36

I am indeed

bronsa12:05:40

no, that’s exactly what I’m talking about, :const is only for things that the compiler can roundtrip from value to string and from string to value again

reborg12:05:21

You must have an override of print-dup somewhere, but yeah, all to avoid a type hint? Probably not worth it

borkdude12:05:37

clj -e “(def ^:const d (java.time.LocalDateTime/now))”

borkdude12:05:39

works for me

bronsa12:05:52

usage time is the issue

bronsa12:05:54

not definition

reborg12:05:05

Try to use it in an expression

bronsa12:05:25

clj -e "(do (def ^:const d (java.time.LocalDateTime/now)) (fn [] d))"

borkdude12:05:33

$ clj -e “(def ^:const d (java.time.LocalDateTime/now)) d”
#’user/d
#object[java.time.LocalDateTime 0x52815fa3 “2018-05-22T14:59:16.834"]

bronsa12:05:38

that’s not even good enough

bronsa12:05:47

the compiler will evaluate d in interpreter mode there

borkdude13:05:11

ok, I see the issue

borkdude13:05:17

I’ll use a type hint 🙂

reborg13:05:32

exactly, if you close it in a function def, it can't generate the class because it doesn't know how to convert it into a string

fabrao13:05:04

Hello all, I´m starting a project that it has to be scalabe itself. I´m researching about BaaS providers and I saw some that closed already, even http://parse.com from Facebook. I saw others that is "vendor lock-in", and others that is limited by APIs for 3rd party integration libs. So, is it worth to use BaaS? This project will integrate with credit cards payments, chatbots, machine learning, file storing, and so one ... Could anyone give me an advice what starting way do I have to go throuth?

borkdude14:05:18

@fabrao I think this is more suited for the #off-topic channel

fabrao14:05:40

ok, sorry about that

pyr15:05:10

Hey! If you run backend code, I strongly suggest looking at Sentry to centralize error reports, it's available either self-hosted or as a service. We rely on it and we just updated our client library for it at: https://github.com/exoscale/raven

lwhorton16:05:59

is there a way to integrate variadic & body functions with a multy-arity definition ([a & b] ...) ([a b c] ...)? Seems fishy, but i’m not sure how else to get at a macro definition i’m trying to write

lwhorton16:05:41

“im a macro that takes a binding, option, & body OR a binding, & body”

lwhorton16:05:26

i guess i could sniff the body, but that seemed wonkier to me:

(let [options? (map? (first body))
        opt (if options? (first body) {})]
 ...

thegeez16:05:30

@lwhorton You'll need to do the wonkier option. You can document the intended way to call the macro by using :arglists such as done here: https://github.com/clojure/clojure/blob/08e592f4decbaa08de570ded9ac169785b1608f9/src/clj/clojure/core.clj#L449

👍 4
madstap18:05:02

@lwhorton Parsing stuff like that is one of the usecases for spec, btw, (s/cat :options (s/? map?) :body (s/* any?))

borkdude20:05:26

@pyr We have been using Sentry for over a year now, very pleased. Also using your lib.

❤️ 4
grzm21:05:02

Hey, all. Anyone know of where I can get some parens stickers? I'm fresh out and need to deck out my laptop.

grzm21:05:04

(Actually, what's worse, is I have one left. Can't leave myself hanging!)

huthayfa23:05:50

hello guys how do I merge two maps and take the key in the source if the key exists in both maps. merge take the key in the source if the keys are similar

noisesmith23:05:44

I am not sure I understand your question, but you might want merge-with

andy.fingerhut23:05:47

You can merge them in the other order, or write your own variant of merge.

andy.fingerhut23:05:23

Oh yeah, merge-with could be used to prefer keeping the value in the earlier map, rather than preferring keeping the value in the later map.

noisesmith23:05:56

I think you understood the question better than I and my answer coincidentally applies

andy.fingerhut23:05:46

I am answering the question as if the last sentence was changed to "merge take the key in the later map if the keys are similar"

👍 4
andy.fingerhut23:05:21

merge behavior: (merge {:a 1 :b 2} {:c 3 :a -1}) => {:a -1, :b 2, :c 3}

andy.fingerhut23:05:39

one possible merge-with behavior: (merge-with (fn [earlier later] earlier) {:a 1 :b 2} {:c 3 :a -1}) => {:a 1, :b 2, :c 3}

huthayfa23:05:10

I need the keys in earlier if the keys are similar

noisesmith23:05:37

that's what merge already does

👍 4
noisesmith23:05:30

unless you really mean "earlier" as in the order of the map? (can't really be done without a different datatype)

huthayfa23:05:31

thanks merge-with worked