Clojurians Log v2

i’m always annoyed at the default reporter going to a temp file and not the console. But what’s in the full report?

the full report contains 221 lines. I’m in the process of reading it to see if I can figure out what’s happening. Certainly the problem is that I’m refactoring macros and trying to debug them. So certainly there are errors in my code I’m still trying to find. Just that clojure is not being overly helpful.

Here are the first few lines in the log file.

{:clojure.main/message
 "Unexpected error (ClassCastException) macroexpanding rte-case at (rte_case_test.clj:51:14).\nclass clojure.lang.PersistentList cannot be cast to class java.util.Map$Entry (clojure.lang.PersistentList is in unnamed module of loader 'app'; java.util.Map$Entry is in module java.base of loader 'bootstrap')\n",
 :clojure.main/triage
 {:clojure.error/phase :macroexpansion,
  :clojure.error/line 51,
  :clojure.error/column 14,
  :clojure.error/source "rte_case_test.clj",
  :clojure.error/symbol rte-case,
  :clojure.error/path "rte_case_test.clj",
  :clojure.error/class java.lang.ClassCastException,
  :clojure.error/cause
  "class clojure.lang.PersistentList cannot be cast to class java.util.Map$Entry (clojure.lang.PersistentList is in unnamed module of loader 'app'; java.util.Map$Entry is in module java.base of loader 'bootstrap')"},
 :clojure.main/trace
 {:via
  [{:type clojure.lang.Compiler$CompilerException,
    :message
    "Unexpected error macroexpanding rte-case at (rte_case_test.clj:51:14).",
    :data
    {:clojure.error/phase :macroexpansion,
     :clojure.error/line 51,
     :clojure.error/column 14,
     :clojure.error/source "rte_case_test.clj",
     :clojure.error/symbol rte-case},
    :at [clojure.lang.Compiler macroexpand1 "Compiler.java" 7551]}
   {:type java.lang.ClassCastException,
    :message
    "class clojure.lang.PersistentList cannot be cast to class java.util.Map$Entry (clojure.lang.PersistentList is in unnamed module of loader 'app'; java.util.Map$Entry is in module java.base of loader 'bootstrap')",
    :at [clojure.lang.APersistentMap cons "APersistentMap.java" 45]}],
  :trace
  [[clojure.lang.APersistentMap cons "APersistentMap.java" 45]
   [clojure.lang.RT conj "RT.java" 697]
   [clojure.core$conj__5474 invokeStatic "core.clj" 87]
   [clojure.core$merge$fn__6047 invoke "core.clj" 3073]
   [clojure.core$reduce1 invokeStatic "core.clj" 946]
   [clojure.core$reduce1 invokeStatic "core.clj" 936]
   [clojure.core$merge invokeStatic "core.clj" 3072]

i find that at first i think it’s overwhelming. But after a second it can tell you precisely where the error is. Here it’s telling you in ret-case-test on line 51 it had an issue. and if there are more stack frames it will tell you precisely where that issue is

what is java.util.Map$Entry ? is this what java thinks the clojure hash map is? or is that something totally different?

(-> {1 1} first type)
clojure.lang.MapEntry

f=> (-> (doto (java.util.HashMap.) (.put 1 1)) first type)
java.util.HashMap$Node
f=> (ancestors *1)
#{java.lang.Object java.util.Map$Entry}

it looks from the error message like you've done something like (merge {} (list (list 1 2)))?

where do you see that?

you’re on the right track. I’m doing that operation quite a bit in my dubious refactoring

except that i’m hopefully merging hashmaps with hashmaps

the error you got is inside of merge and it’s having an issue because it got a list where it expects a map’s entry

> clojure.lang.PersistentList cannot be cast to class java.util.Map$Entry This means something was expecting to find a map entry, and it found a list. > [clojure.core$merge invokeStatic "core.clj" 3072] This in you stack trace means you've called merge.

yes I see that. but how did you reconstruct your clever guess (merge {} (list (list 1 2))

PersistentList ?

that’s putting a list where a mapentry would be. Often things are seq’d on your behalf. And this is putting a list instead of a mapentry inside of a sequence

I found the error in the mean time. I had a helper function which was returning a vector of [map list] and I was destructuring it at the call site as [list map]

then merging with what should have been a map, but wasn’t

> i’m always annoyed at the default reporter going to a temp file and not the console Just in case: -Dclojure.main.report=stderr.

yes i sprinkle that everywhere and forgot to come back to it

thanks for the reminder

examples of fns you are talking about?

into / update-vals / select-keys

conj, assoc, disj, dissoc, empty, etc.

any fn that returns an updated version of a user-supplied collection will preserve meta

all of the examples above fall into that principle

Yep.

unlikely to be added to all docstrings

my 2c

describing a principle in one place more likely

why would it be unlikely to be added to docstrings?

It'd be most productive to let the core team answer whether or not it will be added, rather than having a meta discussion of what's likely or not likely.

That would include @ghadi so it seems like you have...?

as a general principle, collection ops preserve meta, seq fns don't

Oh, that's my mistake then. I was under the impression the core team consisted of Alex, Fogus, and Jarrod.

we adopt Ghadi sometimes :)

he's too useful in too many other places for us to get him all the time

I apologize, Ghadi.

> as a general principle, collection ops preserve meta, seq fns don't Alex, is this documented anywhere?

a notion similar to this came up recently. Someone used a (sorted-set) to sort an argument to clojure.set/difference and I pointed out they needed to sort the return value of it. It turns out that the function uses the first argument as the return value and disjs from it. > clojure.set/difference > ([s1] [s1 s2] [s1 s2 & sets]) > Return a set that is the first set without elements of the remaining sets But it’s not clear to me if that’s an implementation detail or if that’s something that can be relied on. Not exactly the same as metadata but not too far from it either I don’t think.

I doubt it, but this would be a good topic for future dialect discussion group

Like, we all get that it works like this, but if it's not documented, and won't be considered for documentation, then that implicitly means you're suggesting it not be considered for testing as well. Correct me if I'm misinterpreting.

I can also just table this for future discussion.

sometime like this should be documented somewhere, so def a good topic. the difficult part is inferring from current behavior where it's intended vs where it's incidental

Agreed.

docstrings (or ref docs) should declare this where it is (and that is the case in many places I think)

in general we don't repeat the principle at all the callsites, but rather state the principle somewhere. Generally the ns docstring is good for stating principles, but not for core as it has broad functionality

are there any exceptions, like dissoccing everything preserve meta?

I am open to an alternative solution which documents this in another place, but the challenge there is that it needs to be clear enough documentation to distinguish which core fns should be expected to preserve meta and which should not. Naturally, doc strings handle that on a per-fn basis, but a broader approach would need to be more delicate in its wording. I think perhaps a good next step is to get a list of all fns which are currently doing this so we can decide if there are any in that list which don't earn a promise for preservation. As Ghadi pointed out, we can also consider any edge cases for each of them and then decide from there how/where to document.

Regardless of contract, it would be useful if someone said “heads up, Clojure 1.13.0-alpha3 breaks metadata on merge”

If this is agreeable, I will be accountable (though perhaps not responsible) for that before resuming discussions.

This is also useful to know for squint. Right now it's messy 😅

ISTR that over time, some fns have surprised ppl by not preserving metadata and some of those situations were deemed bugs and fixed. Regarding dissoc of the last key -- that preserves metadata, at least today:

user=> (def m ^:hello! {:a 1})
#'user/m
user=> (meta (dissoc m :a))
{:hello! true}

@dpsutton it's either "breaks" or "changes" depending on what it promises and that's the fine details that matter here

hyrum's law aside etc

And if it's "breaks" then that version would never happen. 😛

yes. I was thinking it should be deliberate. I imagine unanticipated change might be good signal.

and thinking it would be tests in third party stuff like coal mine or the other corpus based ones. Those failures don’t constrain core, but is good to know.

but going back to the general principle - • collection fns generally take collection as first arg, return instance of "same" collection, and preserve meta which is a property of the collection. • sequence fns take a seqable last, return a seqable (or a sequence - this is another very tricky point which goes to laziness), and don't try to preserve meta there are lots of gray areas where a fn has aspects of both, or neither, and some observed behavior that may be at odds with the above

where it's unclear, I think it's fair game to want the docstring to clarify

or possibly a reference doc that can make blanket statements w/ exceptions in the docstrings

How would you like the next steps to look, Alex?

I'm in no rush on this, but I do want to make sure we have a clear way forward.

topic for dialect discussion

can put it on Ask if that puts a pin in it for now

Ok, we can table for now and resume for more discussion on it later.

Thanks for the input, everyone. 🙂

the walk functions were also fixed in a recent release wrt to metadata

prob a better place for this thread would have been in #clojure-dev btw

You're right, Alex. I will keep that in mind for the future.

It's pretty straightforward. First, you store your events in a database or a message queue (Postgres/Kafka for example). Each event has a field "type" or "action". Then you write code that accepts an initial state and a seq of events. Usually, the code is either a single case form where you branch on the type/action field. Or you write a multimethod with per-type implementations.

Alternatively to what Ivan Grishaev suggested, instead of Pg/Kafka you can use Rama! At least check it out — it's event sourcing + materialized views done right

Thanks I'll check that out! Always happy to find stuff that will prevent me from shooting myself in the foot :)

Ok, what exactly is this? A library? A SaaS? Paas?

It's rather a framework/platform which comes with its own builtin database, message queues, background workers and so on. But I've never tried it.

If you’re interested in event sourcing, you should take a look at datomic!

For data at scale, I tend to prefer transformation after loading, so that I can use faster primitives in databases that are purpose built for this. i.e, ELT instead of ETL. It also separates data transfer and data transformation, which are different problems wrt reliability. With kafka, you have off-the-shelf sinks that you’d be able to use. You also have storm/flink/spark to take inspiration from. Storm should have the declarative transformation you’re talking about.

I probably abused the term here a bit. In my understanding datomic provides event sourcing basically out of the box as more of an implementation detail. I guess what I was actually asking about was projecting state from an append only log of transactions on the level of business logic, to build a view on those data. That might technically be very similar but the "events" are actually entities already. So proper event sourcing in that case would probably mean that these events/transactions would in turn be created by projecting an event log that tracks their state over time. I hope this is not too confusing

You can also look at real-time analytics databases like pinot. They do the state aggregation you’re talking about out of the box, if it’s simple enough.

> those specialized reducers have turned out to look very imperative to me could you elaborate?

The short version is: I was trying to build a tree like structure from a flat event log. So you basically have something like Level 1 Started, Level 2 started, Several events with more information on level 3, Level 2 end, Level 1 end. Then the reducer that would rebuild the implicit tree structure would have to keep track of what level is currently "waiting for an end". If you want to see code, this commit will provide a comparison of the two approaches I tried: https://github.com/aaronmeinel/barbiff/commit/fc9efe6ce1845e8fe1b90ab593f2306d2e4a3aad

But no worries if you can't be bothered to read through all of that

I think a lot of the event sourcing stuff is just a reduce over some state, which is a really good fit for transducers cos they're transformers for reducing functions.

I've designed, built, and maintained an event sourced (CQRS) system that was the glue between user facing stuff and other services. There's a lot of nuance that you won't hit until you're in the weeds of the impl. Feel free to reach out if you'd like to chat in more depth.

I do agree that the idea of "reducing your events onto your state" is a good starting point.

Cool, I'd love to learn more about this from someone who has actually done it. Most resources I see online stay pretty high level and I've never really seen the pattern in action (as in seen the code of a system running in production) so my exercises are a lot of guesswork

re: reducers looking imperative IMHO that's a problem if you are creating side effects outside the reducer, but the reducer itself is going to be made of imperative steps, that's how its supposed to work