For the server side, you have several options: datalevin, datahike, xtdb, asami, all can do what you want. The first two should allow you to sync with datascript easily, as they were datascript port. For you specific case, one user per DB, that’s a common use case for datalevin, as one DB is just one file in datalevin.

I agree that if it is truly 1 isolated graph per user, then using a separate DB per user is fine (at least in datalevin, datahike and xtdb as far as I know). But if you ever want to query across these graphs, you might get more leverage by just adding an attribute on entities to mark ownership. That's roughly how you'd do it in any graph DB: just point at the things from a user node. As for syncing the client and server database: That's where it gets tricky IMO. There are a lot of things to consider. First you need to decide what sync means in your application and whether you favor consistency or availability. Then you might want to restrict the transactions the user can do somehow, and likely append stuff to them as well? I'm not aware of a db in that list that does these kind of things for you. CouchDB/PouchDB come to mind but those are document databases, so you'd have to build the graph on top of them.

correct

Thank you!

not actually sure that is correct though. doubles should be a symbol there to be equiv to ^doubles x

Strings also work and are the only (?) way to hint an array https://groups.google.com/g/clojure-dev/c/-rcdtIlhilI Is this documented anywhere on http://clojure.org? I did not see it on https://clojure.org/reference/java_interop

> Any advice? Extract non-obvious steps into properly named functions. Particularly, anything that's higher-order, like keep and map.

Also, when you deal with map entries it's better to use key/`val` instead of first/`second`.

Thats what I have now, k/v in the first example. Putting this is in a named function would be overkill imho as its used only once when a process runs and is not needed by any other function.

I guess that would be more a matter of writing style choice

Doesn't matter how many times something is used. Proper names and arglists are the best documentation. Right now I have no clue what your code does without carefully reading every single form. With good names, I'd be able to figure it out in a few seconds. Plenty of literature on the matter, lowering cognitive load most of the time is much more important than squeezing that extra 2% of performance.

Good point, it is a part of a larger function that is named. Its actually a binding that creates a map, which is actually described but not in the code snippet. So then that would indeed make sense to used named functions but unnecessary in this case being that its described by the binding name 😁

Thanks for the input

have a look at letfn https://clojuredocs.org/clojure.core/letfn

maybe you can make use of it here.

Thanks, will do

What's the motivation behind #(let [k (first %) v (second %)] ...) instead of (fn [[k v]] ...) or explicit key and val?

No motivation at all actually, just trying out different options. The first one works just fine, but its long and kind of hard to read if you dont know what its doing and dont read the doc

Also, you have map into {} map into {} , maybe you can shorten that into one map and one into {}

While I will guess a lot, because I don't know the exact shape of the data... - read-string isn't safe, you should use the proper parser- and if data are JSON, you can use something like org.clojure/data.json read-str with :key-fn and do that string replacement - then (filter (comp pos? val)) - then sort-by and group-by - then update-vals (I think you never modify keys)- fn will be extracted and named

I use read-string as its a small clojure map stored as string in a database not accessible from anywhere else.

Also just using let works ;)

> I use read-string as its a small clojure map Use clojure.edn/read-string as this is intended for reading data (rather than code)

thanks for the tip @U04V15CAJ did not know that existed.

Take a look at the docstring of read-string , last sentence :)

I'm unsure what the benefit of the sort-by followed by the group-by is ... Surely the second undoes the first?

Also, repeatedly running into {} seems unnecessary?

(into [] cat col) is the same as (into [] (apply concat col))

> Putting this is in a named function would be overkill imho To demonstrate the value of naming, I refactored your first example to move the anonymous functions into a let binding. (letfn would also work. I tend to favor let, unless needing mutual recursion, because of more familiarity and commonly needing to intersperse non-function bindings.) Except, one function (fn [f] (when (.contains f i) f)) uses a non-local symbol. Once the parent-function that i is scoped to was extracted, it and seemed cleanest to leave this inline. And one function (fn [m] (keys m)) was replaced by a built-in (which I expand on below, before showing the example). I really tried my very best to fight the urge to twiddle with things beyond naming functions. But for two changes, I could not hold back. First, I removed the first (into {}). You are taking a sequence of pairs, collecting it into a map, then passing that to map, which transforms it right back into a sequence of pairs before doing any work. So the into call results in 2 extra full-traversals over the collection, with no meaningful effect. Then, we have a kind-of strange callback to keep. We can refactor that, starting with a point-free use of keys.

(keep (fn [m] (keys m)) v)
; ->
(keep keys v)

; ->
(keep identity v)

; ->
(remove nil? v)

(let [pos?-clean-match-order
      (fn [[k v]]
        (when (> v 0)
          {(clojure.string/replace k #"myneva.match-order." "") v}))
      concat-val
      (fn [[k v]]
        {k (into []
                 (apply concat (remove nil? v)))})
      in-matchfields
      (fn [i]
        (keep (fn [f]
                (when (.contains f i) f))
              matchfields))
      val-in-matchfields
      (fn [[k v]]
        {k (into []
                 (apply concat (keep in-matchfields v)))})]
  (->> (get settings :myneva.match-order)
       (read-string)
       (keep pos?-clean-match-order)
       (sort-by #(second (first %)))
       (group-by #(second (first %)))
       (keep concat-val)
       (keep val-in-matchfields)
       (into {})))

Thanks for the elaborate answer, will dive into that tomorrow. You mentioned the (into {}) which I indeed used to make a sequence of maps into a map and then back into a sequence of maps. That was me struggling with my thread not accepting the sequence of maps 😶

I am sorry for not including the source data (confidential), what happens with your solution (which is nice btw) is what happened to me struggling with the thread.

; nth not supported on this type: PersistentArrayMap

Which is why I ended up putting the (into {}) into the thread

In the end I cleaned it up (not done yet) like so:

(->> (get settings :myneva.match-order)
                               (edn/read-string)
                               (keep (fn [[k v]] (when (> v 0) {(string/replace k #"myneva.match-order." "") v})))
                               (sort-by #(second (first %)))
                               (group-by #(second (first %)))
                               (into {} (keep (fn [[k v]] {k (mapcat (fn [m] (keys m)) v)})))
                               (into {} (keep (fn [[k v]] {k (mapcat (fn [i] (filter #(.contains % i) matchfields)) v)}))))

matchfields is a sequence like so:

("full-email"
 "first-lastname"
 "lastname-only"
 "birthdate"
 "customfields.[Res_ID;personCode]")

refactored a bit to remove some sensitive info

matchorder

"{\"myneva.match-order.full-email\",2,\"myneva.match-order.email-username\",0,\"myneva.match-order.first-lastname\",3,\"myneva.match-order.lastname-only\",4,\"myneva.match-order.birthdate\",4,\"myneva.match-order.customfields\",1}"

And the end result

{1 ("customfields.[Res_ID;personCode]"),
 2 ("full-email"),
 3 ("first-lastname"),
 4 ("lastname-only" "birthdate")}

I'll have a deeper look at your solution tomorrow @U90R0EPHA, thanks!

I still feel like it's more complicated than it needs to be

(let [settings    {:myneva.match-order "{\"myneva.match-order.full-email\",2,\"myneva.match-order.email-username\",0,\"myneva.match-order.first-lastname\",3,\"myneva.match-order.lastname-only\",4,\"myneva.match-order.birthdate\",4,\"myneva.match-order.customfields\",1}"}
        matchfields ["full-email" "first-lastname" "lastname-only" "birthdate" "customfields.[Res_ID;personCode]"]

        parse-settings (fn [s] (clojure.edn/read-string (clojure.string/replace s #"myneva\.match-order\." "")))
        matches?       (fn [f] (first (filter #(.contains % f) matchfields)))
        collect-matches (completing (fn [r [f i]]
                                      (if-let [match (matches? f)]
                                        (update r i (fnil conj []) match)
                                        r)))]
    (->> (get settings :myneva.match-order)
         (parse-settings)
         (transduce (filter (comp pos? val)) collect-matches {})))

Here is my first-draft solution (not taking advantage of transducers). No reason to sort when the end-product is a Map, because a Map is not a sorted data structure. If it looks like sorting "works", that is because very small Maps happen to be implemented as PersistentArrayMap for performance reasons. But that is an implementation detail subject to change at any time. If you need to display sorted data to a user later, then make sorting part of the rendering transformation. Cool tip from my example that you can also apply directly to @U0P0TMEFJ’s solution is to replace the .contains lookup function with a Set. Using a Set as a unary function returns the input value if it is in the Set, otherwise nil. This makes for a great high-perf matching predicate. I went ahead and transformed matchfields into a Set up in the let binding, to avoid repeating that transformation for every value, with a name that matches its intended use as a function (verb "match" on a single "field").

(defn map-vals [f m]
  (into {}
        (map (fn [[k v]] [k (f v)]) m)))

(let [matchfields ["full-email"
                   "first-lastname"
                   "lastname-only"
                   "birthdate"
                   "customfields.[Res_ID;personCode]"]
      match-order "{\"myneva.match-order.full-email\",2,\"myneva.match-order.email-username\",0,\"myneva.match-order.first-lastname\",3,\"myneva.match-order.lastname-only\",4,\"myneva.match-order.birthdate\",4,\"myneva.match-order.customfields\",1,\"myneva.match-order.SHOULD-GET-FILTERED-OUT\",2}"]

  (let [match-field (set matchfields)
        clean-field #(str/replace % #"myneva\.match-order\." "")]
    (->> match-order
         edn/read-string
         (filter (comp pos? second))
         (filter (comp match-field clean-field first))
         (group-by second)
         (map-vals #(map first %)))))

; =>
({2 ("myneva.match-order.full-email")}
 {3
  ("myneva.match-order.first-lastname")}
 {4
  ("myneva.match-order.lastname-only"
   "myneva.match-order.birthdate")})

@U90R0EPHA your map-vals is update-vals, and that .contains seems to be String/contains, not Collection/contains

@U01RL1YV4P7 Just realized right before you posted that my map-vals step is wrong, leads to a <list of maps of lists> instead of just a <map of lists>. Not sure what you mean about .contains. In my code, the Set match-field called as a function finds an equal string value that comes from values in matchfields, which aligns with the desired output.

Corrected map-vals function in my original post.

Look at this part from Ed's code: (filter #(.contains % f) matchfields)- .contains is called with string and string, that is String/contains, so you can't replace that with filtering with set

Oh, I see. matchfields has values that are not exact string matches, but should still match:`customfields.[Res_ID;personCode]`. Using a Set misses that.

And if you want to sort the results, replace {} with (sorted-map) in the call to transduce ... @U90R0EPHA is correct that the results in the original code are only sorted by coincidence.

Thanks for all the input everyone, I really like how this community give upbuilding/uplifting critique where you can actually learn something from. @U0P0TMEFJ Thats a real nice solution, I am going to have a deeper dive into that since it uses transduce which is something I have not yet wrapped my head around. @U90R0EPHA Your solution indeed worked partially as its returning the full match-order map keys as string inside a list and it's missing number 1. Much thanks

@U90R0EPHA @U0P0TMEFJ I was counting on it to be sorted indeed. I read that a small map would indeed return 'sorted' because it would be an array map and not a hash map. What I did not know is that would be subject to change at anytime. I have indeed used sorted-map before so that will be the preferred choice.

@U0P0TMEFJ okay, so I tried to wrap my head around transduce and completing (one big mistery for me there) and added comments behind each line in your example. Please correct me if I'm wrong, but is this a good understanding of what you wrote? (see inline comments)

(let [parse-settings  (fn [s] (edn/read-string (string/replace s #"myneva\.match-order\." "")))  ;; removes the text defined in the regex
        matches?        (fn [f] (first (filter #(.contains % f) matchfields)))  ;; checks each matchfield if it contains f
        collect-matches (completing (fn [r [f i]]  ;; r is the result, f is the matching field, i is the order integer
                                      (if-let [match (matches? f)]  ;; check if f matches
                                        (update r i  ;; if so update r using i as key and conj the match
                                                (fnil conj []) match)  ;; fnil conjoins the match into a vector while overriding the first argument
                                        r)))]  ;; if not return r (the sorted-map)
    (->> ordermap  ;; thread ordermap where the contents get applied as last argument to the next in thread
         (parse-settings)  ;; send it to parse-settings
         (transduce  ;; transduce the result
          (filter (comp pos? val))  ;; if the value of each key is a positive integer 1 and higher
           collect-matches  ;; collect the matches
           (sorted-map))))  ;; into a sorted map

Yeah. I did the string replace first, then parsed the string into the settings cos that seemed easier to me. fnil is used to transform conj so that if the first arg is nil then we use a vector instead. So it will default to a vector if the key i doesn't exist in r in the update. collect-matches is the reducing function that's transformed by the transducer (filter ...) (which replaces your keep + when).

Does that make sense?

(sorry I'm currently on my phone on a train and struggling to type)

completing takes a 2 arg reducing function and adds a few more arities to make a valid reducing function for transduce

Yes it makes sense and the outcome is the same. Breaking something that I havent used before down with comments like I did helps me understand what something does. Plus I always add comments for later reviewing.

👍

One thing isn't completely clear. If completing is a reducing function that takes [r [f i]] as arguments and because ->> is thread last that would mean that the outcome of parse-settings is applied as last argument to transduce correct? If not threaded, it would look like (transduce (filter (comp pos? val)) collect-matches (sorted-map) (parse-settings ordermap))? Does that mean that (parse-settings ordermap) gets filtered with (filter (comp pos? val)) first and then called like (collect-matches (sorted-map) ordermap) where ordermap is the outcome of the filter? Trying to get a good picture of what transduce does here

Answered my own question by breaking in down into pieces 😁

Completing returns a reducing function. It's not one itself.

I noticed when breaking down indeed, got an exception. Calling it with reduce fixed that.

Look at how transducers combine with comp Vs ->>

like so (reduce collect-matches (sorted-map) ordermap)

Yeah, but with the filter transformation applied first

yes

For understanding what it exactly does I broke it down into pieces like so:

(let [parse-settings (fn [s] (clojure.edn/read-string (clojure.string/replace s #"myneva\.match-order\." "")))
                   matches?       (fn [f] (first (filter #(.contains % f) matchfields)))
                   collect-matches (completing (fn [r [f i]]
                                                 (prn r f i)
                                                 (if-let [match (matches? f)]
                                                   (update r i (fnil conj []) match)
                                                   r)))
                   ordermap (get settings :myneva.match-order)
                   ordermap (parse-settings ordermap)
                   ordermap (filter (comp pos? val) ordermap)]
               (prn (reduce collect-matches (sorted-map) ordermap)))

not how I will use it ofcourse

👍

you can setup a end condition in recur calc

(defn foo [source-ch]
  (let [cancel-ch (async/chan 1)]
    (async/go-loop []
      (async/alt!
        cancel-ch :canceled
        source-ch ([msg]
                   (prn msg)
                   (recur))))
    (reify
      java.io.Closeable
      (close [_]
        (async/>!! cancel-ch true)))))

still another info to give: when put nil to a channel, the channel will be closed

no, nil values are not allowed in core.async you will get IlligalArgumentException on attempt to put nil to the channel

you are right! i missed some senario!

that closeable idea is pretty cool, I like that quite a bit

you don't need to actually put anything to cancel-ch, you can just close it

yes, shame on me) closing the channel is much better

no

yeah i guess i need to add the shutdown hook myself

you may want to back way up to whatever problem you are actually trying to solve

go loops are run on non-daemon threads so a running go-loop will not prevent the jvm from exiting

I find it counterintuitive that closing the result channel from the go-loop macro doesnt kill it 😅

the result channel from the go conveys the result of the go to you, it doesn't communicate anything to the go block

and the transformation of the go macro doesn't insert any kind of extra logic in your code to poll for exit conditions

does this mean for every use of the go macro I should also use something similar to above for proper handling of cancellation?

it depends

the most common pattern is for the code in the go block to check the results of putting and taking from the channels it uses, nil from a take means the channel is closed, false from a put means the channel is closed

and then using those signals to determine when to stop

for more complicated stuff I do something like the above, I pass an extra channel, often called "stop" and use that to signal the go block to stop

thanks all

https://github.com/hiredman/roundabout/blob/master/src/com/manigfeald/roundabout.cljc has some examples of that (for some reason I called it abort here)

Perhaps something like this, which explicitly specifies an initial value? At least for reduce -- I haven't yet thought about transducers:

user=> (def c1 [-1 2 3])
#'user/c1
user=> (reduce - (first c1) (rest c1))
-6

I guess that is the "split the collection initially" approach you mentioned.

If you re-read my message in the other thread @U0K064KQV, Rich doesn't advise against using transduce without an initial value. The regret was scoped to reduce without an initial value. Transduce's initial value (when not explicit) comes from an arity of the rf, not the collection being reduced

(reduce (completing -) [-1 2 3])
=> -6

to your original question, I would prefer that it return the -4 that reduce returns when provided with an initial value of 0. And its funny that (- -1 2 3) is implemented as (reduce1 - (- x y) more)

(not that that form return -4, but that’s what i would want from a reduction of that collection)

@U01RL1YV4P7 The completing does nothing really here right?

(reduce (completing -) 0 [-1 2 3])
-4

The challenge, is with a function like - or / , it's non-trivial (impossible?) to find an identity value to return when (f) is called. (- 1 1) != 1 (- 0 1) != 1 (- ? x) = x ;; no value seems to satisfy this for all x So if you want to reduce as: e1 - e2 - e3 - eN instead of init - e1 - e2 - e3 - eN it doesn't seem there's any way to do so using only transduce (or a reduce where you don't allow yourself to not provide an init value) If your f has an identity value, you can make that the init, and get the e1 f e2 f e3 f eN semantics, not otherwise it seems.

Actually, seems you could do something like:

(defn with-identity
  [f]
  (fn
    ([] ::ident)
    ([x] x)
    ([x y] (if (= x ::ident) y (f x y)))))

=> (transduce identity (with-identity -) [-1 2 3])
-6

=> (transduce identity (with-identity -) 0 [-1 2 3])
-4

Can you make it moniodal with a transformation? Like (transduce (map #(* -1 %)) + 0 coll)?

normally the identity has to be from the underlying set. (transduce identity (with-identity -) []) would be ::ident

Ya, but there's no real identity in the set here, that's why (-) is an error, unlike (+)

(transduce identity - [])
clojure.lang.ArityException: Wrong number of args (0) passed to: clojure.core/-

Seems there's some math precedence on this concept: https://en.wikipedia.org/wiki/Forcing_(mathematics) where you expand the set to allow another value that's forced into it, and then can handle that. The details go way over my head though. But I think of it as, if there's a function without an identity, you can expand it so it has one, "force" the set to also have an identity.

Reduce without an init value also behaves funny with zero or one element in the collection. > If coll contains no items, f must accept no arguments as well, and reduce returns the result of calling f with no arguments. If coll has only 1 item, it is returned and f is not called. >

(reduce str []) => ""
(reduce str [1]) => 1

Ya, it's definitely not great. But the case that I'm not sure how to handle without the current reduce is the one I described, when you need to do e1 f e2 f e3 f eN and f is a binary function that doesn't have an identity.

Something else that's weird is that the 1-ary for transducer is the completing function, so there's no way to handle a 1 element list. For reduce, the 1-ary is called for a 1 element list without initial value.

I've settled on:

(defn with-identity
  ([f]
   (with-identity f ::ident))
  ([f identity-value]
   (fn
     ([] identity-value)
     ([x] x)
     ([x y] (if (= x identity-value) y (f x y))))))

(transduce identity (with-identity -) [-1 2 3])
-6
(transduce identity (with-identity -) 0 [-1 2 3])
-4
(transduce identity (with-identity -) 0 [])
0
(transduce identity (with-identity -) 0 [1])
-1
(transduce identity (with-identity -) [1])
1
(transduce identity (with-identity - nil) [])
nil
(transduce identity (with-identity -) [])
:user/ident

(= (transduce identity - 0 [-1 2 3])
   (- 0 -1 2 3))

(= (transduce identity - 0 [-1 2 3])
   (- (- 0 -1 2 3)))

If you know there are at least two elements in the list, reduce is fine. I think the trouble with - is not just the lack of identity, but that it's not associative.

I think it might be https://github.com/matthewdowney/rich-comment-tests

@U043RSZ25HQ this is for RCF as tests, not docstrings as tests; unless it supports both?

No idea, ideally eric would tell us 😛

@U8WFYMFRU I played around with this idea some years back - https://gist.github.com/pithyless/c7d6954c055ad45b6dfd2fca6ddfcce9 In the meantime, I've seen this idea pop up several times on #C06MAR553 - so I'm sure there are at least a couple of github repos experimenting with this.

I think it's also interesting there is https://clojuredocs.org/clojure.test/with-test but nobody seems to use it (that I'm aware of).

This could be a good addition to rich-comment-tests, provided the functions whose doc strings are tested are tagged with some kind of metadata.

> What do I mean by that? Here's an example from the Clojure world. Recently there was a lot of buzz about a system for having examples in comments, runnable examples. You put the examples in your dock strings in your code. When you run this tool on it, it will generate an HTML page where those examples are runnable. Totally awesome, right? You can actually run in the browser where you're reading your documentation. Run that code.

https://github.com/generateme/metadoc maybe that one?

Maybe this one? https://github.com/lread/test-doc-blocks