Fork me on GitHub
Matt Roll00:06:07

more specifically I’m interested how people are integrating datahike/datascript into projects? Do you run clojure on a server-side framework like pedestal and have datahike as a dependency for communicating w/ say postgres?


Datascript is normally used in ClojureScript web-apps as an in-memory DB where you maintain the app state (like the UI/UX state). Datahike is used for very small applications, it be a good candidate for a desktop app written in Clojure. Or a client/server app, but with very small number of concurrent users. So maybe internal tooling and such. It doesn't scale to large dataset and it runs on a single machine, so it is made for client/server apps where your server is expected to run in a single node, thus non-distributed. Don't expect it to be resilient either, it won't replicate itself or anything, so have backups. In neither case is PostgressSQL involved. With Datascript, sometimes people have POstgressSQL as their backend DB to their webapp, and DataScript as the client-side in-memory DB used by the client.

👍 2

Otherwise, if you need high scale, resiliency, fault tolerance, you are looking at using Datomic on the backend or Crux. Those would replace your use of PostgressSQL.


So, best to think of it this way: For server backend: - Where you'd use SQLite -> use Datahike or Datalevin - Where you'd use PostgressSQL, MongoDB, etc. -> use Datomic or Crux For client-side apps: - Where you'd use SQLite -> use Datahike or Datalevin or DataScript (where you'd save/load to/from EDN on disk) - Where you'd just store data in-memory in atoms or variables -> use DataScript or Asami


Or you can always choose to also just use SQLite, PostgressSQL, H2, and other relational databases. The above is if you'd prefer using a Datalog based more graph oriented DB instead.

Matt Roll02:06:13

@U0K064KQV this is exactly the kind of answer i was looking for! Very helpful in wrapping my head around where these tools fit into the stack.

Matt Roll02:06:21

I’m looking to try a graph oriented db for a project I’m starting. It’s a relatively standard web app with a user interface in the browser and needs auth and data store. Crux sounds like the best free option?


Well, I'd say it depends what this project is. Like if its a hobby, you can start with Datalevin or Datahike. That will give you a chance to learn the whole Datalog thing. And then if you need to scale, you could afterwards consider switching to Crux.


And if your browser user interface is a SPA, you can try using DataScript in it.

Matt Roll03:06:35

Yes it’s a hobby project with no users yet so the datahike idea might be good to try. And yes it’s an SPA in react so I think I’d like to try datascript as well.

Matt Roll03:06:24

One question about hooking the client up to the server - do you just use standard JSON over http to communicate, and then stick the parsed data into datahike/datascript?


Its more typical to use Transit when your use case is Clojure backend and ClojureScript frontend, and you don't intend other languages to be using the APIs. You should look into: [preferred most commonly used] [simpler, you could just use EDN directly with this one, only downside is slower than muuntaja]


The advantage of Transit over raw JSON is that Transit can encode all the Clojure standard types like keywords, where-as JSON can't.


Using EDN directly is nice, but the disadvantage is the compression and performance of serializing/deserializing JSON isn't as optimized as Transit.


A good place to start on the server would be with: ring/ring-core ring/ring-jetty-adapter metosin/reitit metosin/muuntaja

Matt Roll05:06:28

you’re a saint @U0K064KQV this is a treasure trove of info for a beginner and gives me a solid foundation and direction to work in. Many thanks! 🙂


Asami also has disk storage now btw

👀 4
Matt Roll00:06:37

Trying to do research but there doesn’t seem to be a lot written about this that is accessible for someone new to the clojure ecosystem

Endre Bakken Stovner15:06:10

I've never used spec. Is there a way to ensure that (during development) whenever the key :mykey is used in a map the corresponding value has to have certain properties?

Alex Miller (Clojure team)15:06:15

As a blanket check, no - you need to either explicitly validate with a spec or instrument a spec’ed function to enable this

👍 2
Endre Bakken Stovner16:06:50

What is a good and fast way to find the hash of a hash-map that will be the same between different invocations of my Clojure? And preferably between different versions of Clojure.

Endre Bakken Stovner16:06:20

Like, will (hash {:a "hey"} always return the same value?


don't rely on hashes unless you've defined the hashing algo

Endre Bakken Stovner16:06:41

Okay, I'll find my own then.


what goal are you trying to achieve with a stable hash?

Endre Bakken Stovner16:06:40

I have lots of metadata about how a file was created. I'd like to turn that metadata into a path prefix so that I can easily tell whether the file was made with the current settings or needs to be recreated.


i've been down similar roads before, and hash is very unforgiving of equivalence. if you change a representation but it "means" the same thing

Endre Bakken Stovner16:06:50

I know that I will have many false negatives, but it is worth it to ensure the files from my workflows are up to date.

Faiz Halde16:06:21

how can one branch out in a transducer pipeline?

 (map parse-string)
 (xif some-branching-condition
      (comp (map extra-work) (map some-work))
      (map some-work))
 (map process))
i.e. how to implement something like xif? some-brancing-condition is a function of the input i thought the following would work but ofcourse it isn’t
(defn xif
  [f if-xform else-xform]
  (fn [rf]
      ([] (rf))
      ([result] (rf result))
      ([result input]
       (let [nrf (comp (if (f input)
         (nrf result input))))))
I can certainly get away with it by doing the following
 (map parse-string)
 (map (fn [m]
        (if (some-branching-condition m)
          (some-work (extra-work m))
          (some-work m))))
 (map process))
i wonder if there was a better way


 (map parse-string)
 (map (if some-branching-condition identity extra-work))
 (map some-work)
 (map process))

Faiz Halde16:06:21

yes certainly, for this simple case i didn’t want to introduce any hypothetical scenario just for the sake of a solution, but a construct like

(defn xif [f if-xform else-xform])
could be more expressive i don’t think its trivial to implement a linear transducer if the if-xform & else-xform in itself were some complicated transducers

Faiz Halde16:06:11

i wonder if i am hitting the limitation of transducers here? or is this possible?

Faiz Halde16:06:39

tried looking at xforms library. it didn’t have anything either

Faiz Halde16:06:00

actually i wonder if it even makes sense

Faiz Halde16:06:52

e.g. if it was then what would the following even do if some-branching-condition kept alternating for each element

 (map parse-string)
 (xif some-branching-condition
       (take 10)
       (filter extra-work)
       (map some-work))
      (map some-work))
 (map process))

Faiz Halde16:06:29

my bad, some-branching-condition is a function of the input


oh - I misunderstood here, deleting my irrelevant suggestions

Faiz Halde17:06:25

i don’t think it makes much of a sense except for simple branches as explained here;cid=C03S1KBA2

Faiz Halde17:06:16

i was probably trying to squeeze too much out of transducers 😅


from a higher level vantage point, it's much easier to use comp / fn first then optimize with transducers later

🙌 2

doing the transducing part first puts more demands on your code reading ability, is much more error prone, and can't do anything functions can't, except for doing it faster

🙌 2

(defn if-tx [test-fn true-xf false-xf]
  (fn [rf]
    (let [t (true-xf rf)
          f (false-xf rf)]
        ([] (rf))
        ([result] (f (t result)))
        ([result input]
         (if (test-fn input)
           (t result input)
           (f result input)))))))


  (sequence (comp (map inc)
                  (if-tx even? (map str) (map double)))
            (range 10))

I think this does what you want ... but I'm not sure it's a good idea ...

Faiz Halde17:06:19

i certainly won’t use that 😛 but i learnt something about the inner working of transducers today (that let binding) thanks2


I do keep thinking that it would be nice to have more expressive ways of combining transducers together ... but I can't say for certain that I've really got that problem ... I keep having doubts that I'm just creating a big mess and just writing the straight code would be easier to read and work with ... I guess it depends on how much you end up sharing transducers across modules (or whatever)

Faiz Halde17:06:41

i am actually using core async which is why I'm leveraging transducers more often. without them i end up something like

Faiz Halde17:06:47

which is fine

Faiz Halde17:06:33

anything that gets difficult via transducers can be easily plucked out into its own function that accepts from an in chan and outputs to an out chan


The transducer loops one element at a time and applies all transforms. So you can just do:

(sequence (comp (map inc)
                (map #(if (even? %) (str %) (double %))))
          (range 10))


that's a good call - there's no reason the function inside map can't contain a conditional


What is a chunked-seq? , chunk , etc. and why is there no documentation about it? In the definition of concat, why is there a check for chunked-seq?


chunking is a behavior that realizes N elements of a lazy seq instead of just one at a time


I'm not sure about the documentation


concat is preserving the chunkiness of the input


@ps just subjectively I've seen little documentation about chunked lazy collections, and when people run into bugs caused by chunking the standard reply is that if realizing 32 items for one call instead of one item per call breaks your algorithm you shouldn't be using a lazy data type anyway

Alex Miller (Clojure team)19:06:18

these functions are designed for implementors of chunked sequence functions, not for general api usage (which is why they are not part of the public docs)

Alex Miller (Clojure team)19:06:17

Rich did a great talk about chunked seqs at the bay area clojure meetup during JavaOne in 2009. I happened to be there but I don't know that it was recorded.