Fork me on GitHub
#xtdb
<
2022-04-01
>
v3ga00:04:45

Would I be abusing XTDB if I used to for a PERSONAL CMS? I’m working on a project now that’s sort of random but I would use to to track and write MANY notes and thoughts on books I’m reading or planning to read. I also have a section where I want to track things along the lines of world news. In a way I plan on adding different journals and each one would have it’s own personality/schema etc…and I want flexibility as I may want to create odd relations between different notes/posts. Does that make sense?

tatut04:04:10

I don't see why that would be "abuse", it is perfectly capable of storing flexible documents

tatut04:04:44

there are CMS stuff made with XTDB, I've worked on one as well https://github.com/solita/lipo

❤️ 1
today-i-learned 1
nice 1
Hukka04:04:23

Do you need bitemporality, though? That sounds like logseq, athens and roam do, which are also made with clojure and datascript

Hukka04:04:20

Common term for those are PKMS, personal knowledge management system. With that short description is a bit difficult to see how a personal CMS would differ

tatut04:04:24

there are ready made mind map and zettelkasten software, so there's no need to make your own... unless you want to

Steven Deobald04:04:52

@U8ZQ1J1RR fwiw, I've been using xtdb for nearly 2 years and I've never touched it's bitemporal features other than to try them out for fun. They're totally optional and (imo) not that important. If you need them, they exist. If you don't, you're not exposed to them.

Steven Deobald04:04:50

@U06FM9QN9 I've toyed with this a bit, using xtdb as a post-anki language learning tool and PKM/note-taking doodad. In general, I've come back to other zettelkasten with a proper UI. At the moment, I'm using GitJournal and Zettlr over markdown because my data just isn't that graph-shaped, but the other Roam clones work just as well. I think the only reason to jump straight to xtdb would be if you need (or anticipate) custom functionality, beyond what the off-the-shelf tools give you. Or if you prefer a REPL to a GUI, I suppose? But that might get tiresome after a while. 😉

👍 1
Steven Deobald05:04:27

@U06FM9QN9 A sensible middle-ground between hosted services you don't control and using a raw database as a CMS might be to start with Athens (https://github.com/athensresearch/athens) and customize it if/when you need to. As @U8ZQ1J1RR mentioned, Athens is made with DataScript, so it should be pretty familiar.

v3ga05:04:09

Not immediately but possibly in the future. I'd like to have the flexibility of having it.

v3ga05:04:38

I actually cry because I had ideas for an ‘intelligent notebook’ as far back as 2008 but ideas are useless until put into motion. But to get back on topic I'm just looking to build my own thing that will function as a personal system and parts of it will be public facing. Toy project for learning and entertaining myself and a few friends. So if I do give xtdb a chance am I fine just using the rocksdb backing? At the moment I'm using postgresql but if I can get away from it and be safe with just the xtdb/datalog defaults that would be fine.

v3ga05:04:09

@U11SJ6Q0K definitely a personal project and it's not really a mind map. Not in the way you're thinking…

v3ga05:04:13

One more question…would I be ok storing basic images? Literally book covers and more than likely no more than a few hundred. Or should I just continue to put them on the file system and store the string/path?

tatut05:04:36

for a small scale project I guess storing images would be ok in the db

tatut05:04:16

but for a "real" site, I would say it is better to put images in S3 or similar for hosting

1
tatut05:04:19

as datalog doesn't really give you anything for binary blobs, you are just bloating the indexes

👍 1
v3ga05:04:22

Yeah I originally was going with an s3 bucket then I thought of the amount of images I'd have after a years time and I figure I may as well use some of the space my server will allot. Plus I hear a lot of bad experiences with aws/billing if an account is compromised etc and it's just not worth the risk to me. I'm a hobbyist programmer so getting hit with a significant bill would potentially really hurt me

v3ga05:04:18

Though if traffic scales then of course I'd adjust. I'm keeping that in mind as I build…

tatut05:04:27

if you don't have clustering needs (only 1 server), you could put them in the filesystem as well

❤️ 2
v3ga05:04:58

Yeah. Im thinking that's best…the file system

v3ga05:04:51

This lipo project is very clean by the way. Thank you.

Martynas Maciulevičius06:04:54

Hey. Can the simplified OR be used without beta releases? Or is it supposed to come out in 1.21.0?

refset10:04:16

Hey again, we haven't considered backporting that particular change so far. Is there a concern about consuming the beta? The proper release should happen in the next week or two

Martynas Maciulevičius06:04:39

I get this exception when I try to run 1.21.0-beta1 and 1.21.0-beta2 . Is there a change regarding some kind of dependency? What should I look for? I use clojure [org.clojure/clojure "1.11.0"] And for Java I use this:

openjdk version "1.8.0_322"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_322-b06)
OpenJDK 64-Bit Server VM (Temurin)(build 25.322-b06, mixed mode)
3. Unhandled xtdb.IllegalArgumentException
   Error locating module
   {:xtdb.error/error-type :illegal-argument,
    :xtdb.error/error-key :error-locating-module,
    :xtdb.error/message "Error locating module",
    :module xtdb.node/->node}
                 error.clj:   12  xtdb.error/illegal-arg
                 error.clj:    3  xtdb.error/illegal-arg
                system.clj:  109  xtdb.system.ModuleRef/fn
                system.clj:  106  xtdb.system.ModuleRef/prepare_dep
                system.clj:  131  xtdb.system/opts-reducer/f
     PersistentVector.java:  343  clojure.lang.PersistentVector/reduce
                  core.clj: 6885  clojure.core/reduce
                  core.clj: 6868  clojure.core/reduce
                system.clj:  156  xtdb.system/prep-system
                system.clj:  141  xtdb.system/prep-system
                system.clj:  142  xtdb.system/prep-system
                system.clj:  141  xtdb.system/prep-system
                   api.clj:  236  xtdb.api/start-node
                   api.clj:  223  xtdb.api/start-node
                    <client's stack trace erased>

2. Caused by clojure.lang.Compiler$CompilerException
   Error compiling xtdb/node.clj at (206:34)
   #:clojure.error{:phase :compile-syntax-check,
                   :line 206,
                   :column 34,
                   :source "xtdb/node.clj",
                   :symbol xt/submit-tx-async}
             Compiler.java: 7132  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6806  clojure.lang.Compiler/analyze
             Compiler.java: 6762  clojure.lang.Compiler/analyze
             Compiler.java: 6137  clojure.lang.Compiler$BodyExpr$Parser/parse
             Compiler.java: 8570  clojure.lang.Compiler$NewInstanceMethod/parse
             Compiler.java: 8076  clojure.lang.Compiler$NewInstanceExpr/build
             Compiler.java: 7952  clojure.lang.Compiler$NewInstanceExpr$DeftypeParser/parse
             Compiler.java: 7124  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6806  clojure.lang.Compiler/analyze
             Compiler.java: 6762  clojure.lang.Compiler/analyze
             Compiler.java: 6135  clojure.lang.Compiler$BodyExpr$Parser/parse
             Compiler.java: 6453  clojure.lang.Compiler$LetExpr$Parser/parse
             Compiler.java: 7124  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6806  clojure.lang.Compiler/analyze
             Compiler.java: 6762  clojure.lang.Compiler/analyze
             Compiler.java: 6137  clojure.lang.Compiler$BodyExpr$Parser/parse
             Compiler.java: 5479  clojure.lang.Compiler$FnMethod/parse
             Compiler.java: 4041  clojure.lang.Compiler$FnExpr/parse
             Compiler.java: 7122  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6806  clojure.lang.Compiler/analyze
             Compiler.java: 7191  clojure.lang.Compiler/eval
             Compiler.java: 7653  clojure.lang.Compiler/load
                   RT.java:  381  clojure.lang.RT/loadResourceScript
                   RT.java:  372  clojure.lang.RT/loadResourceScript
                   RT.java:  459  clojure.lang.RT/load
                   RT.java:  424  clojure.lang.RT/load
                  core.clj: 6161  clojure.core/load/fn
                  core.clj: 6160  clojure.core/load
                  core.clj: 6144  clojure.core/load
               RestFn.java:  408  clojure.lang.RestFn/invoke
                  core.clj: 5933  clojure.core/load-one
                  core.clj: 5928  clojure.core/load-one
                  core.clj: 5975  clojure.core/load-lib/fn
                  core.clj: 5974  clojure.core/load-lib
                  core.clj: 5953  clojure.core/load-lib
               RestFn.java:  142  clojure.lang.RestFn/applyTo
                  core.clj:  669  clojure.core/apply
                  core.clj: 6016  clojure.core/load-libs
                  core.clj: 6000  clojure.core/load-libs
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  669  clojure.core/apply
                  core.clj: 6038  clojure.core/require
                  core.clj: 6038  clojure.core/require
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 6114  clojure.core/serialized-require
                  core.clj: 6123  clojure.core/requiring-resolve
                  core.clj: 6117  clojure.core/requiring-resolve
                system.clj:  107  xtdb.system.ModuleRef/fn
                system.clj:  106  xtdb.system.ModuleRef/prepare_dep
                system.clj:  131  xtdb.system/opts-reducer/f
     PersistentVector.java:  343  clojure.lang.PersistentVector/reduce
                  core.clj: 6885  clojure.core/reduce
                  core.clj: 6868  clojure.core/reduce
                system.clj:  156  xtdb.system/prep-system
                system.clj:  141  xtdb.system/prep-system
                system.clj:  142  xtdb.system/prep-system
                system.clj:  141  xtdb.system/prep-system
                   api.clj:  236  xtdb.api/start-node
                   api.clj:  223  xtdb.api/start-node
                    <client's stack trace erased>

1. Caused by java.lang.IllegalArgumentException
   No single method: submit_tx_async of interface: xtdb.api.PXtdbSubmitClient
   found for function: submit-tx-async of protocol: PXtdbSubmitClient

refset10:04:58

Hey @U028ART884X thanks for mentioning this - it looks like it could be a regression specific to beta2 due to the recent tx-time import changes. Could you share your node config and a small snippet with the submit-tx call you are using please? Feel free to DM me if you're more comfortable with that. I'm surprised if there was a regression here for beta1 though, as I can't see/think what might have changed :thinking_face: Presumably your same code worked fine with 1.20.0?

Martynas Maciulevičius11:04:16

What's funny is that it doesn't happen anymore. I was using an empty config and it was crashing every time. And then I couldn't import a namespace after that (or something like that). I hope that you redeployed because if you didn't and it fixed itself then it may be a problem 😄 But I think it should crash every time because it's a problem with how things are imported and executed (the method declaration, if I'm not wrong). If it'll come back I'll paste it into this thread.

🙏 1
refset11:04:31

Huh, weird. I know protocols and REPL reloading get's confusing & broken pretty quickly. I'm guessing you weren't attempting any strange monkey patching or whatever? Or using some hot-swapping dependency magic?

Martynas Maciulevičius11:04:16

I have this in my user profile: [com.clojure-goes-fast/clj-memory-meter "0.1.3"] and this one does all sorts of interesting things. But I didn't import it into my current project. Also I restarted the REPL and then tried to start only the node. So it couldn't be that dependency. Also it crashes in various ways when I try to import it or measure XTDB on other JVMs than java 8. There are many issues with it.

refset12:04:21

Aha, well, please do shout if it crops up again(!)

1
Martynas Maciulevičius08:04:53

Hm. The exception came back when I started the REPL today. I didn't change anything in particular. And it persisted when I restarted the REPL. But then I stopped my REPL and ran lein clean on the project. And then the next time I started the REPL it was gone.

refset08:04:57

Hmm, is XT definitely only appearing once in the classpath?

refset08:04:17

feel free to dump the output of lein classpath and lein deps :tree

Martynas Maciulevičius08:04:28

I think I fixed it by running lein clean. I'll try it next time it appears. Currently classpath has only one of each: :/home/_/.m2/repository/com/xtdb/xtdb-core/1.21.0-beta2/xtdb-core-1.21.0-beta2.jar: :/home/_/.m2/repository/com/xtdb/xtdb-kafka/1.21.0-beta2/xtdb-kafka-1.21.0-beta2.jar:

[com.xtdb/xtdb-core "1.21.0-beta2"]
   [org.agrona/agrona "1.11.0"]
   [org.clojure/data.json "2.3.1"]
   [org.clojure/tools.cli "1.0.206"]
   [org.clojure/tools.logging "1.1.0"]
   [org.slf4j/slf4j-api "1.7.30"]
   [pro.juxt.clojars-mirrors.com.stuartsierra/dependency "1.0.0"]
   [pro.juxt.clojars-mirrors.com.taoensso/nippy "3.1.1"]
     [org.iq80.snappy/snappy "0.4"]
     [org.lz4/lz4-java "1.7.1"]
     [org.tukaani/xz "1.8"]
   [pro.juxt.clojars-mirrors.edn-query-language/eql "2021.02.28"]
 [com.xtdb/xtdb-kafka "1.21.0-beta2"]
   [com.cognitect/transit-clj "1.0.324" :exclusions [[org.msgpack/msgpack]]]
     [com.cognitect/transit-java "1.0.343"]
       [javax.xml.bind/jaxb-api "2.3.0"]
   [org.apache.kafka/kafka-clients "2.6.1" :exclusions [[org.lz4/lz4-java]]]
     [com.github.luben/zstd-jni "1.4.4-7"]
     [org.xerial.snappy/snappy-java "1.1.7.3"]
   [pro.juxt.clojars-mirrors.cheshire/cheshire "5.10.0"]
     [com.fasterxml.jackson.dataformat/jackson-dataformat-cbor "2.10.2" :exclusions [[org.clojure/clojure]]]
     [com.fasterxml.jackson.dataformat/jackson-dataformat-smile "2.10.2" :exclusions [[org.clojure/clojure]]] 
If it comes back I'll post the info from then. But then why would REPL and leiningen get contaminated.

refset08:04:37

Cool, thanks, and I don't fully understand how classes get compiled and cached across REPL sessions, but running lein clean definitely is a good idea when trying deps upgrades

Martynas Maciulevičius09:04:03

I don't know either. But the thing is that it worked on Friday when I was using the same configuration. And today I started everything again and got the exception. That's what was strange.

Martynas Maciulevičius09:04:36

Edit: I didn't try to run this via leiningen. It's only the REPL that was giving me this error. I think I didn't get it via CLI. So probably it's more of a REPL's problem (I use CIDER in spacemacs).

👍 1