Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2020-09-04

Channels

#datomic
<
2020-09-04
>
David Pham05:09:56

With Datomic on-prem with PostgreSQL, would it be possible to increase the read capacity of datomic by adding Postgres recplicas of the database? I did not see any documention on whether it was feasible in the docs. I guess it would not make sense because of transactors.

cmdrdats09:09:11

I'm finding using enum's as refs in datomic (on-prem) extremely tedious - in order to find the enum keyword, I need to pull or query/join the :db/ident every time... writing and explicitly querying is simple enough, but where I want to do something like (case (:message/status m) :message.status/read ...) it's a pain.. is there something I'm missing? Currently, I'm heavily leaning toward just using the :db.type/keyword

pithyless09:09:58

One advantage is you can hang other attributes off of an :db/ident enum (e.g. docs, etc.). This may or may not be useful to you. If it's just a question of dealing with the pull result - and all I care about in a specific context are the keywords - I find it useful to have a helper function that will postwalk the pull result and replace all the enum maps (cardinality one-or-many) with the :db/ident keyword.

cmdrdats11:09:23

ye, I'm just not sure it's worth it when I can just keep a map about the enum's around (most might have a displayname of some sort, but that's really it).. the postwalk sounds too fiddly for my liking, and I'd much rather have us not have to think about it 😕 Trying to find out if there's another reason for using refs instead of keywords? is the indexing more efficient?

favila12:09:40

Pull with xform is another option

favila12:09:19

{[:ident-Val-attr :xform lift-ident] [:db/ident]}

favila12:09:23

Define lift-ident as :db/ident

cmdrdats16:09:28

But why do all that instead of just keyword? Besides being able to attach other metadata to the enum value, of course? Surely there's another tradeoff?

Nassin20:09:27

enums gives you more constraints (if you need that), a keyword type would accept any keyword, with refs, only already asserted enums/entities are valid

cmdrdats20:09:01

That's ok, I can work around that easily enough, even with the constraint functions.. i forget what they're called.. surely there's a more fundamental reason?

Nassin20:09:49

Using keywords directly will require more storage I guess(duplicates), instead of pointing always to the same enum.

Nassin20:09:41

you could create refs to keywords but no idea if that is less or more efficient than refs to enums

Nassin20:09:08

docs don't say, datomic being a black box, only cognitect knows

favila21:09:04

This is really just the difference between a pure value and something you would assert things about

favila21:09:29

if it’s really just a pure value: sure, keyword, fine; but if you want to assert anything about it (e.g. give it unique names, change its name, say it belongs to a set of other values in the same domain, give them aliases for different contexts, say who introduced them, give them sorting priorities, etc etc), then entities give you that flexibility

cmdrdats06:09:50

@U011VD1RDQT interesting point about the storage requirements, would be interesting to know if this is the case

cmdrdats06:09:51

@U09R86PA4 thanks, that's useful - some great examples. I guess it's about whether that data should live in code or in database

kipz09:09:20

Hi. I've setup a Datomic Cloud in us-east-1. I'm going through the ion tutorial at https://docs.datomic.com/cloud/ions/ions-tutorial.html. Everything is working until I try to clojure -A:ion-dev '{:op :push}' when I get a 403 AccessDenied trying to download the ion-dev pom/jars. I have an AWS_PROFILE environment exported, which seems to be working enough to get ion jars downloaded and the samples working up until this point (i.e. I can load the sample db, run queries etc). Any thoughts on this would be much appreciated.

kipz09:09:07

Could not transfer artifact com.datomic:ion-dev:pom:0.9.265 from/to datomic-cloud (): Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: XXX; S3 Extended Request ID: XXX)

kipz09:09:24

Incidentally, I'm curious as to why these are authenticated maven repos in the first place. Anyone know why this would be?

kipz11:09:40

This is now working, but intermittently. Unfortunately I can't say what fixed it as I've made no changes - I've just tried again. Perhaps an AWS issue?

Jake Shelby15:09:38

there were pretty big IAM issues yesterday - so it's possibly an AWS thing, depending on when you were getting the issue

kipz15:09:05

Thanks - yeah, must have been that. I'm not seeing issues doing this now at all. It didn't make sense that some artefacts could be downloaded, but not others.

Jake Shelby16:09:12

(in case anyone else is scratching their heads wondering why it looks as if there are conflicts for the latest Datomic Cloud versions not using the latest client libs ... you just have to use the actual latest version of ion-dev)

jaret17:09:26

@jake.shelby Thanks for the report! I just fixed this …an artifact in our docs build failed to get updated.

👍 3
Matheus Serpellone Nash19:09:44

Hello! We noticed our application tends to crash after a few weeks of no restarts (due to no deploys or cycles), we see increased GC activity and CPU usage until it becomes unhealthy and we have to cycle it. I used jmap to extract a thread dump of a recent pod (24h), and ran it through eclipse memory analyzer, to see if i can find any leaks. It’s telling me that there is an object datomic.cache.WrappedGCache retaining 9+GB of heap. I have attached the stack from gcroot I tried to search for this class to understand what it does, but nothing shows up. Do you guys have any advice on how to debug this? Is this Datomic’s peer getting too bloated from all the queries? Can I tune this to not take too much space? I’m a bit uncertain on how to proceed. Also, sorry if this is not the right channel.

Matheus Serpellone Nash19:09:39

Ah ok! I just also realized that the “95% usage” im getting is probably from the used heap, not total heap

csm19:09:09

Though we also see our peer processes eventually start consuming a significant amount of CPU, and inspecting the process, it’s almost all GC scans, and almost all the memory is in datomic’s cache. I haven’t figured out if there is a good way to tune the GC to handle a large in-memory cache, to avoid this CPU usage

Matheus Serpellone Nash19:09:33

What are you using to inspect the process? visualvm?

csm19:09:25

top -H -p <pid>

csm20:09:57

(and jmap)

marshall20:09:13

what JVM gc flags are you using and what heap size

marshall20:09:01

you mostly likely want to use the recommended flags from our docs: -XX:+UseG1GC and -XX:MaxGCPauseMillis=50

Matheus Serpellone Nash20:09:55

Let me confirm. I’m sure we use G1. Not sure about gc pause

marshall20:09:58

@m.serpellone this is a peer application, yes?

marshall20:09:52

keep in mind the peer is your app so you need to be aware of things like head holding/etc

marshall20:09:27

if looking for memory leaks/etc

Lennart Buit21:09:14

Can I ask more lighthearted questions here? Stuart Halloway has this party trick of using a vector as vector as a database value, what makes that that doesn’t work for the client api? Sounded like a simple way to test rules ^^

marshall21:09:52

@lennart.buit you can do that in the peer because the work of query happens in the peer

marshall21:09:12

client sends the request over the wire to the peer server (on prem) or to a datomic cloud system (cloud)

Lennart Buit21:09:28

Ah yes that makes sense 🙂! I should have thought of that

kenny21:09:43

It'd be great if ion-dev bumped its version of tools.deps. The version it is using still uses the s3-wagon-private lib which transitively brings in a logback.xml file. This causes the multiple slf4j bindings warning. I can manually bump tools.deps to 0.8.709 to work around the warning.

👍 3