Fork me on GitHub
#datomic
<
2019-02-15
>
Per Weijnitz10:02:13

Hi! Can someone recommend an example project demonstrating unit testing of functions that write/read to datomic, please? I read a blog post about Yeller's testing, which seems fine, but as a beginner I would benefit from actual code to study (they refer to a function empty-db which is not included).

val_waeselynck12:02:12

Clients or Peers ?

Per Weijnitz12:02:02

Hi! I am still a bit unsure about this terminology, as a peer server, iirc, sits between a traditional client and the datomic server. The functions I need to test boil down to calls to "(d/transact (get-conn) {:tx-data ...data..})" and "(d/q some-query (get-db))". I thought it could be a suitable unit test to verify that the program can assert and read back stuff using these functions as expected.

Per Weijnitz12:02:53

But perhaps it is better to refactor these functions to leave out these mutable calls, and write tests only for these immutable bits (that go before the actual datomic calls).

Per Weijnitz12:02:37

Still, if it were feasible to create unit tests that could use datomic (without actually destroying anything in your real datomic instance), it would be possible to do testing of functions that compose a number of datomic transactions.

val_waeselynck13:02:14

Consider using d/with as well

Per Weijnitz13:02:20

Thanks! Yes, d/with seems like the way to go. If I understand it correctly d/with can make safe unit testing possible, if you provide a connection to an actual datomic instance. It could be handy to be able to setup/teardown throwaway in-memory databases as well, so your main instance does not need to be available to run the tests.

val_waeselynck13:02:27

You want Peers to do that

Per Weijnitz13:02:30

Ah! That is very, very interesting, thank you!

👌 5
twl8n15:02:00

It seems like Datomic Ions grabs clojure.tools.log which prevents userspace code (my ion function) from using normal logging. What is the recommended way to do logging from an Ion function?

lilactown15:02:29

@tlaudeman what I've done is use cast/event 😕 I know it's not exactly what it's meant for, but cast/dev can't currently be shown in the cloudwatch logs

lilactown15:02:14

if there's a more official answer I'd like to know. by "not what it's meant for" I mean that I've been using it to occasionally troubleshoot certain issues, not log actual events to cloudwatch

marshall15:02:27

^ cast is the “official” answer 😉

lilactown15:02:44

FWIW I find the biggest struggle with ions atm is how opaque the system can be when things aren't quite working

lilactown15:02:37

is the issue in the lambda? in the api gateway? in the Ion? where do I get the feedback for it? Having a way to direct cast/dev somewhere would be an improvement

marshall15:02:54

^ that should get a feature request 🙂

marshall16:02:00

i’ll see about making one for it

lilactown16:02:14

😄 you're right. thank you!

marshall16:02:15

any reason cast/event isn’t suitable?

marshall16:02:31

incidentally, that is what we use internally

lilactown16:02:42

ah, it has been once I figured it out... just per the docs it seemed like not what it was meant for

marshall16:02:47

for that kind of investigation/debugging

marshall16:02:07

maybe i should work on improving that - that is definitely the intention

lilactown16:02:15

> An event is an ordinary occurrence that is of interest to an operator, such as start and stop events for a process or activity vs. > Dev is information of interest only to developers, e.g. fine-grained logging to troubleshoot a problem during development

marshall16:02:19

dev is for local dev; event is for getting things wired up

marshall16:02:53

Yeah, I think we could improve that description

👍 5
twl8n16:02:02

Soo... I guess we should have a thin logging abstraction layer. I'm bringing over legacy code that is rife with (log/infof "Important thing happened to id: %s" my-id). There are also implications for local unit tests outside of Datomic Cloud.

marshall16:02:35

I wouldn’t hesitate to use cast/event for that purpose; you can always ‘switch’ it on the env or parameters

marshall16:02:49

if you don’t want to remove it once you’re done using it

lilactown16:02:04

so e.g. for some reason my api stops working after I deploy the change. in reality, it's a problem with coercing the response payload properly. I'd like to log the the response payload. I'd initially reach for cast/dev since that sounds more applicable, but in actuality I want cast/event.

lilactown16:02:18

I gotcha. so it is supported, just I misinterpreted the intent of the tools

marshall16:02:10

i’ll look at how we might improve the docs i would consider dev for things you’d never want to end up in your prod system, events are for anything that you would want there, either for monitoring or troubleshooting

marshall16:02:52

i kind of think of dev as the ion equivalent of println 😉

mss19:02:49

hey all, beginner datomic/clojure related question. I have an attribute I want to use to do a lookup, a :user/id that’s a datomic uuid. schema looks like:

{:db/ident       :user/id
 :db/valueType   :db.type/uuid
 :db/unique      :db.unique/value
 :db/cardinality :db.cardinality/one}
when actually using the attr to do a pull, I’m running into an issue where datomic can’t recognize the uuid as such without the #uuid reader literal, but using the reader literal breaks the compiler because the actual value is a var. rough example here:
(defn my-var-fn []
  "users-uuid")

(let [user-id (my-var-fn)]
  (datomic/pull (datomic/db conn)
                '[*]
                [:user/id user-id])) => nil

(let [user-id (my-var-fn)]
  (datomic/pull (datomic/db conn)
                '[*]
                [:user/id #uuid user-id])) => compiler error

(datomic/pull (datomic/db conn)
              '[*]
              [:user/id #uuid "users-uuid"])) => pulls data properly
any tips for how to get around this?

Lennart Buit20:02:14

(UUID/fromString ...) instead of using the reader tag, that would at least solve your issue 😛

mss20:02:48

exactly what I was looking for. knew there had to be some dead simple solution to this like that 😂

mss20:02:50

thanks for the help

Lennart Buit20:02:19

reader literals work… well only with literals ^^

pvillegas1220:02:12

Trying to upgrade my datomic cloud system. Currently have three stacks, two of which are nested (compute and storage stacks under a common ancestor). When upgrading the compute stack I get a warning telling me I should upgrade the parent instead. Does it matter?

johanatan22:02:16

are backups created by backup-db compressed? I'm seeing my dynamo backing store usage at 236.94MB but the s3 backup created with bin/datomic backup-db ... is only 50MB. Is this normal?

johanatan22:02:29

there shouldn't be any garbage in this system. i care about retaining the full history

favila22:02:38

to expand on what he said, backup-db follows the roots to the branches so it doesn't have garbage

favila22:02:56

(a fresh backup db; an incremental one will have garbage)

johanatan05:02:11

so... after running gcStorage, my dynamo db is still: 189.30MB yet the backup is 50MB. is this normal?

favila15:02:00

It seems high but not bonkers high

favila15:02:10

What was your ago time?

favila15:02:31

Are there any other dbs in there?

favila15:02:58

Did you delete any dbs? (There’s a separate command to reclaim deleted db space)

johanatan22:02:14

my ago time was: 1 day

johanatan22:02:18

no other dbs

johanatan22:02:36

i don't think i've ever created any other dbs so shouldn't be any dbs to reclaim

johanatan23:02:47

hmm, dynamo now says 76.79MB. I think it may have been the delayed stats update

marshall22:02:08

sorry, I meant to type more but I was literally on my way out the door

johanatan22:02:22

hmm, what is "garbage" in this context? if i only ever write new values, will garbage still accumulate?

favila22:02:43

datomic stores datoms in compressed blocks of fressian

favila22:02:56

they are linked together in a wide tree structure

johanatan22:02:11

ah, ok. so there's some striping going on?

favila22:02:21

everytime the transactor reindexes, it updates this tree, some blocks become garbage

favila22:02:23

no, not really

favila22:02:39

^^ read this

johanatan22:02:19

ah ok. so do i need to explicitly invoke gcStorage periodically in order to free space on the dynamo or will it happen automatically at some point under just normal operation?

favila22:02:29

you need to invoke it

johanatan22:02:56

does it need to be during a time when the transactor is not busy?

johanatan22:02:26

i assume not based on: "Garbage collection does not lock, block, nor even read any of the segments representing live data."

favila22:02:11

it adds operation load on the storage

favila22:02:24

(you are deleting blocks after all)

johanatan22:02:28

ok, in this case the storage is dynamo so that is amazon's problem ?

johanatan22:02:38

i.e., not a problem 🙂

favila22:02:10

doesn't dynamo make you set capacity limits?

favila22:02:17

operational capacity not storage

johanatan22:02:21

i have it set to on demand

johanatan22:02:24

rather than provisioned

johanatan22:02:30

because my load is tiny

favila22:02:31

ok, then no, it will scale up whenever

favila22:02:42

then scale down I presume

favila22:02:02

i'm trying to find the "operational guide" to datomic on prem

favila22:02:11

I thought it covered this maintenance stuff you need to do

johanatan22:02:08

it probably does. i have a rather scattered method of consuming docs

favila22:02:19

no I can't find it

favila22:02:23

it may not exist

favila22:02:18

big oversight imo

favila22:02:34

anyway, you should run gcstorage after a big import right after indexing

favila22:02:28

and you should run it regularly (weekly or daily) with an "ago time" that is generous enough to encompass any peer's last d/db call

favila22:02:52

I think the baseline recommendation is run it weekly for a time one week ago

favila22:02:11

if you have lots of writes, run it more often to recover storage faster

favila22:02:29

if you have long-running jobs, set the "ago" time to something longer

favila22:02:54

sorry, I mean not encompass any peer's last d/db call

favila22:02:26

e.g. if you have a peer still using a db from a d/db call it made a week ago, your gcstorage ago time should be more than one week

favila22:02:40

or you may be collecting blocks that that peer still has references to

pvillegas1222:02:41

I am getting a dependency conflict on

{:deps
  {com.cognitect/transit-java #:mvn{:version "0.8.311"},
which conflicts with other libraries

pvillegas1222:02:03

is there a way to tell datomic cloud to use a more up to date library of transit?

johanatan05:02:11

so... after running gcStorage, my dynamo db is still: 189.30MB yet the backup is 50MB. is this normal?