Fork me on GitHub
#datomic
<
2020-04-14
>
Janne Sauvala06:04:55

I got started with Datomic Cloud (solo) during the weekend. Is there a recommended way how to setup a development environment? I saw a few others asking about this on the forums and their solution is to use datomic-client-memdb (https://github.com/ComputeSoftware/datomic-client-memdb) or separate dev-cloud environment.

em07:04:45

@UJZ6S8YR2 Also checkout https://github.com/markbastian/replion ! Will get you a lot closer to the Cloud/Ions based environment

thanks 4
Joe Lane13:04:52

@UJZ6S8YR2 I made a development solo system which I used for over 2 years successfully without a local offline system. I wouldn't complicate things with the datomic-client-memdb or any other replacement for just having an internet connection until you run into an actual pain point. If cost is the concern you can use the datomic CLI Tools to turn on your bastion+solo node when you begin doing development for the day and shut it off at the end of the day (or set an Autoscaling group to do it for you on a schedule).

Joe Lane13:04:53

The last thing you want to do when just getting started is have divergent library behavior between local development and production.

kenny15:04:22

FWIW, I wrote that library and agree with @U0CJ19XAM. It's very unfortunate there is no local Datomic Cloud solution. If you can get away with using a solo deployment for your needs, you most certainly should. You will need to build some tooling around how to use a single solo deployment with multiple developers but that isn't too difficult (we prefix DB names). We have the ability to easily switch to that library for those that have a poor/no internet connection. We run integration tests against Datomic Cloud.

👍 4
Janne Sauvala16:04:07

Thanks @U0CJ19XAM and @U083D6HK9! I didn’t know you could turn on/off the bastion and node like that with CLI Tools, that sounds quite handy approach. I was also concerned about the integration with CI-pipeline but it should be okay just to use the Cloud like you are doing, Kenny.

kenny16:04:37

It took some work to get it to work on CI 😬

Joe Lane16:04:52

When you set up CI, do it in the us-east-1 region.

Joe Lane16:04:09

Otherwise you will have headaches...

kenny16:04:13

> When you set up CI, do it in the us-east-1 region. What does this mean?

kenny16:04:41

Datomic Cloud region or CI region? We use CircleCI and I don't think they have a region selector.

Joe Lane16:04:21

Sorry, I assumed codebuild+codepipeline. If you use those, set them up in the us-east-1 region.

4
Joe Lane16:04:03

Its the classic "cross-region s3 bucket access from within a vpc" issue :)

johnj18:04:02

you can also avoid creating the bastion and connect directly to the node, set a firewall rule to allow and protect access

kenny18:04:07

I’d suggest against that due to the security implications.

kenny18:04:27

You have created a hole in your VPC to the outside world 🙂 unless you have 100% certainty (and a security policy that allows such a thing) that your hole will always hold true (e.g., reserved static IP), you have an issue.

4
johnj18:04:58

yep, for solo, restricting access to an IP in the security group

johnj18:04:54

how will that won't always hold true?

kenny18:04:44

You just be 100% sure that the IP is owned by you. Typically an IP provided to you by an ISP is not static. Most hosted CI solutions (e.g., CircleCI) do not make any guarantees about the IP addresses of their worker nodes.

kenny18:04:45

An example of an IP owned by you would be an EIP.

johnj18:04:26

Yeah, I understand, but for a solo dev on solo playing on the weekends it might be enough

johnj18:04:10

anyway, I just saw that now there are some CLI tools to simplify access

johnj18:04:47

so there's less reason to avoid bastion

Joe Lane19:04:16

Never do that.

4
naomarik15:04:05

Hello! I’ve made a mistake and did a restore instead of backup on my database. App is still running though and all cached data is there. Is there a way to dump those datoms into a file I can restore with via the repl on the running app?

onetom15:04:53

u would need to have a valid DB reference to that state of the DB which contained the datoms still before the restore, then u can do something like (seq (d/datoms (d/history db-value-before-restore) :eavt)) , but not sure if it would work

naomarik15:04:38

Then I could import them with which command?

naomarik15:04:56

nearly done writing an import script anyway — but want to try this first

onetom15:04:07

you have to program that yourself

naomarik15:04:18

gist online of something like this about 200 lines.. looks sketchy

onetom15:04:22

u just get back a sequence of Datom objects, which you can either thread through (map (partial into {})) or (map (partial into [])) , then if u want to transact them back to the DB, you would need to massage them into [:db/add ....] or [:db/retract ...] form

onetom15:04:32

i haven't seen any off-the-shelf solution for this kind of situation, but technically you want those datoms transacted back into your DB what you see thru the history DB. you might even want to use d/since to just get the tail of the history db after the time of restoration.

onetom15:04:52

it's a bit unrelated, but i often work with the aggregate differences of a database between 2 points in time. for that i use this function:

(defn db-diff [db-before db-after]
  (-> db-after
      (d/since (d/basis-t db-before))
      (d/history)))
this way i don't have to care what kind of transactions have led to the db-after state; i just see all the added and retracted datoms.

onetom16:04:12

Let's say i have a database like this:

[{:thing/name "thing0"}
   {:thing/name "thing1" :thing/container [:box/id "box1"]}
   {:thing/name "thing2" :thing/container [:box/id "box1"]}
   {:box/id "box1"}
   {:box/id "box2"}]
where :thing/container is a ref card/one and :box/id is uniq/identity in plain english: 1. thing0 is not in any box 2. thing1 and thing2 are both in box1 3. box2 is empty How can I express in datalog that im looking for empty boxes? I would expect [:box/id "box2"] as the result.

mgrbyte16:04:08

if a box will always have a name when not empty, then I believe you could do something along the lines of:

[:find ?box
 :where
   (missing? $ ?thing :thing/name)
   [?thing :thing/container? ?box]
   [?box :box/id ?box]]
Ignore my response; i see you need the inverse of this.

onetom16:04:07

thx for trying!

onetom16:04:39

I can of course just pull all box entities and all container values and do a set difference, but that feels way too low-level, eg:

(set/difference (->> (d/datoms db :avet :box/id) (map :e) set)
                (->> (d/datoms db :avet :thing/container) (map :v) set))

onetom16:04:42

I know about the missing? predicate, but i don't see how could that help in this case

onetom16:04:40

Here is a full example

onetom16:04:31

this actually seems to work:

(-> '[:find [?box ...] :where
          [?box :box/id]
          (not [_ :thing/container ?box])]
        (d/q db))

onetom16:04:40

my problem earlier was that i haven't ignored the thing id:

(-> '[:find [?box ...] :where
          [?box :box/id]
          (not [?thing :thing/container ?box])]
        (d/q db))
so i was getting this error:
Execution error (Exceptions$IllegalArgumentExceptionInfo) at datomic.error/arg (error.clj:79).
:db.error/insufficient-binding [?thing] not bound in not clause: (not-join [?box ?thing] [?thing :thing/container ?box])

onetom16:04:34

but then i tried this too:

(-> '[:find [?box ...] :where
          [?box :box/id]
          (not-join [?box]
                    [?thing :thing/container ?box])]
        (d/q db))
and it didn't work at that time, but seems to be correct now. maybe my actual use-case is actually different from this example; i will investigate it tomorrow.

Ben Hammond17:04:52

data storage question; I represent state as a 218-bit (ish) BigInt, like

34673886561508028522262350969951764751176091809504658889186481737N
for the purposes of HTTP transmission I am encoding this into a Base62 type string; thus
6qzEQWjdQUt25HqXqMiUSY8Yu63ru31LbeJoc
I need to store this value in datomic; I had assumed that it would be more efficient to store as :db.type/bigint (rather than as a string) but I have no evidence for this, other than intuition; can anyone confirm or deny? I mostly want to use this value as a lookup identity. would there be much performance difference between storing as a bigint, or storing as base62 encoded string?

Ben Hammond18:04:59

perhaps I should look at the fressian encodings of each and see which comes out smaller

Joe Lane19:04:20

@ben.hammond are you concerned about encoding size or execution time converting to and from values? How often do you need to look up these values and do a base62 conversion? Is that efficient? Do you ever need to query over these values numerically in datalog? If so, you would probably want them represented as numbers so you can use primitive range predicates. etc, etc,. If you want leverage over these numbers in the database I'd say store them as numbers.

Ben Hammond19:04:35

efficiency really

Ben Hammond19:04:58

the numbers are opaque so theres no arithmetical logic

Joe Lane20:04:55

Efficiency of what though?

Ben Hammond19:04:21

I guess I was thinking, that in terms of entropy, the raw number must be more efficient than a UTF-8 string where you are only using 62 chars of a possible 254 single-byte chars

Ben Hammond19:04:51

but I dont' suppose it is particularly significant either way

Sam DeSota19:04:51

Hi, what would be the best way to generate a a random datomic cloud id before transacting an entity? I attempted to generate a random Long but it seems to be failing intermittently. Squuid does not appear to exist in cloud client.

Sam DeSota20:04:35

Actually I'd like to generate an actual entity id, not a tempid. I'd like to save a file to S3 including the entity id before I commit the entity to datomic, so the file is available for other systems.

Sam DeSota20:04:28

I could use a generate my own uuid instead, and use :db/unique, but was wondering if there as a way to do this with :db/id

Joe Lane20:04:47

Make a unique attribute with a UUID

donyorm21:04:56

I'm getting this error when trying to call this lambda:

(def ion-handler-lambda-proxy
  (apigw/ionize
   (fn [request]
     {:status 200
      :headers {"Content-Type" "application/edn"}
      :body "ok"}))

donyorm21:04:45

But I'm getting this error, and I have no idea why No implementation of method: :->bbuf of protocol: #'datomic.ion.lambda.api-gateway/ToBbuf found for class: nil . The troubleshooting docs says to ensure your lambda function is returning a string, but that produces the same error.

donyorm21:04:04

I seem to be getting this error even in the ion-starter tutorial