datomic 2022-01-11 | Slack Archive

cl_j12:01:00

Hi everyone, does datomic have something like SQL delete from table where condition , i.e., to delete all entities by some condition?

jacekschae12:01:41

In Datomic Cloud there is https://docs.datomic.com/cloud/transactions/transaction-functions.html#db-retractentity.. This does not delete it makes it "hidden". Datomic On-Prem supports https://docs.datomic.com/on-prem/reference/excision.html, which is not supported with Datomic Cloud.

cl_j13:01:56

Does this mean to remove data for privacy reasons, the only option is to use excision?

magnars13:01:49

Yes, excision is the tool to use for privacy concerns.

👍 1

magnars13:01:19

There is retractEntity in Datomic Peer also, just to make that clear. 🙂

👍 1

cl_j13:01:51

@U8A5NMMGD what's the options to delete data if we use datomic cloud?

jacekschae13:01:55

AFIK there is no option to delete from Datomic Cloud. Workarounds include: storing personal data in different db or using encryption key and then removing the key at appropriate time so that you can't access the information later. Both of them are not ideal and come with their own set of challenges. @U1QJACBUM would there be anything to add? PS. Is there a place where we can vote for features for Datomic Cloud? I feel like excision would be pretty high on that list?

jaret14:01:07

We monitor customer feature feedback and "votes" in three ways (http://forums.datomic.com , http://ask.datomic.com, and support cases). When an issue/discussion in one of these places is around a feature (like excision) we cross link. We add the context to our internal stories in shortcut for context when looking at features for development. We are aware that excision in Cloud is a feature users desire. regarding this thread I think it is important to state that we want to understand in what scenario you (@U53B6QVDX) would like to "delete" data from Datomic cloud. The start of this thread simply asks about SQL like conditional deletes. Perhaps there is another driving reason beyond privacy (GDPR) that is causing you to consider the option of deleting and I would like to understand that. But @U8A5NMMGD is correct. There is no "delete" or excision in Datomic Cloud.

cl_j14:01:59

Thanks @U1QJACBUM! Yes, we are deleting data for privacy reasons. And I am currently using :db/retractEntity, deleting more than 100k entities can be very time consuming, that's why i ask whether there is something easier and faster

jaret14:01:29

Hi @U53B6QVDX a few important clarifications. Retractions is not deletion. The data is still there. With retract you have created an atomic fact in the database dissociating an entity from a particular value of an attribute. The fact of that retraction remains along with the history of it's previous values.

jaret14:01:10

I am happy to look at performance questions around retractEntity. We can help you speed that up or consider another approach so that it isn't as time consuming.

jaret14:01:32

If that's something you are interested in shoot me a line at <mailto:[email protected]|[email protected]> with a gist of what you are doing and I can work from there to determine what performance aspects are and what advice might be relevant.

👍 1

Benjamin13:01:35

do you know what SecureString means for aws parameter store? And what is a good way to put/get secure string paramaters?

✅ 1

jacekschae14:01:40

SecureString means it's encrypted and by using aws apis for getting the string out you will get it decrypted at read time. If you are using Datomic Cloud you can use https://docs.datomic.com/cloud/ions/ions-reference.html#get-params to get it from aws ssm; see https://github.com/jacekschae/learn-datomic-course-files/blob/f2378c84bade5cb64018f72aa9179a8c8bb25df4/increments/complete/src/main/cheffy/ion.clj#L11.

Benjamin14:01:20

I see

Benjamin14:01:38

would you use params over secrets manager?

jacekschae14:01:51

If i'm using Datomic Cloud I would use SSM -- AWS Systems Manager as it comes with ion helpers. Don't have a good overview of why would you pick Secrets Manager over Systems Manager.

👍 1

Michael W16:01:31

Secrets Manager costs per encrypted string and allows you to enforce lifecycle and rotation. Systems Manager gives you an encrypted string, and you have to handle lifecycle and rotation yourself. I prefer SSM myself since it's cheaper, and a simple lambda function can do lifecyle and rotation if needed.

👍 1

Benjamin16:01:30

[datomic.ion.cast :as cast]
     [datomic.ion :as ion]

(cast/initialize-redirect :stdout)
(cast/dev {:msg "SlackApi"})

=>

1. Unhandled java.lang.IllegalArgumentException
   No implementation of method: :-dev of protocol:
   #'datomic.ion.cast.impl/Cast found for class: nil

          core_deftype.clj:  583  clojure.core/-cache-protocol-fn
          core_deftype.clj:  575  clojure.core/-cache-protocol-fn
                  impl.clj:   14  datomic.ion.cast.impl/fn/G
                  cast.clj:   74  datomic.ion.cast/dev
                  cast.clj:   65  datomic.ion.cast/dev

what do I do wrong with local cast ?

Ivan Fedorov17:01:27

Heyy, nice to meet you all again! How would one test an ions lambda before production deployment? Any pre-made handles to simulate the message wrapping by AWS?

stuartrexking22:01:15

Handlers are just functions.

stuartrexking22:01:34

Call them with whatever body they will get called with by Lambda.

2022-01-11

Channels