Per @stuarthalloway’s advice, Ive prepended a hash of the html content we store in the db to assist indexing. In order to clear up the non-hashed older versions of each page. I’ve done a (d/transact (conn) [{:db/excise :pages/content :db.excise/before (instant (time/minus (now) (time/days 1)))}])which I think should clear anything saved before yesterday. The command does start a lot of work in datomic, but after about 2 minutes it hits a Critical failure saying the index retry limit is reached. No amount of restarting mysql or datomic, or the app for the matter can break this cycle. The system boots, then craters after 2 minutes.

And while we’re on excision. The tx above also kills the latest value if its not saved before yesterday - Is there a way to kill all but the last tx?

@laujensen Excision shouldn’t be used to remove large amounts of data. it is a very expensive operation and can result in the situation you found where the indexing job is too large to complete

So how do I go about my history @marshall ? We’ve talking 5000+ pages that need some help

see: https://docs.datomic.com/on-prem/excision.html#performance

you can either: 1) run multiple smaller excision jobs, waiting for each to complete before starting the next 2) create a new database and “decant” the data from the current DB into it, filtering out the things you want to remove

1) How do I query the status of the indexing job?

you will see a metrics report in the logs (and cloudwatch metrics) that includes :CreateEntireIndexMsec when the job has completed

thanks

I’m interested in testing a couple small datomic queries in unit tests. I realize I could spin up an in-memory db but I think there’s a way to use a plain Clojure data structure as the db, is that true? What shape would it need to have so that something like the q and transact functions would work with it (treat it as a real database)?

@marshall - Ive just tried running a backup-db on one of our shards to import it locally using restore-db, but on import I see that several transactions arent carried over. They are visible on the live system, but not locally. And they’re about 1 day old. What causes this?

does anyone have any experience running aws lambda with clojure connecting to datomic-cloud? any gotchas (like uberjar size, startup time, ...) that I can avoid running into?

when i'm following the Getting Started guide, i'm seeing the following error when trying to create a database

(d/create-database client {:db-name "movies"})
ExceptionInfo Forbidden to read keyfile at s3://...

all steps before worked properly

that’s an issue with your aws creds

yes, but how can I specify the AWS credentials to the client?

i'm using a aws profile to authenticate

the normal ways - ~/.aws/credentials, AWS_ACCESS…

AWS_PROFILE

hmm that's still not working. is the client using the order of the aws java sdk for credentials?

the client uses the default credentials provide

provider

which has an implicit order, documented by AWS

https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html

ok, it seems like I'm running into this issue https://github.com/aws/aws-sdk-java/issues/803

delegation of credentials to a root profile doesn't work the same way in the java sdk and the cli 🙂

ah. good find

@marshall: I’ve tried partitioning the excisions into chunks of 100 entities. The indexing service has been running for 30+ minutes now with half-load on a quadcore system on just the first chunk. Is that to be expected? If so, I guess there’s no way around putting a heavy load on the live system until all 5000 pages have had their history removed?

Hi, just to check is datomic rest the only way to connect to datomic from another language?

@folcon At present, yes. We provide Clojure and Java versions of the Peer library and a Clojure Client library. We intend to publicize the client wire protocol in the future to allow other language clients (see: https://www.datomic.com/cloud-faq.html#_will_you_publish_the_client_protocol_so_i_can_write_my_own_datomic_client)

@marshall Thanks, if you wouldn’t mind me asking, I have two questions. 1) Is there anyway to secure datomic rest other than sitting a nginx with basic auth in front of it? I’ve successfully managed to deploy it and it’s working and now need to work out at least a basic level of security. 2) How would I go about writing a subquery then? I’m trying to ensure that all queries to the system respect some level of auth, so I want to ensure that each user only looks at a database which contains their datoms. My understanding is that this is possible by first querying what a user can see, and then passing on the user query to the resulting database. I’m just unsure how to specify that to datomic rest.

@folcon https://docs.datomic.com/on-prem/rest.html

in case you miss the no more development notice

I am aware it’s been deprecated for some time, I’m just wondering if there exists anything for this? I’m basically bumbling my way with these unfortunately :)… And unfortunately the client api’s don’t seem targeted to what I’m doing at the moment. I can’t access them from clojurescript as far as I can see, and running yet another server just to act as a clojure proxy to talk to datomic seems a little extreme?

exposing datomic directly to the internet seems more extreme to me, even if behind nginx

but nginx has a lot features and is highly configurable, maybe you can use something like openresty

@folcon “running another server just to act as a clojure proxy” is exactly what the REST service is. It is a Datomic Peer that consumes HTTP calls and executes them against Datomic

If you want native access from your language of interest and are already on the Peer model, I’d suggest writing a Peer-based system that consumes whatever it is your application uses (http/tcp/other) and do it that way

That’s a valid point, however for the moment the rest service does meet most of my needs. Is my understanding of my second question correct or should I be trying something different? I’m having a fair bit of difficulty finding information about what is the best way of querying against a subset of the datoms in the database.

you can use filters (https://docs.datomic.com/on-prem/filters.html) to restrict what a given query can “see”

Thank you, I’ll give that a read 🙂

I might be doing this completely wrong, but trying to reproduce the filtering technique is giving me odd results: q*:

[:find ?e ?ent ?doc ?f_db
 :in $plain ?filterfn
 :where [(datomic.api/filter $plain ?filterfn) ?f_db]
              [(datomic.api/entity ?f_db ?e) ?ent] [$plain ?e :db/doc ?doc]]

[{:db/alias "sql/test"} (fn [_ datom] (< 20 (.e datom)))]

?e	?ent	?doc	?f_db
8	{:db/id 8}	"System-assigned attribute set to true for transactions not fully incorporated into the index"	datomic.db.Db@de24db77

Ok, so from what I can tell, the query language cares what the names of the variables are, and you can’t define a database and reuse it as you’ll have a var that starts with a ? not a $.

java.lang.Exception: processing rule: (q__355 ?e ?ent ?doc ?f_db), message: processing clause: [?f_db ?e :db/doc ?doc], message: :db.error/invalid-data-source Nil or missing data source. Did you forget to pass a database argument?

the entity with EID 8 does pass that filter

@folcon ^

sorry?

the entity ID you found there is ‘8’, which is < 20

oh, am I checking a string?

no

entity id is a long

sorry i misread your filter function; one second

I’m pretty sure (< 20 8 ) is false?

Sorry, it’s been a long day 🙂

ahh. i think i see

you need to get a filtered value of the db as an input to the query

one sec

sure :)…

so the functionality you’re looking for doesnt require a join on 2 dbs

(d/q 
 '[:find ?e ?doc
  :in $
  :where 
  [?e :db/doc ?doc]]
 (d/filter (d/db conn) (fn [_ datom] (< 20 (.e datom))))) 

find all the entities in a filtered db

for perf reasons, you could join against a non-filtered db

(which is what the example shows

i’m not sure if/how you’d do the filter inside the query body and then also bind it to a datasource

i have to run, but i’ll be back a bit later

So wait, do I need to make two queries to the rest api? I’m trying to understand how to translate that query, my two args that I can use are the q* and args. Args needs to be at least [{:db/alias "sql/test"}] from what I understand and as far as I can work out I can’t pass a filtered db as I have no idea how to reference it…

thanks

I’ll be at it for a bit :)…

I suspect you may not be able to pass an arbitrary filtered database to the rest API

That’s frustrating

@marshall regarding the issue above on AWS credentials, we are using IAM role assumption to access different aws accounts for dev, staging, prod. In order to make this work, you need to add a dependency [com.amazonaws/aws-java-sdk-sts "1.11.210"]. Not sure if you want to add this to the datomic client library or mention it in the documentation, but debugging the problem was a bit annoying since d/create-database swallows the original error of the aws sdk

which was

(.getCredentials (com.amazonaws.auth.profile.ProfileCredentialsProvider.))
ClassNotFoundException com.amazonaws.services.securitytoken.internal.STSProfileCredentialsService java.net.URLClassLoader.findClass (URLClassLoader.java:381)

some more info: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/securitytoken/internal/STSProfileCredentialsService.html

it works for me now, but you can maybe help other customers with that info 🙂

is java 9/10 not supported for the datomic cloud client?

Huh, you can introspect the db enough to get the database credentials by querying it.

@lockdown- you seem to be assuming it’s not - any reason why?

I haven’t tried it, but one problem that arises in several places right now is due to the removal of javax.xml.bind from the default classpath. Using --add-modules java.xml.bind on the jvm will add it back in.

@stijn Thanks for the heads up - I will see if I can find a place in the docs where that would fit well

@alexmiller correct, adding java.xml.bind fixes it, was curious if anything greater than java 8 was discouraged by datomic devs.

that was just a guess, would be curious if it’s a datomic lib dep or something else that you’re running into

the datomic client eats the stacktrace I think but per your clj -J-verbose:class advice it might be the aws sdk version but not sure, didn't dig more

probably a good thing for @marshall to know if he doesn’t already

yes indeed. thanks!

i will also look at adding that to docs

@marshall indeed is discouraged? using something greater than java 8

@marshall I have got the filtered db in the query though and I can inspect it:

[:find ?e ?ent ?doc ?f_db ?prs :in $plain ?filterfn :where [(datomic.api/filter $plain ?filterfn) ?f_db] [(keys ?f_db) ?prs] [(datomic.api/entity ?f_db ?e) ?ent] [$plain ?e :db/doc ?doc]]

(:id :memidx :indexing :mid-index :index :history :memlog :basisT :nextT :indexBasisT :indexingNextT :elements :keys :ids :index-root-id :index-rev :asOfT :sinceT :raw :filt)

(fn [_ datom] (< 20 (.v datom)))

I’ve been trying to pass a string query, as the api states that’s possible:

[:find ?e ?ent ?doc ?f_db ?prs :in $plain ?filterfn :where [(datomic.api/filter $plain ?filterfn) ?f_db] [(:filt ?f_db) ?prs] [(datomic.api/q "[:find ?e :where [?e :db/doc _]]" ?f_db) ?ent] [$plain ?e :db/doc ?doc]]

java.lang.Exception: processing rule: (q__1170 ?e ?ent ?doc ?f_db ?prs), message: processing clause: {:argvars (?f_db), :fn #object[datomic.extensions$eval1162$fn__1163 0x1b0f3213 "datomic.extensions$eval1162$fn__1163@1b0f3213"], :clause [(datomic.api/q "[:find ?e :where [?e :db/doc _]]" ?f_db) ?ent], :binds [?ent], :bind-type :scalar, :needs-source false}, message: java.lang.Exception: processing rule: (q__1171 ?e), message: processing clause: [?e :db/doc _], message: clojure.lang.PersistentList cannot be cast to clojure.lang.IFn

@lockdown- no, i meant it was useful for me to know. No reason not to use 9 or 10 if that fix works

@folcon I wonder if you can then use the filtered DB in a nested query

@marshall That’s what I’ve been trying to do here -> https://clojurians.slack.com/archives/C03RZMDSH/p1521753296000093, but it doesn’t seem to be working?

[:find ?e ?ent ?doc ?f_db ?prs ?filtecount
 :in $plain ?filterfn 
 :where [(datomic.api/filter $plain ?filterfn) ?f_db] 
        [(keys ?f_db) ?prs] [(datomic.api/entity ?f_db ?e) ?ent] 
        [$plain ?e :db/doc ?doc]
        [(datomic.api/q '[:find (count ?ents)
                                :where [?ents :db/doc]]
                              ?f_db) [[?filtecount]]]]

try that ^

i’m on a phone call or I’d try

@marshall Funnily enough:

com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to resolve symbol: ' in this context, compiling:(NO_SOURCE_PATH:0:0)

java.lang.Exception: processing rule: (q__1315 ?e ?ent ?doc ?f_db ?prs ?filtecount), message: processing clause: {:argvars (?f_db), :fn #object[datomic.extensions$eval1307$fn__1308 0x1f4747dd "datomic.extensions$eval1307$fn__1308@1f4747dd"], :clause [(datomic.api/q (quote [:find (count ?ents) :where [?ents :db/doc]]) ?f_db) [[?filtecount]]], :binds [?filtecount], :bind-type :rel, :needs-source false}, message: java.lang.Exception: processing rule: (q__1316 ?ents), message: processing clause: [?ents :db/doc], message: clojure.lang.PersistentList cannot be cast to clojure.lang.IFn

[:find ?e ?ent ?doc ?f_db ?prs ?filtecount
:in $plain ?filterfn 
:where [(datomic.api/filter $plain ?filterfn) ?f_db] 
    [(keys ?f_db) ?prs] [(datomic.api/entity ?f_db ?e) ?ent] 
    [$plain ?e :db/doc ?doc]
    [(datomic.api/q "[:find (count ?ents)
                            :where [?ents :db/doc]]"
                          ?f_db) [[?filtecount]]]]

java.lang.Exception: processing rule: (q__1277 ?e ?ent ?doc ?f_db ?prs ?filtecount), message: processing clause: {:argvars (?f_db), :fn #object[datomic.extensions$eval1269$fn__1270 0x6f32c7f9 "datomic.extensions$eval1269$fn__1270@6f32c7f9"], :clause [(datomic.api/q "[:find (count ?ents)\r\n                                :where [?ents :db/doc]]" ?f_db) [[?filtecount]]], :binds [?filtecount], :bind-type :rel, :needs-source false}, message: java.lang.Exception: processing rule: (q__1278 ?ents), message: processing clause: [?ents :db/doc], message: clojure.lang.PersistentList cannot be cast to clojure.lang.IFn

copy paste issue with single quote probably

Not sure what the casting issue is

got it

(d/q 
 '[:find ?filtecount
   :in $ ?filterfn 
   :where [(datomic.api/filter $ ?filterfn) ?f_db] 
          [(datomic.api/q '[:find ?ents
                            :where [?ents :db/doc]]
                          ?f_db) [[?filtecount]]]]
  (d/db conn) (fn [_ datom] (< 20 (.e datom)))) 

bad var names. sorry i’l fix

more interesting with correct name:

(d/q 
 '[:find ?filtecount
   :in $ ?filterfn 
   :where [(datomic.api/filter $ ?filterfn) ?f_db] 
          [(datomic.api/q '[:find (count ?ents)
                            :where [?ents :db/doc]]
                          ?f_db) [[?filtecount]]]]
  (d/db conn) (fn [_ datom] (< 20 (.e datom)))) 

find the count of entities with :db/doc in the filtered db

So the reader can’t deal with the single quote at all, so I’ve been replacing the query with the string version or manually calling the quote function, however there’s a relatively consistent issue of message: clojure.lang.PersistentList cannot be cast to clojure.lang.IFn in both cases.

I’m really not sure what the problem is here.

This is specifically with the rest api?

I'll have to try that tomorrow morning

yep, all of the queries I’m running are through the rest api.