Clojurians Log v2

New times: lein test netgrok.core-test Importing o1 into asami Importing to asami "Elapsed time: 19520.553168 msecs" Imported 0 Importing o2 into asami Importing to asami "Elapsed time: 208748.131329 msecs" Imported 0

So yah, I can confirm your estimate on perf win with Alpha6

I’m guessing that you’re saying “imported 0” to mean a count of tempids?

yah, just ignored that this time since I havent' updated my tests yet -- still drinking first cup of coffee

Have a look at the count on tx-data. That’s the number of statements inserted

If you want the number of entities inserted… do a count on your input 😊

The tempids, is so you can provide a negative number for :db/id on an entity and it will generate an ID for you and tell you what your negative number got mapped to (like Datomic)

I have some utility functions for exploring the shape of the data and the schema

next step is to do some query clause generators

for functional composition of where clauses...

just doing export-data a bunch helped me grok what is happening

export-data gives you a view of everything, but if you insert individual things (or just small numbers of entities) then have a look at the contents of tx-data in the results of the transaction. That shows you the triples that were generated and inserted.

I’m curious how many triples you got from your data that took 3m28s to load. (I have to work to improve this)

data coming up...

ein test netgrok.core-test Importing o1 into asami Importing to asami "Elapsed time: 19489.224782 msecs" Imported 31881 statements Importing o2 into asami Importing to asami "Elapsed time: 209601.057757 msecs" Imported 296271 statements

Thanks for that

heading out for brunch

Yup? What’s happened?

Locked up importing a few thousand objects

I'll break up the txn

and we'll see what's happening. I must first eliminate my own stupidity...

Actually, breaking up the transaction is a bad thing to do. How big is the file that you’re importing?

(bad, because you end up expanding the indexes significantly)

Also, if you’ve made a mistake, I’d like to know about that too. I should document gotchas, and mitigate some of the more obvious ones

yah, so it's not locked up, but just slow. Mind you I'm throwing a lot of large complex objects at it

I'll get data to you shortly

“large complex” is going to be an issue. Zuko (the module that breaks it up into triples) is now faster than it used to be, but there’s still a lot of work for it to do

Yah, I'm thinking it's a chance to intrument the whole thing with metrics data

Importing o1 into asami Importing to asami "Elapsed time: 53402.866849 msecs" Imported 4503 Importing o2 into asami Importing to asami "Elapsed time: 644524.77233 msecs" Imported 41484

FAIL in (load-test) (core_test.clj:21) Test loading and parsing expected: (= (count o1) (count (:tempids tx1))) actual: (not (= 271 4503)) lein test :only netgrok.core-test/load-test FAIL in (load-test) (core_test.clj:22) Test loading and parsing expected: (= (count o2) (count (:tempids tx2))) actual: (not (= 2570 41484))

So the failures are me expecting the entity count to be the input object count. The difference tells you just how complex some of the objects are, with many nested entities...

It creates lots of temporary IDs, but unless you ask, I would generally think they’d match the provided objects. 🤔

I’m assuming that some or all of this data can be shared?

not sure. it's packet dumps form my home network

I will find some representative data

The :tempids in the tx would include the nested objects right?

I didn’t think so (unless you asked it to). So unless I’ve forgotten something it’s a problem

It creates lots of IDs, but that map is supposed to just be for the top level entities, and things you’ve provided your own temporary IDs to

yah, so that seems wrong then

I provided no temp IDs for anything

So, I ran the same thing with in mem db

lein test netgrok.core-test Preparing o1 "Elapsed time: 0.005525 msecs" Preparing o2 "Elapsed time: 8.82E-4 msecs" Importing o1 into asami Importing to asami "Elapsed time: 262.289293 msecs" Imported 4503 Importing o2 into asami Importing to asami "Elapsed time: 2622.369329 msecs" Imported 41484

Is zuko involved in that too?

yes

It’s a library that pulls entities apart into triples

ok, so it's not in zuko then eh

BTW, the large number of entities in tempids was expected, but I’m revisiting it, and I think they should not be included.

So I’m going to update Zuko to remove them