Clojurians Log v2

Here's a thing that I don't fully understand: if delete-database is a no-op for memory-databases, and in fact I can still query an in-memory db after deleting it, when is the data garbage collected?

There’s a map that connects URLs to connections. So if you get rid of your last reference to a connection, you can get it back by asking for it again by URL. Calling delete-database will remove this entry, so when your last reference to the connection is gone then it can be GC’ed

(Connections hold databases as per the diagram at https://github.com/threatgrid/asami/wiki/Dev:-1.-Code-Layout#asamimemory )

but what happens if I'm developing the sofware in shadow-cljs for example? I mean, I tried commenting out all the references to a db (I already deleted it from the connection list) but I still get the values back when I query

What are you querying?

something like:

(defonce workspace (db/connect "asami:"))

(comment
  "Cancel the current database-try"
  (db/delete-database "asami:"))

@(db/transact workspace {:tx-data [{:db/id -1 :name :do}]})

(db/q '[:find ?a ?b ?c :where [?a ?b ?c]] (db/db workspace))

workspace is a reference

You didn't let it go, so it's not eligible to be garbage collected

but, when I comment out workspace and then do > workspace at the repl, I get undefined

then I uncomment the things, and at the query I get a bunch of stuff

So, you delete then reconnect to the same URL and there's data? That's unexpected. It should be removed from the connection map. Can you show this with some code?

something like:

(comment
  "the connections are emtpy"
  @db/connections
  ;; => {}
  "but still, there's data"
  (db/q '[:find ?a ?b ?c :where [?a ?b ?c]] (db/db workspace))
;; => ([:tg/node-25 :name :do]
;;     [:tg/node-25 :db/ident :tg/node-25]
;;     [:tg/node-25 :tg/entity true]
;;     [:tg/node-19 :name :do]
;;     [:tg/node-19 :db/ident :tg/node-19]
;;     [:tg/node-19 :tg/entity true]
;;     [:tg/node-23 :name :do]
;;     [:tg/node-23 :db/ident :tg/node-23]
;;     [:tg/node-23 :tg/entity true]
;;     [:tg/node-14 :name :do]
;;     [:tg/node-14 :db/ident :tg/node-14]
;;     [:tg/node-14 :tg/entity true]
;;     [:tg/node-27 :name :do]
;;     [:tg/node-27 :db/ident :tg/node-27]
;;     [:tg/node-27 :tg/entity true]
;;     [:tg/node-16 :name :do]
;;     [:tg/node-16 :db/ident :tg/node-16]
;;     [:tg/node-16 :tg/entity true]
;;     [:tg/node-17 :name :do]
;;     [:tg/node-17 :db/ident :tg/node-17]
;;     [:tg/node-17 :tg/entity true]
;;     [:tg/node-20 :name :do]
;;     [:tg/node-20 :db/ident :tg/node-20]
;;     [:tg/node-20 :tg/entity true]
;;     [:tg/node-22 :name :do]
;;     [:tg/node-22 :db/ident :tg/node-22]
;;     [:tg/node-22 :tg/entity true]
;;     [:tg/node-12 :name :do]
;;     [:tg/node-12 :db/ident :tg/node-12]
;;     [:tg/node-12 :tg/entity true]
;;     [:tg/node-18 :name :do]
;;     [:tg/node-18 :db/ident :tg/node-18]
;;     [:tg/node-18 :tg/entity true])
  )

That shows you querying a connection that you got before removing it from the map

if I do:

(db/q '[:find ?a ?b ?c :where [?a ?b ?c]] (db/db (db/connect "asami:")))

I could reinitialize the connection when you delete, but you called delete, so initializing it would seem to be the wrong thing?

I guess my surprise comes from the fact that after I make sure that workspace is not defined anymore, when it's redefined it still shows up with the old data

I'm thinking about just avoiding references and using always (db/db (db/connect "url")). Would that strategy have any countereffect?

It's like you're asking

(let [a (get a-map :key)
      b-map (dissoc a-map :key)]
  (println a))

That strategy would be a tiny bit slower

So you want to use delete-database as a shorthand to drop all the data and clear it out?

yes, that's what delete-database would imply for me; but maybe I just need a quick way of clearing out the db - I wasn't able to find that in the code

it's also a thing about hanging resources; I would have expected delete-database to free the resources used. But again, it's not really necessary that that function does it, as long as we have some convenient way to do it

It does free the resources. But since most of that is just memory structures and you were hanging onto references, then the structures are still where your holding them

Is more apparent with files, since files actually get removed

I expect to do a release today. It's a small change to update a connection to an empty database when it gets deleted, so I could do that

oh I see than, it's probably not an asami problem per se, but the way I'm using references when using shadow-cljs at the top level; yes that would fix my problem I think

Give me time. Right now I’m lying in bed on a Sunday morning with a phone. 😴

sure, there's absolutely no rush 🙌

Just putting this here for later. In Clojurescript, this returns no watchers:

(def workspace-str "asami:")
(def workspace (db/connect workspace-str))
(add-watch (:state workspace) :simple-print
           (fn [a b c d] (println "On workspace: " [a b c d])))
(.-watchers (:state workspace))

Asami 2.2.0 is now out. There are 3 main changes: • transact is now synchronous on the JVM, and will not return until the end of the transaction, or until timing out. The timeout can be set with a system property of asami.txTimeoutMsec, or datomic.txTimeoutMsec. Long timeouts could be needed for multi-gigabyte imports. transact-async returns a future that will be completed when the transaction is done. • Datoms returned in transaction results are now reflective of database changes, and not just what was requested. So if 20 statements are inserted, and 5 of them already existed, only 15 datoms will be returned. If 2 deletions are requested, and only 1 of those statements exists, then only one delete datom will be returned. • The delete-database function will now clear and reset any existing connections for that URL. This allows the connection to be reused. If data is transacted into the connection, then it will be picked up again by the registry

Strangeloop is behind me, so I have time to myself again.

Unfortunately, it looks like I can’t spend work hours on Asami anymore, but that’s OK. I did a lot of Asami outside of work hours anyway 😊