Fork me on GitHub
#datomic
<
2019-08-06
>
eoliphant13:08:56

Hi, running into a weird issue on datomic cloud. Where comparisons on float fields are failing to match, in the sample below from one of my devs, test-var is something like 123456.0. Even grabbing it directly from the db and handing it right back in the next query, isn’t matching.

(def test-var (first (ffirst (d/q '[:find (take 10 ?nga)
                              :where [_ :nga-can/nga-id ?nga]]
                         db))))
                         
                         
        (d/q '[:find ?e
       :where [?e :nga-can/nga-id test-var]]
  db)     

souenzzo13:08:40

test-var is quoted in this code.

eoliphant13:08:26

ah hell copied and pasted that from him, i;m on my phone lol. missed that thanks will dbl check

👍 4
Alex Miller (Clojure team)13:08:22

comparisons on floats are generally problematic due to imprecision (this is a generic issue in any language using IEEE 754)

4
✔️ 4
Alex Miller (Clojure team)13:08:59

that may not be your problem, but just something to be aware of

eoliphant13:08:40

yeah I thought that that might be the case. But, even using the ‘same’ value seems to be problematic.

matthavener13:08:56

@eoliphant could it be NaN? that will fail even self equality tests

eoliphant13:08:06

looks like it’s good ol IEEE floats at the end of the day

m0smith17:08:07

Hi all, how do I retract all entities match some filter. For example, the entitiy has a :date attr and I want to retract all entities where :date is in a given date range

matthavener17:08:33

(d/transact conn (map (fn [e] [:db.fn/retractEntity (:db/id e)]) (filter #(in-range? (:date %)) entities))) ?

matthavener17:08:04

there’s not really any notion of DELETE FROM table WHERE date >= ... if that’s what you’re looking for

John Miller18:08:42

Could somebody sanity check a model I came up with? We need to use a full-text search to find entities, and then return a bunch of attributes that are stored in Datomic. So I have created a query function as an ion that calls out to cloudsearch and returns a set of ids and scores. I then look up the id normally. The query looks something like this:

(d/q '[:find (pull ?e [*]) ?score
       :in $ ?query
       :where [(ions.cloudsearch/find-by-query ?query) [[?id ?score] ...]]
         [?e :generic/id ?id]]
     db
     query)
So two quesions - Is this a reasonable model? If so, are there any resources we might need to worry about if Cloudsearch take a long time to respond (e.g. JVM threads? Memory? Certainly not CPU since it would be io bound). We’re expecting potentially 10's of requests per second, and CS typically responds in 10's of ms but sometimes stalls and takes multiple seconds. When that happens, 40-50 requests might build up before CS responds. We saw a rash of :busy anomalies once since we started trying to make this work. It may be unrelated, but I wanted to see if anybody knew of any potential pitfalls before we go too far down this path.

m0smith18:08:10

@matthavener Thanks. I wish google were better at finding this information

m0smith18:08:53

I would do a separate query to get the "entities"?

csm18:08:22

yes, you would use a separate query in that case. Though remember that those entities aren’t removed from the database, the history is intact; what you’re trying might be better done in your queries with a rule that filters :date after the date you’re interested in

csm18:08:47

if you’re attempting to get rid of old data to save space, datomic might not be the right fit for your problem

m0smith19:08:41

For now I think I do want to retract them. we want the history so we can always trace the data. thanks again

Joe Lane19:08:01

@jmiller If your cloudsearch request blocks I would say its a bad idea to do that in the query. Instead, why not issue the cloudsearch query, then take the results and pass them into the datomic query? We have a homegrown datomic cloud lucene integration that does something similar to this. Granted, the lucene indexes are relatively small (3gb) so its low overhead.

John Miller20:08:25

Thanks for the response, Joe. It generally only stalls for a couple seconds so my hope is that Datomic handles it fine. I’ve successfully done similar things in UDFs in Mysql, but that isn’t built on Java so I am concerned that the gotchas are different.