@csm do you have a small repro case? Also, what version of datomic?

I'm doing some profiling, and I'm seeing some threads: query-1, query-2, query-3, query-4. They're holding the bulk of our allocations right now. Wanted to know if they're from datomic & what they do, should I be concerned about their usage.

They aren't actually growing, so it may be caches or something. But thought I'd ask.

@dominicm I believe they are the threads that actually execute queries

I always see them light up (and no other threads) during a long-running query

I see… Particularly for d/q or for all queries e.g. d/datoms? @favila

there are no other queries

or am I missing something?

Sorry, do they get used for d/datoms do you know?

I believe not

d/datoms has hardly any cpu impact

so I can't be sure

No, I thought it shouldn't.

Bit surprised at what would be querying right now is all.

Anything big should have been converted to d/datoms already

I think you get a # equal to number of cores

and a single query will run on many of those threads at once

Hmm, I'm only seeing a single query thread light up allocations at once. Also, all 4 threads (before my job started) were sat at ~25% usage of the memory.

Now they're at 20% (big job running)

maybe there's no parallelism to exploit?

The big job isn't supposed to allocate much (lazy sequence), so I suspect we're doing something wrong there.

Ah, only when parallelism can happen. That makes sense. 🙂

I kinda thought 99% of datomic was parallelisable.

only large intermediate sets benefit

if a clause is very selective, there isn't anything to parallelize over

Makes sense. I think we're doing enrichment of data from other entities. Inferring certain properties, I'm not overly familiar with the intimate details of the queries we use running over these datoms though

watching google next keynote live stream – spanner looks really cool. i know it's been discussed before, but my next thought was immediately "i wonder if this would work as a datomic backend"

@marshall I don’t have a repro case yet (this was provoked by two µ-services, one writing, one reading) though I would like to put one together. This is with 0.9.5561.

I actually tried (dissoc (d/db conn) :t :next-t), and at least couldn’t reproduce the issue

Has anyone integrated Datomic as part of a project that includes a separate GIS system? I'm going to building a PostGIS backed system and I want to see if I can use Datomic as well for as much of the non-spatial data as possible, but I'm not sure exactly where I should be putting those boundaries and if there are any real-life issues that I'll run into that I haven't considered yet.

@shaun-mahood I don't know about GIS specifically, but I do know people have used Datomic alongside other storage/indexing systems.

The general rule is to store things in the system that's optimized for storing that kind of data: small, relational, transactional values in Datomic; geo-spatial data in a geo-spatial system; binary blobs in a blob store; etc.

@stuartsierra: Thanks, that's kind of what I'm leaning towards - good to know there is some success out there for other areas and it's not just a plain old bad idea. In unrelated news, the Lambda Island video on component finally got me over the hump to start using it 🙂