Fork me on GitHub
#datomic
<
2021-02-15
>
em05:02:27

Just my two cents, but if your primary concern is financial in nature with data conflict concerns and an emphasis on immutability, Datomic would be a great fit as the sole database solution. I understand that you and your team might hear about Datomic's limited write-throughput due to global transactor in a single system, but this is again extremely relative, and is only really a problem at very high throughputs, or poor product fits like realtime applications (gaming, heavy clickstream logs, etc.). I doubt for financial data you reach anything close to troubling Datomic's write throughput, and even if someday you do, Cognitect's new backing is literally Nubank, the largest digital bank in the world, run primarily on Datomic (with advanced sharding etc., I'm sure you could get support if this ever becomes a necessity). Read latency is a more relevant problem especially if you want to force your application to use Google Cloud's stack for Cloud Functions etc., though unless this is a strict requirement of your organization, I'm not sure why AWS Lambdas alone wouldn't be good enough. AWS Cognito also works pretty nicely with lambda hooks into automatically syncing user information into Datomic with some simple setup. There's also just HTTP-direct for production query groups which is really fast and cuts out lambda latency - JWT authentication would let your mobile app directly hit this endpoint too without much fuss.

alekszelark06:02:20

Hi! I know Datomic doesn’t support ordered lists. However, there are some 3rd party implementations of linked list or indexed list data structures. We want to support some simple tables. Let’s say we have a following schema: a simple table with 2-column rows.

[{:db/ident :table/row
  :db/valueType :db.type/ref
  :db/isComponent true
  :db/cardinality :db.cardinality/many}

 {:db/ident :row/a
  :db/valueType :db.type/long
  :db/cardinality :db.cardinality/one}

 {:db/ident :row/b
  :db/valueType :db.type/long
  :db/cardinality :db.cardinality/one}]
Then, we want to transact some data
(d/transact conn [{:table/row [{:row/a 1 :row/b 2}
                               {:row/a 10 :row/b 2}
                               ; thousands of the other rows
                               
                               ]}])
Considering two things: all rows are committed within single transaction, and we won’t add other rows to the table down the road — can we rely on entity ids for order?

tatut07:02:12

I don’t think there’s any guarantee about :db/id monotonically increasing

tatut07:02:50

best to treat them only as opaque identifiers from what I gather

akis20:02:28

Hey all! What's the simplest way to use Datomic for a hobby project? I've tried using datomic ions for a week or so, but the additional complexity around AWS resulted in spending most of time at infrastructure related stuff

jjttjj20:02:38

Have you looked at dev-local? It seems designed for what you describe https://docs.datomic.com/cloud/dev-local.html

akis20:02:00

I've tried dev-local at first for learning datalog, but main concern I have with using it in a deployed app is durability and eventual scaling

em20:02:20

What's been a pain point for Ions? I've found it to be really simple to use and actually abstracts away a lot of the annoying AWS complexity that you usually need to do to setup nodes, VPN, etc.

akis20:02:44

I've managed to follow the tutorial fine, but here's where I got stuck: I created a lambda proxy resource which acted as a single entry point to my system, with Cognito Authorizer. It looked something like this:

(def handler
  (ring/ring-handler
   (ring/router
    [["/api"
      ["/test" {:get (fn [_]
                       (-> (ok {:foo "bar"})
                           (res/header  "Access-Control-Allow-Origin" "*")))}]]])
   (ring/create-default-handler)))

(def handler-lambda-proxy
  (apigw/ionize handler))
After enabling authorizer on that resource, preflight requests were getting rejected with 401 error (it makes sense since all requests need to be authorized). I've enabled CORS on it, but couldn't get it to work

akis20:02:23

So definitely not issues related to datomic, but I feel like I've added a lot of infrastructure complexity, and would like to take a step back to a more simpler system

JohnJ20:02:30

datomic starter

Joe Lane20:02:10

@UKDLTFSE4 your problem is that your cognitive authorizor in api gateway is authenticating the options method causing the cors preflight to fail. If instead of using ANY for your http method try explicitly making get put post and delete authenticates but leave options unauthenticated.

akis20:02:45

@U0CJ19XAM does that mean I shouldn't use a lambda proxy at all?

akis20:02:00

it seems that lambda proxy endpoint needs to have ANY

em20:02:33

Ahh, I see, that's far more of an AWS documentation encouraging complex solutions, rather than help clarifying what's going on. Cognito works great with Datomic Ions, but as @U0CJ19XAM is pointing out, it's more of an issue of the extra layer of authorization you're putting on top of API gateway. I really suggest not using an extra lambda step - it's entirely wasted latency that you could easily bake into the actual processing of your code, given that unless you're on production, you already have a lambda ingress

Joe Lane20:02:35

Not what I’m suggesting, lambda proxy is fine. It does NOT need to be ANY.

em21:02:20

You could also fix the lambda proxy, but if you want to do Cognito auth, just verify the public key with one function in your handler would save you all this hassle

Joe Lane21:02:09

@UNRDXKBNY I think for "getting started" ion lambda proxy is a fantastic fit, otherwise he will need to "get started" with a production topology to get access to http-direct, kind of a non-starter. He has ALSO already correctly configured cognito, it's just a confusing interaction between api-gateway and cognito.

Joe Lane21:02:02

@UKDLTFSE4 Make 5 explicit methods, each calling the same lambda. Then don't auth the options request and you should be good to go 👍 FWIW, I'm so confident this is the issue because I've hit it so many times. I brought it up with the product team literally this week as something we could better document.

akis21:02:54

Thanks a lot for you suggestions, I appreciate it! I'll give it another shot and try that

em21:02:56

Yeah that's entirely fair, although even in the Ion solo configuration, the lambda ingress into the system isn't a blocker for simply doing the Cognito JWT auth in your handler. It saves a lambda's worth of latency, adds just one dependency to your code (buddy or other JWT library you like), still uses Cognito as IDP, and also saves about $10 a month since the solo lambda is so terrible in latency unless you do provisioned concurrency.

em21:02:57

The one benefit of having an extra external step of a lambda proxy authorizer is blocking a lot of unwanted traffic without taxing your server, but this doesn't seem very applicable on a hobby or even medium startup level situation

Joe Lane21:02:13

You can set up a heartbeat on your ion so it's warm before going to provisioned concurrency. Let’s keep the costs down

Joe Lane21:02:14

@UKDLTFSE4 if you're still stuck after the second try dm me and we will get it sorted out.

em21:02:13

@U0CJ19XAM Yeah, you're absolutely right about the heartbeat approach, forgot about the good old cloudwatch ping since recent clients insisted on AWS guaranteed "provisioned concurrency", which is honestly basically punting on serverless entirely and going back to full instances... On that note, I've always wondered, is there a specific reason Lambda the ultimate is on the JVM? As far as I understand, it's literally just a nice proxy/entry point and doesn't need heavy computation, so a clojure script implementation on node.js with a much less punishing cold-start ceiling seems like it'd solve a lot of these headaches. It's a feature I desperately wanted for a long time and seems it'd mesh better for serverless-esque workloads

Joe Lane21:02:06

@UNRDXKBNY If you're starting a new project, the heartbeat is sufficient for keeping costs down and avoiding coldstart. If you are scaling and need to increase the number of concurrent lambdas and worried about coldstart in this context, add a timeout to the client request and retry after you're over x% of your average latency (where you pick x). You'll likely be routed to a different lambda at that point and still finish the request faster than waiting for the coldstart.

em21:02:09

@U0CJ19XAM That's a cool approach! Haven't thought about the probabilistic retry method when concurrency on lambdas factors in, super helpful. Would it be fair to say that with these methods to combat cold start, the JVM lambda has better throughput considering warm instances than other platforms with less cold-start ceiling? Love Datomic, just trying to understand the design choices as a matter of curiosity and learning

Joe Lane21:02:55

Glad you're a fan! There are many other factors besides performance that factor in to the choice of JVM lambda vs "other" like maintainability, stability, ecosystem, stdlib, etc. I'm not going to make a general performance claim here, because the second I do, someone will throw a benchmark at me for a specific case where x is faster than y and call me a liar 🙂 Luckily, this question has nothing to do with Datomic and you can defer to all the other liars on the internet who made their own benchmarks and conclusions. In all seriousness, if performance characteristics are critical for your use-case, YOU need to do the measurements for your scenario and decide for yourself.

💯 1
souenzzo12:02:20

@UKDLTFSE4 checkout #datalog, #rdf, #asami, #datahike and many other datomic-like db's for hobby 🙂

akis01:02:30

@U2J4FRT2T thanks for the suggestions, I know about datahike but didn't know about rdf and asami

akis01:02:02

Just a quick update on my problem, in case it becomes relevant to anyone else. It turns out that CORS issues I was having were related to */* binary media type. After removing it, OPTION requests are resolving successfully, and other methods are properly authenticated with cognito