Fork me on GitHub
#datascript
<
2018-05-22
>
kiemdoder05:05:16

@ronb you mentioned that your version of Datascript is not backwards compatible. What are the breaking changes?

ronb09:05:23

@kiemdoder Attributes are stored as Integers (d/datom 0 :attr :val) is not allowed anymore Queries return attributes as Integers, not Keywords You need to define the Attribute order in the schema -> {:attr {:db/order 0}} Transaction Id’s are created from 2^24 -1 downwards New range for Entity ID’s is from 0 to 2^24 New range for valid Transaction ID’s is from 2^24 to (2^24) - 2^20

ronb09:05:39

Every attribute is represented as an ID (similar to Datomic) All attributes need a db/order key in the schema

ronb09:05:57

{:attr {:db/order 0} :attr2 {:db/order 1}}

ronb09:05:14

Will do some benchmarking and better Docs. @tonsky Thanks for the reply on the pull request. I will reply later

Niki09:05:20

I’d say, semantically, biggest setback for existing users would be: — lack of datom call with keyword attribute — unexpected integers in query results (although I believe impl can be changed to return keywords) — the need to specify all attributes in schema (:db/order could be probably calculated, but it requires all attributes to be specified explicitly)

ronb09:05:29

@tonsky I completely agree. Right now the backwards incompatibilities are too big. If I address the issues you mentioned, would you be willing to merge the pull request or do you prefer a seperate project?

ronb09:05:00

most incompatibilities could be addressed with code in db/-search and (transact-add), but changes in how (datom) constructor works would remain and probably be confusing to users

Niki09:05:14

I don’t see how we can work around requiring that every attribute must be specified in schema. So far it’s not required and I believe most users rely on that

Niki09:05:38

And then there’s JS story, which lacks 64-bit numbers so that schema wouldn’t work as well there :(

Niki09:05:47

it feels like jvm-only, separate project with no backwards compatibility guarantees would be best

ronb09:05:22

Yeah it would require additional logic in transaction handling to add new attributes on demand and is a bit messy 😄

ronb09:05:34

i use 64 bit floats to guarantee compatibility with JS environments. All tests run successfully on JVM and JS as far as I can see

ronb09:05:00

No problem. I will close the PR later. Thanks for the feedback

ronb10:05:06

You can cheat if you do the following: (/ num (js/Math.pow 2 32)) -> gives access to the upper part of float numbers in the lower 32 bit region

ronb10:05:18

for changing the upper part you can do (+ (* upper (js/Math.pow 2 32)) lower-bits)

Niki10:05:09

isn’t it like SUPER slow?

ronb10:05:34

as far as I can see this is in conformance with IEEE Floating point spec

ronb10:05:43

Will test the performance in the evening, but i compared this with bit-shift operators and they have similar performance. Looks like JS engine optimize (/ num some-binary-exponent) into (bit-shift-right )

thedavidmeister14:05:04

^^ maybe helpful?

Niki14:05:29

btw maybe instead of trying to fit everything into one 64-bit value, it might be easier to fit everything in two 32-bit values? memory consumption will be the same

thedavidmeister14:05:51

that's what goog.math.Long and therefore ^^ does

Niki14:05:25

No, I mean, there’s no reason to pretend it should look like single 64-bit at any point

thedavidmeister14:05:47

depends what you want to do

Niki14:05:03

what @ronb is doing

ronb16:05:30

@thedavidmeister Looks interesting, thanks for the link. I looked into using goog.math.Long based solution. but if i understand correctly how js engines work, it would use twice the memory of a single double number

ronb16:05:11

although maybe i am mistaken and engines actually detect that you only use 32bits of the number and optimize the rest away

ronb16:05:31

this benchmark compares a bitshift operation with division. and shows a 30% performance advantage for shift based operation. which is not significant enough in my use case to be detectable, as performance is largely memory bound

ronb20:05:11

just tested memory usage of 32bit numbers and values that go beyond 32bits. they both use the same amount of memory