datahike

alekcz 2024-11-09T06:39:11.523609Z

@whilo if I import datoms that were exported to cbor, will my history be preserved?

alekcz 2024-11-09T08:01:05.529739Z

I need to go sort out some family stuff. I'll pop and example on a bit later.

alekcz 2024-11-09T08:01:15.167539Z

This is an open issue it seems: https://github.com/replikativ/datahike/issues/633

whilo 2024-11-09T08:10:43.091849Z

Oh, right. Thanks for pulling it out.

whilo 2024-11-09T08:10:57.175679Z

@hoertlehner also provided a workaround there.

whilo 2024-11-09T07:09:11.206399Z

The history should be exported https://github.com/replikativ/datahike/blob/main/src/datahike/migrate.clj#L16

whilo 2024-11-09T07:09:36.630919Z

But I haven't used this code lately, I am not sure whether the history is also correctly imported.

whilo 2024-11-09T07:09:47.291319Z

Are you having problems with that?

alekcz 2024-11-09T07:10:14.640259Z

We shall soon find out. Wanted to ask before I try

alekcz 2024-11-09T07:16:46.282179Z

Yeah it didn't work 😭

alekcz 2024-11-09T07:16:54.900909Z

Bad entity value 0.0 at [:db/add 487 :expense/salestaxamount 0.0 536870922], value does not match schema definition. Must be conform to: double?

alekcz 2024-11-09T07:17:04.429089Z

0.0 is not deemed to be a double

whilo 2024-11-09T07:25:07.378559Z

Can you just copy the import code and add a case to https://github.com/replikativ/datahike/blob/main/src/datahike/migrate.clj#L27 to also convert to double?

whilo 2024-11-09T07:25:40.941039Z

@yeefay.lim wrote this code some time ago. It would be helpful to know a bit more context.

whilo 2024-11-09T07:26:30.575819Z

@alekcz360 can you also explain briefly what your current requirements are?

alekcz 2024-11-09T07:27:15.449859Z

I moved to Google Cloud because http://render.com didn't have ISO certification. They have now achieved the certification. So I'm migrating back.

alekcz 2024-11-09T07:28:07.561349Z

I've backed up my beta environment and I'm testing the migration on 400k datoms. If all goes well I will migrate the 3M datoms

alekcz 2024-11-09T07:28:38.601999Z

The data is accounting data so there are lots of different types. int, double, longs, dates, and strings.

whilo 2024-11-09T07:32:06.108649Z

Different types should be fine, I guess CBOR saved the data as float and not double for some reason. Did you explicitly store floats?

alekcz 2024-11-09T07:32:43.161579Z

Nope. In my schema they are doubles.

whilo 2024-11-09T07:35:08.016829Z

Then the export code is broken.

whilo 2024-11-09T07:41:02.726069Z

The problem is actually in Clojure. float? is true also for doubles.

whilo 2024-11-09T07:41:18.210779Z

Which makes CBOR export it as a float.

whilo 2024-11-09T07:41:41.532379Z

No, wait.

whilo 2024-11-09T07:52:01.724449Z

So doubles seem to roundtrip ok through clj-cbor.

whilo 2024-11-09T07:57:53.527549Z

Hmm, weird, I cannot reproduce that a double attribute in the schema turns into a float on import. If you have a small example to reproduce that would help.