morning

moin moin

'allo 'allo

good morning

morning friends

anyone planning on going to ClojureD?

morning!

hi all. i thought i'd let you know i'm feeling very positive about clojure this week

woop woop!

Are you doing anything fun @conan?

plugging a brand new re-frame app together using the specs i've already made for all my datomic schema

one keyword to rule them all

very cool 😄

Hello All - Good Morning!

@conan - That does sound very satisfying and exciting 🙂

Mmm yes

https://twitter.com/adamjogrady/status/952784870081572864

Oh bravo… 🙂

Have I misunderstood this, or do all datoms get a “timestamp” as they are added to a Datomic database (i.e. there is no need to explicitly include a “date-added” attribute to a schema)?

yes they do, that's how datomic can query at a specific time in history

datoms are 5 things: [entity-id attribute-name attribute-value transaction-id timestamp]

in practice, you usually only work with the first three

@conan - Thanks, I thought so… 🙂 And so, if I’ve understood correctly, schema associate certain datoms together in a “shape” that can be returned together for convenience but more specifically to allow for data deeper than one single fact to be modelled in the database?

yes. a datomic schema is just a bunch of datoms, except these datoms are used in an extra way: to validate incoming datoms to the transactor. they're single-datom validators though, you can't have relational constraints in the way that you can in a RDBMS for example

And so to follow on from that, how do you and indeed other people deal with db.type/ref entries that might be coming from data that is not yet present..? i.e. I am defining a schema at the moment and I want to include a ref to an organisation, but I’ve not defined my organisation schema yet. Is that an issue..? More to the point there is no data yet at all, and when I come to load some in, will I need to load in organisation data before any other data that may require a reference to an organisation?

so you can say that an attribute has to be a ref, or it has to be a string, but you can't say that it has to exist and the thing it points to has to exist as well

@conan - That’s an interesting distinction, but I am happy with not being able to enforce such things

🙂

off the top of my head i'm not sure what happens if you add a schema for a ref attribute that points to another attribute for which there is not yet a schema. my guess is that you wouldn't be able to add it, but datomic is pretty flexible so it might let you get away with it. after all, it's still capable of enforcing the fact that every value for your new ref attribute is a ref that points to something - it may just not let you add any of these attributes if the thing it points to can't exist.

i suspect the schema datoms will be rejected though.

using spec is a great way to pick up on this stuff though, as when doing generative testing you'll definitely end up with values being created, and if datomic isn't happy with them it'll complain at you quick smart

i wrote a whole bunch of specs first, and when i came to model them with datomic schema it literally all worked first time. (admittedly it didn't work second time, i got lucky with a couple of edge cases, but it was very easy to fix - and these edge cases were bugs for sure)

but then you can't put nil as a value in datomic - so if an attribute exists for an entity, you know for sure it is the type defined by its schema. if it's a :db/ref and it exists, then it is definitely pointing somewhere - i'm not sure whether that means the somewhere has to exist, i don't think it does

i love this stuff

Ah… I was starting with schema - perhaps I will get a better result starting with spec(s)..?

i always think you get a better result starting with spec

Datomic schema are not COMPLETELY new to me, clojure.spec is - I may have been being a little bit “easy route” about this…

it's like doing TDD, except you only have to think about the data to begin with, and can add behaviour later. all from your hammock, of course

nothing wrong with doing what helps you learn best

I was intending to write clojure.spec specs for this data…

if you want to learn how to write custom specs, i wrote a blog post about it http://conan.is/blogging/a-spec-for-urls-in-clojure.html

maybe I will start there.

Thanks!

Is there a low-friction way to convert specs to schema or does one still have to hand-ball the schema afterwards?

i'd just get the hang of sticking some specs together for whatever data you already have. the fact that they're all namespaced keywords is very convenient when it comes to making datomic schema from them

i did it all by hand

OK 🙂

Cool

it's simple, if not easy 😄

hehe

If it were easy EVERYONE would be doing it…

RE: specs Spotted this on https://www.reddit.com/r/Clojure/comments/7qdnkx/curated_list_of_public_clojure_specs/

oh handy

Yeah, that’s a handy resource for sure - thanks @shan

Yeah @shan, I saw someone tweeting about that the other day too

Have people reached a consensus yet about how to organise / store and use specs..?

I mean do they “live” in .edn files in say “/resources/specs” and then as part of the app initialisation we read them in, or do they co-exist in namespaces that need them, or something else entirely..? I realise that the beauty of this is that as long as I can access them and don’t break anything else that I can do it any way I damn well please, but I am generally interested in at least knowing about the “best practice” way of doing something, even if that may change or is the subject of some debate…

i put mine in a namespace that's modelling the thing they're for. so for example, i have a personnamespace, and it contains all the specs that are :person/full-name and stuff

try to keep it together so you always have the specs you need near to hand. i think this makes sense when you have an fdef next to each of your functions too

fdef?

a spec for functions if I am not mistaken

Oh right…

and the curated list of specs is a cool and very good idea!

@conan - Would you be prepared to share / show your person namespace - a lot of the documentation I am trying to use to get my head around all of this is very fragmented when all I really__ want / need is a complete example…

(totally understand if not - I realise it might be proprietary etc - but I thought I may as well ask 😉 )

@maleghast: Well technically spec isn’t really edn, it’s clj so putting them in resources isn’t quite right, but you might put some/all of them in a different :source-path/`profile` depending on your needs.

Yeah, I am starting to think that I want them alongside / inline with the code that is going to use them

??

That the whole thing is actually a very different paradigm from other kinds of data template (e.g. JSONSchema)

(as far as I can tell)

It is a different paradigm. Regarding where to put things your intuition os probably a good one. But it can get more subtle. e.g You might want different specs for the same thing depending on context, and they themselves might reuse bits and pieces. Also think about specificity. Some things can be harder to spec precisely… but then if you’re doing generative tests a broad spec can generate invalid inputs (though you can also swap out the generators — but then you also need to think about where to put/share them)

Another aspect is that (require '[,,, :as ,,,]) complects aliasing with namespace loading, so you can only alias specs that live in a namespace.

^ this is really useful i’ve found

but typically putting specs for foo.bar in foo.bar.spec or putting them in foo.bar is a good start 🙂

spec’s can also take up a lot of locs, so splitting them from the code makes sense for mid-sized+ namespaces

@guy It is but it’s worth being aware of the limitations it imposes. As it means you can’t alias keys that don’t exist in namespaces. For example the spec for :mvn/version in tools.deps doesn’t exist in an mvn namespace (but the keys short so it doesn’t matter). I guess what I’m saying is that sometimes you want a short key for brevity, and the spec might want to be in a more fully qualified namespace.

This is all good / useful insights - thanks very much

Another insight I had is that spec can be used for a lot of things (you’ll all know that): - validation - documentation / tool for thought - parsing data structures (conforming) - generative testing - test data generation - instrumentation - error reporting - … However, when starting out go slow, and concentrate on just one or two aspects. With generative testing in particular be cautious of the time cost associated with it, and be careful to apply it to low hanging fruit / well suited areas first. Basically spec supports a lot of workflows; and it can be hard to adopt them all simultaneously. Personal recommendation is probably not to jump in whole-hog, but to add specs where they can help most (e.g. around your core data model), and it’s quite easy to use spec in your existing example-based tests. Later on you may be able to transition to more generative tests; though I’m not so clear on how to integrate this style of testing with e.g. clojure.test and to get good clojure.test errors out. (I’ve found test.check integrates here more easily.)

Also there are no docstrings for spec yet… so the documentation benefit isn’t fully realised.

@jonpither thx for the http://autonomous.ai suggestion. I'm standing in front of mine now. 🙂