Fork me on GitHub
#untangled
<
2016-04-28
>
tony.kay04:04:44

@currentoor: Yeah, @davewo and I worked on a sente-based entity subscription system as a tech spike. We don’t need it yet, so could not spend time refining it for production use yet, but agreed: Datomic tx monitor makes it relatively easy to subscribe not only to an entity, but an entire graph query.

currentoor05:04:30

@tony.kay: a lot of exciting stuff happening over here 😄

tomjack06:04:43

curious if you tried something for unsubscription? I am interested to see any implementation of that

tomjack07:04:47

for the client, I mean

kauko12:04:21

I already asked this question over at #C0620C0C8 and at the posh gitter channel, but I figured might as well ask here too.. I've always thought DataScript looks really interesting, and I thought about starting a small demo project today with Reagent+Posh. But then I realised, I don't know what exactly is the difference of Om Next and Reagent+Posh. If I'm doing the work of wiring up Posh+DataScript to my Reagent application to get colocated data queries and whatnot, why wouldn't I just use Om Next?

kauko12:04:52

Am I not just recreating Om Next if I use Posh? Won't I run into problems that Om Next has already solved for me?

kauko12:04:42

The reason I'm asking here and not over at #C06DT2YSY, is that there were a couple of people waiting for answers to their own questions and I didn't want to bury them simple_smile Besides, I'd probably try to go with untangled and not just pure om, since untangled probably solves some problems I'd run into with just om

cjmurphy12:04:06

One thing to keep in mind is that Untangled currently works with default database format rather than DataScript.

kauko12:04:57

I don't actually know what would be the advantage of using DataScript with Om rather than the default database format either.

kauko12:04:29

I've read the tutorial about using datascript with om, but didn't really see the point

cjmurphy12:04:39

You read my mind just then - about 'what is the point of Datascript?' Datascript's disadvantage is that it is slower.

cjmurphy12:04:08

Posh is similar to reframe from what I understand?? The only disadvantage I can think of with Untangled would be the Om Next learning curve. Posh would I imagine be easier to get up and running quickly with.

kauko12:04:58

Yeah, that's honestly kind of what I feel like too. I've been looking to get into Om Next, but honestly, it's really damn difficult to get into. But then when I started thinking about setting up a project with reagent+posh+datascript+datomic+whatnot, I realised I'll probably just end up learning/recreating all the concepts Om Next already has, you know?

kauko12:04:24

and if Posh doesn't force me to think about a difficult concept that Om Next has, it probably means that I'm going to run into problems later

kauko12:04:39

I mean, it's probably a challenge that Posh doesn't solve

cjmurphy12:04:07

And if Posh is like reframe then it doesn't give you anything on the server side.

kauko12:04:29

wait, om next does?

cjmurphy12:04:32

Yes - both Om Next and Untangled are for client and server.

pithyless12:04:22

I’m very much interested in untangled, but with DataScript integration. My case is mainly client-side only, where I fetch lots of raw data that I need to parse into DS and make datalog queries on. I don’t want these queries to be done server-side. If I used the default om.next structure, I would need to re-invent DS for any non-obvious queries. I’m going to experiment with loading all my business data into DS on the side and use the default untangled state for keeping a materialized view of just the data I currently want to render.

cjmurphy12:04:35

Untangled does your reads for you (in Om Next you write them). And its reads assume default db format. So I don't see it being easy/possible to use DataScript with Untangled.

cjmurphy12:04:21

Except that's why you have the indirection you mentioned...

kauko12:04:17

Uhm, so the advantage of using datascript with om would be that datascript is faster than the default format?

pithyless12:04:09

It would be slower, since there is a level of indirection. But it let’s you write queries you otherwise would not be able to write with the default format.

kauko12:04:13

Sorry, this is a topic that's been bothering me for a long time. I guess I've never really understood what the advantage of datascript is.

kauko12:04:28

Alright, so DataScript has a more refined query engine?

pithyless12:04:14

Well, as far as I gather, all you get with Om/Next is db->tree and a basic parse function you can extend. If all you need to fetch are idents + pull api it should suffice.

tony.kay15:04:05

@cjmurphy: @pithyless @kauko Datascript has definite use-cases, and you can totally use it with untangled: Use the default DB format for the UI...use datascript as a separate database. On post-mutations, you can totally populate things into datascript. This could be useful for example, if you want to use complex non-ui queries to compute things...then put them into the UI database. This gives you the speed of the default db format for fast rendering. The ideal schema for storage (e.g. datomic/datascript) will never match UI...which is the reason for the Om parser...but that is a mess of complexity itself.

tony.kay15:04:42

OR perhaps you set up a web socket, subscriptions, and populate those into datascript. Then do your transform from a DS query -> UI database for rendering

tony.kay15:04:02

This turns the need for a parser into simple data transforms, which are easier to reason about and code

kauko15:04:14

Alright, makes sense!

pithyless15:04:45

@tony.kay: this is exactly what I was thinking of trying; DS for data, default DB for UI. Thanks for the feedback simple_smile

kauko16:04:38

@tony.kay: is the performance thing you mentioned related to the thing where Om has static queries, so you always only need one query to get all the data you need? I mean, does this only work with the "UI db"? Or is the default db just somehow inherently more performant than datascript?

tony.kay16:04:24

@kauko: The default db format is several orders of magnitude faster than datascript. Datascript is a real database with indexes, etc. The default db format is just maps...which are very very fast

tony.kay16:04:04

So, if you need real queries of complex nature, datascript may make sense...but for converting a UI query to UI tree....the plain map thing gets you good UI performance.

kauko17:04:37

Does foam work with untangled by the way?

tony.kay17:04:42

no idea what foam is, sorry

kauko17:04:04

server-side rendering for om. Sorry, should've mentioned that earlier

kauko17:04:36

"Bugs everywhere" 😄

tony.kay17:04:27

@kauko: On posh: The problem with co-located queries and datascript is that your datascript database is not going to match your UI structure...unless you structure it that way. Om has you make a parser, but that is kinda hard too. So, not only is there no server integration, you still have to solve the mis-match between your real data schema and your UI...and you have performance issues with running a real database to server a UI. So, while Datascript sounds like a good idea in theory, I think if you try to drive your UI with it you'll end up with big problems you can't easily solve as the app gets big

kauko17:04:19

Hmm, not sure I understand. Doesn't Om next structure its data in a.. relational manner? It's not a tree like it was in Om?

kauko17:04:51

I don't see how Om's database (if you can call it that) can match the UI structure, but datascript's can not

tony.kay17:04:26

They both can.

kauko17:04:28

Then again I've never actually used either one. Perhaps this would be obvious if I had more experience

tony.kay17:04:59

that is not the point. The point is that typically the reason for wanting DS is to do advanced queries and use datalog on real data....and that schema is almost certainly not going to match your UI

tony.kay17:04:37

so...use a simple db (maps) for UI (which wants speed) and keep DS on the side to do the advaned stuff (which for example only updates on server pushes of subscribed server data in good schema form)

tony.kay17:04:26

There's no point to use DS if all you want is a UI db...it's like putting a thumbtack in with a jackhammer

tony.kay17:04:51

just my opionion, of course..there are those that will disagree

curtosis17:04:59

nested maps is not really the best use of DS. simple_smile

kauko17:04:16

I see, I think I understand. Can you give me an example of an "advanced query"? At what point do you think would it be justifiable to bring in DS?

tony.kay17:04:33

See datalog.

tony.kay17:04:39

Datomic docs

kauko17:04:49

I imagine this is related to the limits of Om's pull syntax (I believe that's what it's called)?

tony.kay17:04:03

right. Pull syntax is for getting a tree from a graph

tony.kay17:04:20

datalog is more like SQL (general-purpose joins, aggregates, etc)

curtosis17:04:24

[unrelated] so I've been looking at the Tabbed Interface recipe, and am wondering ... is that the same approach you'd follow for a (possibly-accordioned) menu bar?

tony.kay17:04:47

@curtosis: maybe, but more likely there you'd just use CSS to do folding

tony.kay17:04:04

might use union query to deal with heterogenous content of the bar items

tony.kay17:04:15

see to-many union example in Om wiki

curtosis17:04:36

@tony.kay: that's sort of what I was leaning towards, with swapping out the main ui section based on whatever the "view" selected is.

kauko17:04:46

Thanks Tony for the comments, I've learned a lot. It feels like some sort of a comparison of Posh and Om could be pretty valuable to the community, although that could be difficult to write. I've been talking to some of the Posh folks over at gitter today too

kauko17:04:03

It feels like the main thing keeping people away from Om is that Om looks pretty hard, and it feels like it requires quite a bit of code to get set up. Hopefully Untangled will help with this

kauko17:04:46

This is one of the things that makes Reagent so popular: it's so damn easy to get going with it. Since Posh was initially built on top of Reagent, it's probably easier for peopel to adopt Posh rather than Om

tony.kay17:04:33

@curtosis: Yes @kauko: Only time will tell....I agree Om is too unapproachable for the masses. Hopefully Untangled helps.

brianosaurus17:04:09

The reason I mention this is that it has a lexical side effect … things lexically scoped within the macro can’t log info unless they explicitly pass in a log level. Also, it’s a test so I’m not sure if only showing fatal is necessary? What do you think?

adambros18:04:45

@brianosaurus: if you remove that you can end up with lots of logging around schema & db setup i could be persuaded to instead of removing it, wrapping only the db-fixture & the stop in the with-level, but not the form

brianosaurus18:04:24

that’s a great choice.

adambros18:04:50

do you need me to do it? otherwise a PR should be fine

brianosaurus18:04:14

I’ll add a PR. Once again, I’m in crunch mode until Monday…it might be a couple days unless I decide to do it tonight.

adambros18:04:35

sounds good

kauko18:04:00

Huh. I'm trying to run the untangled tutorial, I got the thing working once with the "all builds" profile, but now no matter what I do I get an empty page and in the console "Installing cljs-devtools: custom-formatters dirac sanity-hints"

kauko18:04:33

Compiling "resources/public/js/tutorial.js" from ["src/tutorial" "src/shared"]... {:tag :cljs/analysis-error} ANALYSIS ERROR: No reader function for tag js in file file:/Users/teemukau/code/untangled-tutorial/src/tutorial/untangled_tutorial/I_Building_A_Server_Exercises.cljs on file null, line null, column null

kauko18:04:37

Could this be why?

adambros18:04:24

not sure why, but I can tell you that that error is possibly the most annoying & useless error I’ve ever seen i have however had success running lein clean (and optionally git clean -xdf)

adambros18:04:41

but i have my doubts if you just cloned untangled-tutorial

adambros18:04:12

and as a last resort wiping untangled & co. from your ~/.m2/repository/

adambros18:04:04

i might ask that error to #C03S1L9DN, and maybe see if @tony.kay has anything

kauko18:04:22

seems to work now

adambros18:04:30

haha, what fixed it?

kauko18:04:46

lein clean did it. Don't know why I didn't think of it at first, I use it all the time with weird problems like this

adambros18:04:37

i might still ask #C03S1L9DN just because the no reader function for tag is really annoying

tony.kay18:04:54

Didn't A.M. note that as a known bug

tony.kay18:04:26

oh, no. that was something else

tony.kay18:04:38

it is probably a reported bug

tony.kay18:04:43

hard to keep track

adambros18:04:00

as i said, a healthy dosage of lein clean usually does the trick

tony.kay18:04:03

So, just finished the design for full server-push integration. We should see something pop out in Untangled in the next week or so, and I'll finish a decent implementation of background-loading on top of that. I'm thinking I can get the effect of background (parallel) loading with plain send as well as a fallback, but the websocket implementation will be a lot better for the real use-case people want (e.g. long-running things where we don't want to worry about timeouts and inventing thread pools)

anmonteiro18:04:05

@kauko: which version of Clojurescript were you using?

anmonteiro18:04:51

if earlier than 1.8.34, that’s something that has been fixed

anmonteiro18:04:05

has to do with the analysis compiler pass

adambros19:04:30

thanks @anmonteiro, now if only i could replicate it and see if it fixes it once I upgrade

anmonteiro19:04:57

way to replicate is: - compile once

anmonteiro19:04:05

delete some analysis cache file

anmonteiro19:04:08

compile again

anmonteiro19:04:14

see if it breaks

tony.kay19:04:30

In general, we should upgrade all of our deps to the latest and greatest cljs. Lots of bugs fixes happen all the time. Assuming our other deps are not broken as a result. I've been waiting for a time to test that out...anyone is welcome to play with it and let me know what combo of things works (figwheel, cljs, clojure, etc)

tony.kay19:04:57

Good idea to keep the versions bumped on a regular basis

anmonteiro19:04:02

@adambros: truncating your cache.edn file should allow you to repro

anmonteiro19:04:34

@tony.kay: re: deps, CLJS 1.8.51 was released the other day

tony.kay19:04:15

Yeah, I just want to make sure everything works with it...e.g. lein cljsbuild, figwheel, etc

tony.kay19:04:37

dependency tree issues, etc.

therabidbanana20:04:55

I'm looking into helping to get a deployable version of our untangled app packaged, and one of the challenges is providing a proper config.edn file - we prefer using 12 Factor style config in our deployment strategy, where we can change individual configs with environment flags. I'm considering the implications of replacing the config component with our own custom environ-powered config component. Seems like it should work, but are there any obvious issues with that I'm missing?

therabidbanana20:04:52

I don't know that redefining the config key to override the original config component will work as expected, I guess that's the first thing to verify.

therabidbanana20:04:10

Yeah, that definitely works - is it likely to break anything as long as we keep the new config component looking like the old one (map with all config stuffed in :value)?

adambros20:04:48

so i wrote the config component under @tony.kay's direction but I can confirm your last question, that as long as its under [:config :value] in the system it will work however you implement it

adambros20:04:30

i would be curious to know if you think if being able to reference system variables in your config, with something like :ENV/name would suffice?

therabidbanana20:04:48

That would likely suffice, yes.

adambros20:04:51

(or a reader tag)

adambros20:04:59

im not familiar with 12 factor style config, but you can easily have different config files per user & machine

therabidbanana20:04:23

We do mix in some things with a boot task, but I'm not sure if we use that really

adambros20:04:32

eg: when you run the app with java -jar app.jar you have to specify -Dconfig=/path/to/config.edn

adambros20:04:46

you could dispatch that path based on some env var

adambros20:04:56

eg: -Dconfig=$CONFIG_PATH

therabidbanana20:04:43

So 12 factor is a thing heroku came up with (I think), one of the things is basically using the shell environment for any configuration, to avoid having to have a different file per server in your build DB_URL=.... DB_MIGRATIONS=... java -jar app.jar

therabidbanana20:04:31

It makes it pretty easy to update configuration - you don't have to roll a new build or modify config files in place

therabidbanana20:04:59

I'm not sure how much of our decision to go that route is just based on habit from our other apps being hosted on Heroku though

gardnervickers20:04:20

@therabidbanana: Have a look at Aero https://github.com/juxt/aero, it’s a nice compromise between the 12factor style and storing things in a config file. It’s also extensible through edn readers.

gardnervickers20:04:19

{:favorite-color #env [FAVORITE_COLOR “blue”]} reads to {:favorite-color “blue”} only if the env-var FAVORITE_COLOR is not set

gardnervickers20:04:11

There’s also support for Om.next style links through the ^:ref tag, that takes a get-in style vector of keys into the config.

adambros20:04:32

aero looks nice, and as long as you can put the configuration in the system under [:config :value] i see no reason why you couldn’t use it

adambros20:04:59

or any other config library

therabidbanana21:04:37

Nice - thanks @gardnervickers - I'll take a look. Sounds like it could help.

therabidbanana21:04:50

Might be something that could be mixed into the untangled config component? Then people could optionally support env vars without the component needing to get much more complicated to implement?

adambros21:04:09

personally i would be ok with aero, i would just want to see what @tony.kay thinks, as i have little experience with configuration & deployment