Fork me on GitHub
#sql
<
2024-05-07
>
lepistane09:05:23

anyone using dbt for db model management in production? Any helpful tips, tricks, tools and experiences to share?

lepistane13:05:31

Not sure if i can influence decision. We are not using it but there is big push for it for some reason. I know nothing about it so i was asking around. Will check these articles. Thanks! dbt was only mentioned to me in the context of 'creating tables' but i am not sure of any other detail

p-himik13:05:06

Of course, I have no idea about the specifics of your situation, but on the surface it sounds like some manager attempts to enforce a technical decision. If so, the big push can easily be coming due to the stellar work of the sales team from the dbt's side. :) This crap happens all the bloody time in IT, and all it does is siphon money and time from your company and into the pockets of the ones offering "solutions".

✔️ 1
Darin Douglass14:05:50

dbt is really nice in the hands of data engineers/scientists and helps facilitate a LOT of good habits in an ecosystem that's kinda the wild wild west

Darin Douglass14:05:15

i'd not use it on the software side or for "model management", there are better migration tools for that sort of thing

lepistane14:05:02

@U02EA2T7FEH are there alternatives that you'd use?

Darin Douglass14:05:47

it really depends on what they mean by "creating tables" and "model management" so it's hard to say

Darin Douglass14:05:49

but, again, if they meant "i want a table that i can manage in my db that only has data from trial accounts" or something like that

✔️ 1
Darin Douglass14:05:53

dbt would fit pretty well

p-himik14:05:41

> i want a table that i can manage in my db that only has data from trial accounts Wouldn't a plain view be sufficient for that?

Darin Douglass14:05:46

probably; though dbt can materialize the queries as views, not just tables. but who is managing that view? if it's not a view, how often do they need the new data added? how technical are the users? can they write sql queries? if so, as a data engineer, not writing queries for people sounds really attractive

Darin Douglass14:05:18

the data world is in desperate need of a lot of things: • reusable code (notebook are a blight and are almost used exclusively from my experience) • local development (notebooks strike again...) • proper testing (guess who?) dbt isn't a magic bullet. it won't be much use (without paying a premium) if you have a LOT of data and you have a short SLA. but it does solve those 3 problems pretty well and is generally a good fit if you: • don't have a ton of data • have data scientists that need exploration and flexibility • want to provide some kind of self-service interface to your raw (or slightly curated) data sets