Fork me on GitHub
#off-topic
<
2020-05-31
>
kenj17:05:24

Has anyone had the [dis]pleasure of syncing app data into SalesForce via the SF API?

emccue23:05:13

I only know someone who has and they spoke about it like someone would speak about a rough patch in their marriage or the death of a relative

chrisulloa20:06:39

Yeah I have to do this pretty often at work

kenj20:06:54

Any tips on the sanest way to do this? Our app uses Postgres, but the business team wants a sub-set of the data to be synced over to SF which has a matching schema of objects (give or take). At this point it seems like I’m trying to create an ad-hoc replication protocol between PG -> SF, that in turn has to be able to deal with network outages and the like, and later able to catch up with syncing. That in turn seems like a legit difficult problem in distributed computing, but no one else seems to appreciate that fact.

kenj20:06:02

I’m also under the impression that all FK relationships need to be mapped to SF FKs, because SF can’t be bothered to just use the existing PG FKs…

chrisulloa20:06:33

Well, what’s your use case exactly? Storing data in two places seems really terrible. We usually opt to store data in Salesforce or Postgres (but usually not in both places). If our customer reps have the need to access data stored in Postgres, we expose an endpoint to fetch data then our SF devs will build out pages to display that data in Salesforce.

chrisulloa20:06:03

I’m not too sure on the details of how the SF devs do it, but I believe there’s a way to build custom apps within Salesforce to fetch data from a service or DB without having to actually store it in Salesforce

kenj20:06:58

Im 100% in agreement that storing data in 2 places is terrible, yet that’s exactly the requirement. I’ve tried to get the team to use Metabase to pull reports directly from PG but the team is dead set on using SF. We also have no SF devs in house.

kenj20:06:45

I think a big driver is the ability to create automated workflows to take action based on the data that is synced over into SF objects.

chrisulloa20:06:08

Oh shoot, well seems like you can’t get away from that then. As far as network outages, retries, etc. you might have luck using a serverless solution like SQS-Lambda

chrisulloa20:06:09

We have some lambdas that create objects in Salesforce when certain events happen. The SF api is easy enough to work with and with SQS-Lambda it’s easy to configure your retry behavior.

chrisulloa20:06:21

Our app also triggers object creation directly, when a user performs an action we sometimes create things in Salesforce directly from our backend service. Not sure if that helps. It sounds to me like backfilling and making sure data is perfectly synchronized will be difficult.

chrisulloa20:06:26

We also create automated workflows when a user triggers certain events in the app that require manual intervention from our customer service reps.

chrisulloa20:06:37

The Salesforce API is really not that difficult to work with

kenj21:06:49

Yeah the API is easy enough, and you are right, the backfilling and ensuring consistency is the thing I’m most worried about. The single object create/update is too slow to backfill, so I also need to use the bulk insert api

kenj21:06:47

The thing that literally kept me up last night was figuring out the existing syncing was triggering in a separate process before the PG DB transaction committed

😱 4
kenj21:06:20

And also there were concurrent worker processes potentially syncing the same object

kenj21:06:23

Between those the facts, it was possible for an older copy of an object to overwrite a newer version in SF, thus bringing SF subtly out of sync with PG

kenj21:06:09

I miss drinking

p-himik19:05:09

A very particular question, mostly about Linux and partially about Java/Clojure. Is it possible to create a GraalVM-compiled executable from Clojure code that has the setuid bit and: 1. Uses clojure.java.shell/sh to run a third-party program 2. Runs that program in a way that makes it seem like that program itself has the setuid bit I just tried and it doesn't seem to work that way by default, probably a security measure that I'm not aware of. But is it possible to somehow explicitly tell the OS that it's OK in this case?

p-himik19:05:07

Or maybe I have to somehow call setuid from Clojure?

dominicm09:06:08

in terms of running another program, that sounds like dynamic setuid which sounds no fun

p-himik09:06:36

I have come to a conclusion that nothing about setuid sounds fun. :) It appears that even without running other programs, I cannot use setuid on a Graal-VM-compiled binary that easily.

dominicm09:06:09

Really? That surprises me

dominicm09:06:21

The only "restrictions" are on scripts

dominicm09:06:48

And that's because bash executes it, and doesn't have setuid itself

p-himik09:06:57

I have to call the setuid C function from a program that has that bit to make sure that I'm indeed acting on behalf of the other UID.

p-himik09:06:12

So seems like I have to use JNI.