Fork me on GitHub
#clojure-austin
<
2023-03-16
>
staypufd04:03:56

I was sick, and I thought I’d posted here, but anyway, sorry I had to miss. Different note, you emacs users may like this: https://twitter.com/lisperati/status/1636146422658797568?s=46&amp;t=7IBX10LnX9CeGSetNyMqVg https://github.com/drcode/nrepl-gpt

mlimotte13:03:00

Thanks for hosting this past Monday, Cam. Great conversations about Clojure, thanks to everyone who participated. We dug into the limitations of current developer / machine interfaces, with text based CLIs as a main point of frustration for Cameron (sorry if I’m misrepresenting-- see Cam’s copious Soap box notes above for his actual concerns). I have a different, but related frustration, which is the amount of effort it takes to expose some utility function to non-tech co-workers in an organization. For example, let’s say you want to expose a capability to “deactivate a user”. For arguments sake, let’s say that means flipping some field in the database on the user record and archiving some files or whatever. As a developer, I can implement the actual functionality (not interface) quickly with a function that I can run at the REPL. Or maybe a CLI interface. But expecting a typical non-dev to install and then update a CLI tool is not reasonable. So, it took 10 minutes to write up the functionality, but now I have to think about how to expose this capability. I could create a little web interface for it. Or, if I have an existing Internal app, I can add a new widget/section there. Or I can look to use the integration APIs of our third party CRM (Hubspot, Sales Force, whatever). Or I could create an App for Slack (as our “collaboration platform”). All of which take some time and effort. I’d like to be able to just declaratively state that this “function” should be exposed to this user group (or these roles) and be done. This was one of the things I liked, conceptually, about Datomic Ions— you can expose a function just by listing it in the Ions config EDN file and running the deploy. But that only made it accessible as a lambda or REST endpoint or datomic transaction function. None of which are intended for direct use by a non-technical developer-- e.g. no UI. I think, what I want, is: • A generic Interface (a web app, or a Slack interface, or a menu/popup as integration in some other platform) • A one-button deploy (e.g. a CLI command, or something triggered through CI/CD) • And a declarative model to add new features (where you state the minimum information) ◦ The function to run ◦ A doc string ◦ Data types for the fn args, with hints about how to get the value or what UI to show. I think this is the trickiest part. Not all fn args are simple primitives-- and a dumb UI that just has a text box for each arg just won’t cut it. Here’s a first draft ontology of argument types w/ examples: ▪︎ Simple Values • Some args will be enum-like, so we should have a drop-down of values • A date time -> calendar interface • Emails, URLs, etc -> some string format validation • a file picker • etc ▪︎ Resources • database connections • secret keys or other credentials for accessing other systems (e.g. AWS secret key, Hubspot API token, etc) • shared configuration files • a reference to a cache or other shared resource ▪︎ Context values • For example, if I’m on a screen or in a conversation about a specific customer, I should be able to trigger the function with this customer as the implied argument value • The user initiating the action So, I’m just sharing my thoughts here. I think about this every time I have to expose some utility. I’d be interested if anyone else has gone down this rabbit hole — what are your ideas and experiences. Or maybe a discussion topic for another meetup, if there’s interest.

Matthew DiLoreto03:03:48

I have run into this issue before, though I'm usually more focused on the front-end side of things. I think there's two different classes of programs that internal users want: 1. Programs to solve one-off, single-internal-user, specific, transient things. Things that might have decent business impact, but distracting a whole programmer for hours or days to get it implemented would probably not be worth the trouble. As far as I know, there's nothing to help in these scenarios 2. Programs to solve many-time, many-internal-user, recurring system management and reporting problems. These are more appropriately solved by admin dashboards, monitoring/analytics software, Zapier, or the plethora of other existing solutions. Personally I'd like to solve problems in the first camp. I love it when I'm on a call with a non-technical person and I see them doing something clumsy with our product or some internal tool, and I say "let me write a quick UI for you", or "let me automate this". Makes their day, or, depending on the inconvenience it saves them, their week. I've noticed non-technical people can tolerate significant inconvenience, to a degree that shocks technical people, and multiplying their productivity in those cases adds up. I also think that problems in the second domain are quite a bit more complex, judging by the breadth of the potential solutions/required integrations. If I were to address the first class of programs, I think there would be a few problems to solve: 1. Creating an interface 2. Exposing the interface 3. Controlling access to the interface 4. Using the interface If only you could conveniently create, expose, and control access to some interface to your business software within your own organization, and the appropriate people could use them, then the problems would be solved. Creating an interface The level of convenience required here suggests to me that the appropriate infrastructure to host whatever software the non-technical client wants to use is the programmer's computer itself, and for a finite window of time. These are quick utility functions after all. Maybe they can be made more permanent later by some other means after they have proven useful. Maybe SaaS upsell using something like ions, but would need to be more-deeply integrated into the company's stack. The programmer should be able to create the interface inside a REPL for Clojure, but there's no reason this solution would have to be limited to Clojure, as long as an API for other languages can be built/installed on the programmer's machine. Exposing the interface Exposing the function with Ions is a good idea, and I've also used ngrok in the past to expose basic webpages with throw-away apps on them using basic-auth and slack-messaging credentials + the ngrok link, when the non-technical person's request is a transient thing. An admin-panel application has the cumbersome properties of: • requiring permanent infrastructure • requiring some software release process, of varying degrees of inconvenience. • being strongly coupled to the business software An opaque, transient link directly the the programmer's machine using something like ngrok would be the simplest solution, but again, maybe there's upsell potential on a permanent hosted solution based on Ions. Controlling access to the interface Slack is a great option because you can use slack's authorization model to control access. This way a generic solution can be built, ignoring any company-specific authorization complexity (I've been a part of companies with all kinds of weird access rules for different systems). Slack bots are also decently powerful, customizeable, and easily implemented and distributed (to any organization that uses Slack). Using the interface The way I see it, you want to build this system with the smallest possible amount of business-model knowledge as possible, to make the system as generic as possible. When you dive into the problem of providing UIs for customers, it is IMPOSSIBLE to create robust, specific, and exhaustive components (trust me, I've tried doing something similar using JSONschema before and I'm currently doing it with graphql to a limited extent). Imagine the input is a date, do you now want to have a date-picker? What if it's an enum, do you need a UI and some radio buttons? What about some more business-specific domain object, does that need some custom input? Does it need to fetch extra data? How is this all configured? You'll either stick with a limited set of configurations, and your users will complain that it isn't powerful enough, or you will need to essentially recreate or integrate with a frontend framework, and then no one will use it because of the complexity. I believe the problem of auto-generating UI's is intractable, or at least very very difficult. You know what interface every non-technical person at any organization is already familiar with? Spreadsheets. Spreadsheets are great because they can quickly and conveniently model just about any business scenario, and they are ubiquitous. And, unlike most slow, buggy SaaS websites, non-technical people actually like using them. They are also incredibly dynamic, because their only data-type is string. Strings are a good fit for this problem because they put the entire burden of datatype conversion on the programmer, who will be capable of doing it, instead of the user or software system, neither of which will be capable of doing a perfect job. You don't even have to have the user create, save, and upload their spreadsheet either, you can just link them to a Google Sheet pre-filled with the columns of whatever schema the programmer specified when they created the interface. This is also nice because you can leverage Google's authentication/authorization again. Example Imagine the following scenario. Non-Technical User (NTU): 1. Sends a message to Programmer: > "Hey, I need to get all user log-in events between yesterday and today, and if the user mentioned "GPT" in their conversation I want to know their LinkedIn, so I can send them a recruiting message." Programmer: 1. Sends a message to NTU: > "No problem, give me 20mins, you'll get a message from a bot." 2. Creates a function in his repl that does the thing, and submits it to the software system using his language's API, which he has installed locally. NTU: 1. receives this message: > Mark sent you a special interface which he says can: > • take a list of usernames, a start date, and an end date > • return you all the LinkedIn profiles of any of those users if the user mentioned "GPT" in their conversations between the start and end date > > To use this special interface: > 1. fill out this spreadsheet (here)[https://docs.google.com/spreadsheets/d/1J0cIbcbLn94yvPoVBlqyOgPCNqW4Ab3AAdsH8Ar0O10/edit#gid=0] > 2. come back to this conversation and reply: "RUN IT" 2. They complete those steps Programmer: 1. Receives this message: > Lisa wants to submit the data in (this spreadsheet)[https://docs.google.com/spreadsheets/d/1J0cIbcbLn94yvPoVBlqyOgPCNqW4Ab3AAdsH8Ar0O10/edit#gid=0] to your interface. It looks like it's still running on your machine, so review the data and reply: "RUN IT" if it's good to go. 2. He completes those steps. Software System: 1. The spreadsheet data is converted into an object and passed to the programmer's interface on their computer using the generated Ion 2. results are streamed from the programmer's computer into the same spreadsheet under the results column. The programmer can return some specific object from the function to control how results are laid-out in the spreadsheet. Like I said, this kind of system wouldn't be appropriate for tasks which will need to be reused very often and by very-many people, that's what other internal tools are for. This kind of system would be eminently appropriate for small BI-type tasks and system modifications.

mlimotte16:03:37

Cool stuff, @U016VSY8830. Thanks for the the response and thoughts. I do like your first category, as well. And it’s an excellent insight that “non-technical people can tolerate significant inconvenience”. This is very true, and so much so, that even if you are able to dedicate some developer time to solve these problems and ask how you can help, you rarely get actual requests. They’ll often just keep on doing it the manual way, not even recognizing the inefficiency or redundancy. Just assuming, well, “this is the the job”. I’ve found that the majority of things that need to be done once, need to be done again. So I’d lean toward a permanent (but very fast/cheap) infrastructure for deployment. Re the interface, I’d fallback to the old adage, “simple things should be easy, hard things should be possible”. Meaning a library of interaction patterns, but extensible so you can add your own. And, yes, avoid building a whole new UI framework. Again, Slack could be a solution there. It’s an existing Interface that is defined by data structures (i.e. language agnostic). I also love spreadsheets a solution. At Skipp, I’ve built many systems based on Google Sheets for display, reports, inputs, bulk update and loading, triggering actions, collecting user input and so on. Super, super powerful and extensible out of the box. We have extensive libraries for reading sheets, generating new sheets from templates (I did talk about using Instaparse for Google Sheets template library a while ago for the Meetup). But, OTOH, you also have little control over the interface, and they are kind of brittle. For experienced, internal users, this works out. But I do wonder how well this would scale for a larger organization. I like your proposed interface. I think there’s a lot of promise there. There could be ways to extend it some to meet the needs of the more frequent, recurring requests. I don’t think deploying to real infrastructure should be a blocker either (an Ions-like approach is reasonable). Code of this sort should be in a separate category from other production deployments: no code review required, no staging environment, minimal testing. Maybe you also track usage— if it hasn’t been used in 3 months, it’s auto-undeployed by the management system, etc.

mlimotte16:03:46

Thanks again for the comments. Good discussion.

Matthew DiLoreto22:03:53

Good point about often needing to reuse these functions, I'm just not as familiar with the infrastructure side of things, so it seems like more of a headache to set up. My main concerns would be: 1. How would the software be integrated into production systems in a business-domain agnostic way? I like Datadog's approach to this, where you deploy a Datadog service next to your other services at the infrastructure level and you also have an API for whatever language you're using which you hook into all your frontend and backend services and applications separately. 2. How can you control the access the software system has so that it isn't just an arbitrary code executor listening inside your infrastructure at all times? 3. How does the programmer write and verify their interface on their local machine, if the ultimate target is a different production environment? They would have to have a local environment mirroring the production one, but that isn't always feasible.

mlimotte15:03:45

1. That sounds about right. That’s also similar to the Ions model-- a deployed, generic platform that is always running and you can the push code blocks (functions) into it with some standard tooling.

mlimotte15:03:50

2. You’d have to choose an auth model which would be front end interface specific. For example, Slack has an existing authentication model. Then you’d pass along those credentials on a secure channel to the platform. Secure channel could mean it’s in a protected VPC, or you’ve established a secure connection with certificates or provided an api-key. Whatever it is, you’d only have to set it up once. 3. Maybe, again, some inspiration from Ions. Where the developer is just creating a function. And they can run and test the function from the REPL. Most functions are business logic and should be able to do the same things locally and deployed. The only thing you’d have to test “on-platform” (again only once) would be access to protected resources (database connections, an S3 bucket or SQS queue, etc).

mlimotte15:03:28

For sure… there’s a lot of work to be done to create a generic platform like that with language hooks. But it could be a nice product that fits into the “low-code” market. There’s code here, but only for the business logic.

mlimotte15:03:14

It’s long been my contention that the driver for no-code/low-code isn’t so much the “code” part. There are many, many traiing programs that will teach anyone basic coding skills in a few weeks in python or whatever. Rather the hard part is architecture and systems development. None of that is necessary if you’re just deploying single functions onto a generic platform.

mlimotte15:03:45

Nevertheless, I’d still target an initial offering toward developers (not the business users).