Fork me on GitHub
#data-science
<
2023-01-16
>
Mateusz Mazurczak11:01:12

Hi, regarding machine learning (image recognition), do you think community is big enough for hiring devs to work in clojure for ML or it's better to stick to python?

otfrom11:01:38

I think if you ask for ML people here or on zulip you'll find clojurians who are interested. We often cross train people from R or Python into clojure, and still use R and python (via interop) for some things.

Rupert (All Street)12:01:34

I imagine there are many strong AI/data scientists in the Clojure community to try and recruit. Issue with Python are: Code is usually OO rather than functional, it is often very slow (which means you spend more on servers) and doesn't have access to the JVM ecosystem (which is great for building "big data"/enterprise systems). An option is also to use hybrid approach - Clojure where possible but also use python libraries when needed (via lib-pythonclj or microservices).

✔️ 2
chrisn13:01:11

image rec also has the nice property that shelling out to python, as long as you are batching your inferences, has about the same perf as running it in process. The question is a bit confusing though - are you looking for existing Clojure devs with imagerec experience or are you looking for Clojure devs and then expecting them to do some inference with preexisting models? My opinion is that if you find Clojure devs then getting them to do image rec efficiently isn't a huge jump - the tools are there either way.

👍 2
Mateusz Mazurczak13:01:46

We are starting from 0 regarding ML and will need one person for that part, so I will be looking for one person that knows AI. The rest of the platform is built in clojure. That's why I wondered if we should look here in community. But I wonder if that's sustainable to build that in clojure if the team will grow in future.

Carsten Behring13:01:09

When I look at NLP (in which python is similar dominant then in image processing), the pieces which Clojure (or Java) cannot be used are tiny (by lines-of-code) I ones said, it boils down to one line for which you need python, namely "train( ...)" I would as well suggest to look at using Python for the parts where it is needed and orchestrate this via Clojure. Using python from Clojure can be done in at least three ways: • shelling out and start a python process • libpython-clj • using DVC (http://dvc.org)

Rupert (All Street)14:01:49

I work at an AI startup that primarily uses Clojure - there are certainly Clojure AI programmers out there. As others have said, you can always train non-clojure programmers into Clojure if needed too.

aaelony06:01:28

Probably depends on the application. e.g. not driverless cars. That said, I'm old enough to remember things like https://github.com/aria42/infer so there is a long tradition of clojure and ML

Santiago12:01:19

@U0281QDFE1X I do machine learning models in R (fraud detection, not image rec, but it's irrelevant) and do the deployment and serving in clojure. I'm certainly not alone in understanding ML and engineering and using Clojure for the latter as others pointed out here.

Mateusz Mazurczak12:01:33

That's great, thank you all for the responses. It seems like there is a good market for which we can create good opportunities!

noprompt20:01:16

> I wonder if that’s sustainable to build that in clojure if the team will grow in future IMO you shouldn’t build a team around or hire programmers who have their identity tied to a specific language (unless thats really what you want to do). Instead, the emphasis should be on building around creativity and general problem solving skills.

noprompt20:01:45

I’ve been learning ML/AI since August of last year and most of that has been in Python. I’m fine with that. Clojure is great and all but, honestly, Python is too (VS Code tooling + Copilot is awesome). FP? OOP? I’m happy to use either and Python, like Clojure, has pretty good support for both styles or a mix of the two.

Mateusz Mazurczak21:01:10

Yeah sure, that is my attitude to get just great people. But this is also a factor to consider if you would be able to hire people to work in your project or it will become a bottleneck due to used technology.

👍 2
Rupert (All Street)21:01:14

Unlike Clojure, other languages have gotten pretty complex. I wouldn't want a team of noobs building a Scala (with cats library) web application or an Objective C IOS app in a week - even if they have good problem solving skills. I agree that it's nice to take polyglot programmers who can jump into any language - but I think ideally you should have at least one language/stack expert on the team to avoid wasting time or re-inventing the wheel.

noprompt21:01:51

Expertise is definitely useful with respect to the bad parts of languages, tooling, etc. That said, after working in software for 15 years, I think wasting time and re-inventing wheels are both unavoidable and not the biggest problems (I’ve spent 10 of those years in Clojure BTW). Lack of knowledge and technique are the biggest problems. While I agree that Clojure syntax isn’t complex, in practice, thanks to macros and the variety of programming styles out there, Clojure can be as, if not more, complex than other languages; it can be difficult for people read and understand than equivalent programs in other languages, even amongst experts.

👍 4
Rupert (All Street)21:01:00

I guess there are a number of different challenges (language, tools, libraries, business domain, bureaucracy, problem solving, niche knowledge, planning, momentum, focus, luck etc) that are different for each team. Some are easier to predict and avoid (lack of knowledge of tool/language is one of those).

👍 2