Clojurians Log v2

@seancorfield thanks a ton for your follow-up on this, I greatly appreciate it. I'm having a hard time finding where exactly your reply was about the top 2 candidates in your last "hiring sprint" but learning one was twoard the middle and one toward the end illuminates things for me

TIL: The secretary problem. I think one time my colleague @firesofmay and I unwittingly bypassed a key constraint of the secretary problem: > Once rejected, an applicant cannot be recalled. Tl;Dr: We not only explicitly kept the door open for re-applications, after a cooling-off period. We also offered to help people learn stuff we were interested in hiring for, on an opt-in basis. We would send people curriculum and make ourselves available for office hours on a fixed weekly schedule, to those who showed interest in learning. Over about 2013-14, him and I were hiring for QA engineers at a Clojure shop. We wanted curiosity-driven and technically proficient people whom we could teach and train, as we were writing test tools and suites in Clojure. We knew QA-programmer combo is a tough ask in the first place, especially in India, where most of the QA pool is manual-only testers from IT services companies. Sadly, QA is also very low status here (which blows my mind; rant for later), so "technical" people run away from the label. So we had braced ourselves for a noisy pipeline. In this context, I figured it would be crazy to lose anyone with potential. In a past life as a business manager, I had seen companies lose candidates with exactly the right life experiences, but the wrong degree or pedigree or test score. So I spitballed the idea to offer reapplication + opt-in support. My colleague loved the idea, and so it was. We didn't have any HR function, so we were left to our own devices. Luckily, the company culture was also oriented toward "nontraditional" hiring, looking for diamonds in the rough, so to speak. We ended up hiring 4 people directly, out of ~300 inbound. Our retention rate was pretty decent. Each person grew way beyond our expectations, and became tenured staff (> 4 years), as far as high-growth startups go. (Most of the credit for career growth and retention goes to the engineering culture built by all of m'colleagues. Hiring people with potential is critical, but it's only the beginning.) We also indirectly hired at least one kickass engineer (also became tenured), because of the good word-of-mouth our approach earned (which we did not anticipate, but it happened). He was blown away by the learning support we offered his wife whom we had screened but didn't end up hiring. With 20/20 hindsight, I wonder if our approach would run into some HR or legal quagmire (Like, would it creat an implied commitment to hire if <criteria> are met, and if we don't then would we run afoul of some arcane labour law?). Anyway, we just did it because we could, and I'm glad we took our chance. I wish this was the default way to hire.

Wow.

Wow. Wow indeed... > I wish this was the default way to hire. And it's personal this time because https://clojurians.slack.com/archives/C050UG324GM/p1721227349731849. I'm rapidly getting re-acquainted with the exquisite agony of the receiving end of the phenomenon I now have a name for... Which may be a double-whammy. The secretary problem only manifests, if you get passed through by a (mechanical) problem secretary (the dreaded ATS).

Bear in mind you get different probabilities for the secretary problem if you think about the distribution(s) of candidate quality instead of just assuming they’re linearly rankable.

Does it still remain a secretary problem though? If I were to bet, I'd say a global talent pool's quality would fall under a power law curve. Likely within an applicant pool too, for a given role.

You can have different channels as well. My experience is applicants from Hacker News are higher quality than most recruiters, who are higher quality than most bootcamps, for example. But yeah the problem statement is the same - every time you see a candidate who is better than all previous candidates, what’s the probability you should stop? Obviously you could also go crazy and model the opportunity costs of not hiring early or of spending lots of time doing interviews etc too. Not sure how applicable this is to most vacancies at small companies though.

Yeah, it's hard to know in advance what our n should be (assumed size of total applicant pool), and whether the applications received so far are the long tail (weakest) or the steeple (the best) of the assumed pool.

Stats isn't my strong suit, but I feel the main problem of normal conventional hiring practice seems to be that, we play it like a zero sum game even though it is an adverserial game. If everyone uses the same formula (e.g. we have 4 (or 5) rounds and dish out permanent rejections because the best of us (read: FAANGs) do this), then SmallCo. will always lose to GoliathCo. because SmallCo. simply cannot afford to take the beating of a 1:100 funnel (1 offer for every 100 applicants). FAANGs will win simply because it becomes a game of attrition (figuratively, and literally) if we play by their rules, which, surprise, we do.

every time you see a candidate who is better than all previous candidates, what’s the probability you should stop?Yes, I can see how optimal stopping still applies... no idea how one might guess a fixed point for the conditional probability decide the point where we are unlikely to find a remarkably better candidate than the already-seen ones, in a given hiring pipeline.

@adityaathalye i think long-tail simply means delayed in timing, as in the distribution of frequency of applications coming in, not necessarily applicant quality -- right? the "long tail" could have some surprising victories, which is sorta the question I think there is nuance to the optimality, we don't need the best, we need suitable [enough] for the role. Sometimes there may be many suitable candidates, sometimes one, sometimes none. Suitability for the role is easily determined when all you need to do is type quickly, but if you have to be creative, and also name things, then suitability varies not only on candidate, but on context into which the candidate is thrust.

I meant it in the sense that a small percentage of the talent pool is good-to-god-level quality, and a large percentage is not. Some discrete scale of "quality" on y-axis and count of candidates on x-axis ... forcing a discrete scale would force us to bucket candidates. So out of pool of say 1,000, one would be "A+", maybe 10 would be "A", 100 "B", and so forth... Cs, Ds, and Fs. Add to that the timing problem you highlight... We don't know how long we will have to wait to receive most of the B-and-aboves. Whether we will receive any As at all. And would we want to hire the A+ person who may be difficult to retain.

I guess I described stack-ranking as practiced in several companies (people graded to company-specific internal criteria).

And yes, I agree that "optimal" is nuanced and highly context-sensitive.

When I mentioned the tail, I just meant in the timing distribution of applications coming in: you tend to get a big batch when the job is first posted and then that trails off and after a few weeks you are just getting a trickle. Depends on where you post, of course, and how "attractive" a company you are, but that's pretty consistently been our experience.

Depends on the company, the job, and the tech stack. When we've hired for JS devs, we got close to 1,000 applications which HR narrowed down to about 100 resumes for IT to look at which came down to maybe 5 candidates to interview of which maybe 1 was decent enough to consider for the next round of interviews.

When we've hired for Clojure jobs, we get far fewer applications but the quality is much higher, so we typically end up with maybe 10 candidates to interview and 3-4 of them get through to the next round.

And yes, each job opening is the same old process 😞

https://en.wikipedia.org/wiki/Secretary_problem

Yup, if you make your final hiring decision while applications are still coming in, you might miss out on a better candidate. It's a trade off between how long you want to drag out the interview/hiring process and how much of the long-tail of the applicant pipeline you really want to look at.

Yeah the 37% rule! Yes! I saw that a long time ago on slashdot. Never knew it was called the secretary optimal stopping theory

@seancorfield out of curiosity, in the case of 1 developer of 1,000 making the cut, when in that 1,000 did they appear?

@v1nc3ntpull1ng I don't have complete visibility into the entire funnel (the 1,000 applications that came into HR) but of the 100-ish that made it onto the Jira board for IT to review the top candidate was about halfway through and the second best candidate was close to the end. That's from our most recent hiring attempt. I don't remember where in the funnel the best candidates of the previous round were a year or two prior (but the overall numbers were pretty similar).

Oh, that would violate beautifully and dramatically the secretary theorem. Thanks for sharing your experience.

The secretary problem assumes you make a final decision after each candidate, in a single round of interviews. We have two rounds of resume screening, and then three rounds of (short) interviews so we inherently continue screening and interviewing beyond the first "best" candidate at each stage.

Yeah. Makes sense, secretary more of a fungible role if weird HR speak. Uh, just curious, do you find that the candidates you want are still available even after a long period of uncertainty in hiring?

Our process is pretty fast so, yes. I've been a hiring manager for over 30 years at this point so I can blast through a mountain of resume screening pretty quickly, and I've honed my interview process so I rarely need more than 30 minutes to decide whether a candidate will work for me (sometimes a "no" decision can be made in a lot shorter time) -- so when we're hiring, we can fit the whole process into just a few weeks at most if need be. Applications tend to come in a big burst shortly after posting the job, but we can still fast-track any stragglers that apply a few weeks later.

We don't do "technical" interviews insofar as we don't do coding tests of any kind, nor take home projects. Nor do we ask any trick questions. I know those things don't work 🙂

Cool.

Long tail has many interpretations is also something I learned from this thread.

Yeah, it's just a Y v/s X distribution and it so happens that many natural and human-caused phenomena fall under that curve.

Sean's "long tail" distribution is about queuing / arrival rates, and mine is segmentation.

My question still remains, however. Do you need all the applicants? Clearly not, because not all the applicants get jobs. Because from the employer point of view time is burnable but from the job seeker point of view, it's a huge waste of time to apply to a job and not get it. In the optimal sense, how to minimize burned and lost time on behalf of applicants who have an incentive to find employment quickly, versus the company, that has an incentive to mulch through all available question marks at a steady and not-fast rate of movement. Yeah, I guess "who has the leverage" determines a lot, but I find it a bit uncouth and lacking culture in cases where thousands of applicants are encouraged to apply for singular digit positional availability. That's just wasting lots of time, unapologetically. And the result is not markedly improved by adding more people, as you say, it's not correlated to absolute number, but to tolerance window of waiting for applicants. I don't even know what that would highlight about the distribution of talent and finding suitable context to be productive, other than there's a lot of predictable bad moves because bad moves are cheap to make

I would much prefer to get far fewer applicants but to have better quality. But that's what happens with mainstream tech:slightly_frowning_face: I always tell folks to apply for fewer jobs, not more. Pick jobs that you genuinely feel are interesting. Don't apply to everything: it's a waste of your time. If someone is applying to literally hundreds of jobs, I really don't have much sympathy. Those hundreds of jobs cannot all be good matches for you. And if there are fewer applicants, it also makes the employers' jobs easier.

@seancorfield Would you mind explaining why those things don't work for you? > We don't do "technical" interviews insofar as we don't do coding tests of any kind, nor take home projects. Nor do we ask any trick questions. I know those things don't work 🙂

The basic answer is that none of those things tell you how a developer is going to be able to solve problems in the team environment at your company.

Most "technical" interviews are done in a format that bears zero relation to how developers work, period. Take home projects assume a candidate has an arbitrary amount of "free time" to do a non-trivial project, and also assumes a candidate has a work-capable computer at home that is either set up for the technology of the project already, or that the candidate is willing to spend even more "free time" setting up. Take home projects are both implicit discrimination and unpaid labor.