An analysis on if EA is Talent constrained

STM feedback

Machan,

I would pause immediately at the term “talent-constrained”. I don’t understand it. The first step is to describe the claim “X is talent-constrained” in terms of familiar claims so that you can actually test the claim against evidence. Do you mean that there is a low rejection rate for PhD applicants?

A related issue is that, for the same claim, you’re switching between evidence in the form of surveys and evidence in the form of rejection rates. If the claim is about rejection rates, then you either have the numbers or you don’t. If you don’t, you can’t test that claim against a concrete example. If the claim is about surveys, then you’ll have to use the survey. Right now, you’re taking the same claim and mixing multiple kinds of evidence, such as surveys and 80k opinion posts and rejection rates, which left me at the end with no clear answer.

So, split the different kinds of claims: “EA has high rejection rates” and “EA surveys have high percentages saying the words ‘we are talent-constrained’”. You can even make the claims more precise: “Operations manager roles have high rejection rates for candidates with 2 years of similar experience” - notice how you can immediately test that claim given concrete examples.

Contrast that to “In brief, we think our list of top problems … are mainly constrained by research insights”. How do you test that given some data? Imagine if they’d said “Salaries went up by 20% last year but number of open questions solved in published papers went down by 40%”. We can debate whether “number of open questions solved” has been a useful metric but there’s no question that we can test that claim against evidence.

On a different note, a key point that is missing from the analysis, here and elsewhere, is that we talk about a shortage for a given price. We don’t say that there is no supply of onions. We say there is no supply of onions at Rs 10/kg. When the price has eventually risen due to lack of supply, people have even transported onions from other countries to supply them for a profit.

Saying that there is a lack of “talent” or researchers for a given role doesn’t make much sense unless you talk about the current salary. But people are talking as though, no matter the salary, there is not enough talent in the world to do this research. People have in the past moved from country to country and from job to job for higher pay (and other desired characteristics like climate and family members). There are a lot of well-published PhDs and postdocs working on all kinds of other research areas for much less than six-figures and a lot of professors and researchers working for not too much more. Is the claim that they won’t switch for a 2x salary or that they can’t study and catch up on the slightly different field in a few years? If the EA organizations were “desperate” for a particular kind of researcher, did they raise the salary a lot? If they didn’t have enough funds to raise salaries, then aren’t they… “funding-constrained”? Are the two “constraints” actually distinct?

Mission #6: For now, I recommend rewriting the post after splitting the claims till you have narrow claims that are either tested with examples or don’t have any available examples. Check if you see any lingering confusion or ambiguity at the end. One week should be enough time.

Abbreviations

RP- Rethink Priorities
RC- Rethink Charity
CE- Charity Entrepreneurship
CS- Charity Science
TLYCS- The Life You Can Save
EAF- Effective Altruism Fund
FWI- Fish Welfare Fund
PH- Peter Hurford
GH&P- Global Health and Poverty

Entry question

I would like to know if EA Orgs are able to hire enough people with specific skill-sets (e.g., researcher with 2 years of experience) for a given pay (e.g., 70k$). I would like to know if EA Orgs have a “lot” of “good candidates” to choose from. I would like to know if funding is stopping them from hiring people.

Introduction

In the following discussion, I have not used “for a given pay” as I don’t have any examples for them. Similarly, certain positions such as director of operations or fundraising people are not considered as I don’t have evidence for it.

I started off with writing out the claims and examples, checked if it matched the definition, and reiterated a few times before I came to this post in this format.

With the following post I look into the following aspects:

1. If EA Orgs are able to find and hire skilled people that they want.

2. If there are “many” “good” candidates.

3. If there are EA Orgs unable to hire due to lack of funding.

Are EA Orgs able to find people?

Finding people

EA Orgs seem to be able to hire the number of people they want, at the end of the hiring round be it in research or operations. OPP wanted to hire 5 GRs in 2018 and succeeded in it. EAF set out to hire 1 GR and 1 Operations personnel and managed to do that. FWI also managed to get a researcher and “welfare specialist”.

RP, RC, CS and TLYCS seem to suggest that they have no problem finding people (researchers I think based on their recent hires).

I’ve certainly had no problem finding junior staff for Rethink Priorities, Rethink Charity, or Charity Science (Note: Rethink Priorities is part of Rethink Charity but both are entirely separate from Charity Science)… and so far we’ve been lucky enough to have enough strong senior staff applications that we’re still finding ourselves turning down really strong applicants we would otherwise really love to hire. — Peter Hurford

However, EA orgs don’t seem to be able to hire the number of Entrepreneurs they want, at the end of a hiring round. CE in 2019, during its incubation program was only able to accept 17 people despite having ~20 positions. Apparently (in an email to me), they would have hired more if there were better candidates. However, this year they have more than 10 times the applications of last year for the same number of positions and CE thinks that they would probably find more people than they have seed grants for.

Note: I don’t have more info on other hiring rounds and so stick with the above for most of the rest of the discussion as well.

It is unclear if the type of organization (Longterm, GH&P, Meta, Animal EA) relates to finding a hire or not. OPP (Meta), TLYCS, RC, RP, CS (GH&P), FWI (Animal EA) are able to find hires. But it is not clear what are the struggles of Orgs such as MIRI or FHI in finding people (Longtermism) as I don’t have those examples.

Quality of hires

EA orgs are able to hire Researchers (in the last 2 years), either with greater than 3 years of experience related to EA, or people who studied in top universities (based on rankings) in UK and US, or PhD’s in Philosophy or Political sciences or Economics or Math or relevant (e.g., PhD in Marine life for FWI). A look at the recent hiring rounds seems to agree with the claim above. Saulius, however seems to be an exception. But he seems to have done other things (such as to write and criticize EA research on his own) to grab the eyes of recruiters who requested him to apply for the vacancy.

* internships

** did volunteering work excessively and has other research experience in university related to the field of work.

People: Olivia, Grace, Alicia, Marinella, Peter, Jacob, Joseph, Daniel, Saulius, Neil, Marco, Jennifer

Whether you are a new org or an old org you are able to hire Researchers (in the last 2 years), either with greater than 3 years of experience related to EA, or people who studied in top universities in UK and US or PhD’s in Philosophy or Political sciences or Economics or Math or relevant (PhD in Marine life for FWI). RP and FWI are orgs founded in or after 2018, GiveWell in 2007 and OPP in 2014. The same table above is used as evidence.

EA Orgs seem happy with their recent hires. It does not look like they had to settle for someone with “lesser experience”, when they wanted someone with “more experience”. OPP said they got more than 100s of “very good resumes” for the GR positions. In the end they thought multiple people from the pool who didn’t make it, would exceed at OPP in the future.

Peter of RP said,

Based at least on my recent hiring for Rethink Priorities, I can definitely confirm this is true, at least for us. We ended up completely overwhelmed with high-quality applicants beyond our wildest dreams. As a result we’re dramatically scaling up as fast as we can to hire as many great applicants as we can responsibly, taking on a bunch of risk to do so. Even with all of that additional effort, we still had to reject numerous high-quality candidates that we would’ve otherwise loved to work with, if only we had more funding / management capacity / could grow the team even faster without overwhelming everyone.

TLCYS suggests that he is confident and able to find “high-quality” candidates. CE in an email said that they have an absolute bar and wouldn’t take anyone if they didn’t pass that bar. As far as they hired they seem to be happy with the level of the hires.

Was that too “vague”?

How many people

Number of “good quality” people

The acceptance rates of EA Orgs are similar to top Universities like Harvard and the best startup incubator Y-combinator. The following are the acceptance rates for EA Orgs (OPP, EAF, CE, FWI) followed by Top universities (<rank 10) and Y-combinator, and followed by moderate universities (>= rank 20).

Looking at the acceptance rates it appears that EA Orgs are hard to get in just like the top Universities. I am unsure anything related to the number of “good quality” people can be derived from here.

New Orgs seem to have similar visibility as older Orgs (number of people applying per position). In FWI’s (founded in 2019) hiring round for researcher there were roughly 40 apps for 1 vacancy. In OPP’s (founded in 2014) 2018 hiring round for GR, there were more than 20 apps for 1 vacancy.

EA Orgs seem to think that there are more candidates fit for the job than the ones they hired in research. OPP said they got more than 100s of good resumes for the GR positions. In the end they thought multiple people from the pool who didn’t make it, would exceed at OPP in the future. TLCYS suggests that he is confident and able to find high-quality candidates. Peter of RP said,

Based at least on my recent hiring for Rethink Priorities, I can definitely confirm this is true, at least for us. We ended up completely overwhelmed with high-quality applicants beyond our wildest dreams. As a result we’re dramatically scaling up as fast as we can to hire as many great applicants as we can responsibly, taking on a bunch of risk to do so. Even with all of that additional effort, we still had to reject numerous high-quality candidates that we would’ve otherwise loved to work with, if only we had more funding / management capacity / could grow the team even faster without overwhelming everyone.

EA orgs seem to think that there are NOT many candidates fit for the job than the ones they hired in Entrepreneurship. CE had ~20 positions and were able to fill only 17 in 2019. But in 2020 with more than 10 times the applications as in 2019, it is probable that they have more “acceptable” candidates than the people they hire.

I am unable to say more about the type (PhD’s, yoe) and the number of people who didn’t manage a job because of reasons such as, the orgs didn’t have money to hire more, or the orgs were management constrained etc. I don’t have any data or even one example regarding the unhired people who were “good enough” to be hired.

I assume for now (for lack of other better alternatives) that the number of candidates in the last round gives an indication of how many were fit for a job. Most of the final rounds had 2x the number of people hired, for research positions. (And orgs had about 5 rounds including initial application.)

OPP had 17 people in the trial round while hiring for 5 people. EAF had 4 people in the trial round while hiring 2 people. FWI had 4 people in the last round, which was a “reference check and call with finalists”, while hiring 2 people in the end.

The final rounds in orgs hiring for Entrepreneurship, don’t seem to have 2x the number of people hired. CE in 2019 had 27 people in the last round compared to 20 people they wanted hire != 2x. It could be different in 2020 where they have 10 times the applications as last time for the same number of positions.

Jobs in govt/academia in AI Policy

80000hours: “Senior staff within governments and top AI labs tell us that they are struggling to find experienced and qualified AI talent to employ”. I have 0 examples to understand this. I have no idea what each of the words refer to as I don’t have an example. I don’t know what “struggling”, “experienced”, “qualified AI Talent”, “top AI labs”, “senior staff within governments”, means. How many jobs are there? how many jobs were they not able to fill? Did they end up hiring people withe lesser experience? I don’t know.

In the case for working in AI policy, 80000hours seems to suggest that there could be many jobs in this field. But, there doesn’t seem to be 100s of jobs in Govt./Academia (>20) in AI Policy in one year. A look at the 80000hours job board seems to identify only 10 positions including internships, in companies, non-profits, academia and Govt over a 3 month period. This translates to 40 per year at max, including companies non-profits, academia and Govt.

80000hours also suggest that in the future we will need a “lot of people”. It is not possible to test that. But it shall be noted that LOT means atleast 20 senior people with the right background working in the 20 or so US Govt. agencies that will be involved.

There seems to be an underlying assumption that AEAs will have more impact than the people who take up these “senior roles” or other roles in academia and govt. I don’t have any example for this either.

Funding: Short on cash to hire people

Meta Orgs don’t seem to be affected by funding while considering hiring researchers. OPP said, “our current ability to immediately assess and deploy this base of available talent is weak”.

It is not clear if hiring is affected by funding in Longtermism Orgs. There is not info I can find regarding this on MIRI for example.

There are orgs working on GP&H, seem to be willing to hire more research candidates but are short on cash. TLYCS’s Jon Behar says that here,

TLYCS’s experience is very consistent with Peter’s. Money is overwhelmingly the constraining factor, with more funding we’re confident we can get high quality candidates.

PH representing RP, RC, CS says,

I personally feel much more funding constrained / management capacity constrained / team culture “don’t grow too quickly” constrained than I feel “I need more talented applicants” constrained.

However, orgs in GH&P hiring for entrepreneurship don’t seem to be short on cash. CE (via mail) says that, they were not having any issues with money, but suggest that this year (2020) the situation might be different due to the 10 times more applications.

Whether you are a new or an old org you seem to want money to get more hires. RP is a new org (founded in 2018) and would like money to get more hires, but so does TLYCS (founded in 2010) and the whole RC (2013) CS franchise according to Peter which were founded many years ago.

Fundraisers

Some Longtermism Orgs are unable to meet their fundraising targets as small as 1m$.

MIRI in 2017 managed to raise all of the 2.5m. MIRI in 2018 was short by 250k (trying to raise 1.2m). In 2019 they were short by 400k (trying to raise 1m). The severity of this is expressed here:

“Given our $6.8M budget for 2020, and the cash we currently have on hand, raising $1M in this fundraiser will put us in a great position for 2020. Hitting $1M positions us with cash reserves of 1.25–1.5 years going into 2020, which is exactly where we want to be to support ongoing hiring efforts and to provide the confidence we need to make and stand behind our salary and other financial commitments.

I don’t know the scene with GH&P or Meta Orgs regarding fundraising. I don’t see any fundraising efforts by GiveWell, RC, RP, CS or OPP for example. GiveWell and OPP seem to be covered for the most part by Good Ventures and non-restrictive donations.

Reflection

Disentanglement research redacted

I re-read the disentanglement research (DR) post by Carrick Flynn. It seemed to contain opinions on whether DR is “bottlenecked” across whole of EA. What I am interested in are things such as if FHI is able to find people for their vacancies or even a statement such as “you clear the bar you WILL get the job at FHI and that they are always hiring for DR researchers”. But I didn’t see anything like that in Carrick’s post. Hence it was decided to not use that content.

No more holistic decisions on claims

I forget to check the example against the definition correctly. In the last post, I mainly looked only at one claim. The claim was that, “EA Orgs are able to find people with specific skills at the end of the hiring round”. For this what I needed was to check hiring rounds and if they hired those people with specific skills that they wanted.

But what I did implicitly is look at the rejection rates, number of people who applied, comments on the how good the talent pool was, how many people made it to the work trials etc… And somehow “holistically” got the impression that EA Orgs are able to find the people they set out to need.

This doesn’t seem to be a proper way to go about claims. For example, I seem to have factored in rejection rates, but I had no idea of what rejection rate was high or low. I looked at how many people made it to the work trials, and figured that was somehow “good enough”. I should have been explicit about all these proxies and should have tested them individually.

Narrow the claims and feel better

When people say “TC” now, it hurts. I realize I don’t know what they are talking about anymore. Instantaneously I feel confused. There are atleast 5 definitions that I know of¹. Jesus! Pick one at a time and feel better.

Split into familiar claims and question people

People asked me a lot of questions and added further claims such as “You can’t judge TC based on rejection rates”, “Orgs probably hired ‘lesser capable people’ and hence said they were TC”, “new Orgs have problem with funding, what you have is an anomaly”, “there are many jobs in the govt for AI Policy, surely it is TC” and “What more can we do, we already asked the Orgs if they are TC”.

I don’t feel tensed and confused about how to deal with these claims anymore. I know that I can split the claims into narrow claims and have a discussion as opposed to just shrinking my face in confusion and not knowing how to proceed.

P.S.
I answered “Yes, I agree we can’t do more”, to “What more can we do, we already asked the Orgs if they are TC”. Damn it! Split into familiar claims.

Don’t just write, think in claims and examples

I tried writing the previous paragraphs on “reflection” and the main post, and seemed to be ending up with a lot of words, feeling confused (head hurts) not knowing where this was going. Then I changed to “what claim am I trying to make”, followed by an example. And it seems to have gone “smoother”. Less “confused”, more “organized” types.

Too soon to judge

I am too soon to judge an example as satisfying the claim. When I heard an example that DoD JAIC was looking for 100s of people, for the claim: “EA needs people in AI Policy”, I was like wow! This is a great example, showing that there are many jobs in the govt, for EA people. The claim is true I thought immediately. I even called it a beautiful example. But when I spent time on understanding it, it took me more than 3 hrs.

First, the assumption I made was that people who identify as EA somehow make “better impact” at such positions than people who don’t (for which I don’t have any examples of). Additionally, Dod JAIC was looking for all sorts of people, not just AI Policy people. 2 years since they seem to have hired 40 people in total (as per LinkedIn) and a conservative estimate suggests that there would be 10 people in policy across all seniorities. So 5 people per year have been hired. Not 100s of jobs in JAIC, just 5 per year.

Re-write a fresh

Send distractions to the footnote. Don’t force yourself to keep them and weave them in the post. You are out here to find the truth and not weave all the research you have done into a flowing story.

Re-writing in a fresh sheet and re-factoring old stuff seems to have helped in this regard. I got rid of many things I was clinging on to and struggling to weave into the story (such as a section on “why I think EA is TC”, A section on ‘definitions’ etc.). It was taking so much time to make it all fit into a story, until I removed those sections completely.

Time spent

Total time: 48 hrs (Over 2 weeks) + 9 hrs (another edit after sending it to an STM)
Total words: 4700

Number of days: 13
Avg hrs per day: 3.6hrs (including weekends and 2 full days)

Footnotes