A naive analysis on if EA is Talent constrained

Status: This is an essay where I horribly failed to think “concretely”. What Talent-constrained meant was all over the place. I should have re-written each claim to make it more testable.

I am deeply grateful to Aaron Gertler who reviewed this article for over 3hrs atleast (I don’t know the upper bound). His comments were very thorough (he didn’t leave any hyperlink unclicked). He managed to question several of my claims. I have updated parts of this post based on his comments. I also want to thank STM, Carrick Flynn, Peter Hurford, EA Applicant, Jon Behar, Ben West, 80,000 hours for helping me directly or indirectly. They provided valuable info in their posts/comments/email which I have used in this article. That said, again, they should not be viewed as endorsing anything in this. All mistakes are mine. All views are mine.

Introduction

I have been using 80,000 Hours (80k) since 2017 and have read almost all their posts, spent weeks after weeks reading them to figure out what I should be doing in life. They seem to have done a ton of research and put out many many posts, for us readers to benefit from.

In the process they have made a lot of claims which are hard and time-consuming to verify as we don’t have the insights, contacts or the data that 80k is exposed to. For example they estimate “an additional person working on the most effective issues will have over 100 times as much impact as an additional person working on a typical issue”. To verify this with one example, I would need estimates from say Open Phil on the impact of an employee. I tried, but they are unable to put effort into it at the moment.

Maybe 80k can be asked for clarification directly? Unfortunately, 80k doesn’t seem approachable other than through coaching¹ (which is only for the stellar). Comments sections seem to be deserted to ask for help, and at the time, I didn’t know of any other sources doing this sort of research and coaching for people². Based on reading 80k for years I formed the impression as shared by fellow EA applicant:

Hey you! You know, all these ideas that you had about making the world a better place, like working for Doctors without Borders? They probably aren’t that great. The long-term future is what matters. And that is not funding constrained, so earning to give is kind of off the table as well. But the good news is, we really, really need people working on these things. We are so talent constrained…— EA applicant in the EA Forum

And looking at the 277 karma this post got (the highest of any post on the Forum), it might appear that a “lot of people” share(d) this sentiment that EA orgs could potentially be seriously Talent Constrained (TC).

A few weeks back I stumbled upon some articles in the EA forum and to my surprise it appeared that some EA orgs were suggesting that they were not TC. Until this point I don’t think it occurred to me that 80k’ claims (“EA is TC”) could be wrong or lost in translation or that I should test it. Nevertheless, having seen orgs say otherwise, it felt like a good idea to dig into it at least now.

The following article is my naive investigation on if EA is TC. Before we start going deep into whether EA is TC or not, we must first state the definition clearly.

Definitions

We are going to primarily deal with the term “Talent Constrained” (TC). 80k defines TC in “Why you should work on Talent gaps” (Nov 2015) as,

For some causes, additional money can buy substantial progress. In others, the key bottleneck is finding people with a specific skill set. This second set of causes are more “talent constrained” than “funding constrained”; we say they have a “talent gap”.

So, a cause is TC if finding people with a specific skill set, proves to be difficult. The difficulty I assume is in the lack of those skilled people, and not some process/management constraint³. “EA Concepts”, clears this confusion up with a better worded “example”:

Organization A: Has annual funding of $5m, so can fund more staff, and has been actively hiring for a year, but has been unable to find anyone suitable… Organization A is more talent constrained than funding constrained…

In this post, discussions are focused on Orgs that are TC and not Causes that are TC. When I read that AI strategy is TC with the lack of “Disentanglement Research” (DR), I don’t know what to do about it. But if I know FHI and many other orgs are TC in DR, then I could potentially upskill in DR, and close the talent gap. So looking at causes for me, is less helpful, less concrete and is not what I have set out to understand.

Why I think EA is TC

EA has been and is talent constrained, according to surveys based on input from several EA orgs⁴: 2017 survey, 2018 survey, 2019 survey. These surveys were conducted by 80k and CEA. In all the surveys EAs on average claim to be more Talent Constrained than Funding Constrained. For example, in 2019 EA orgs reported feeling more Talent Constrained (3 out of 5 rating) and less Funding Constrained (1 out of 5 rating)⁵.

80k doesn’t seem to have changed it’s position on this matter since a while. In 2015, 80k suggested that we should focus on providing talent to the community rather than ETG, in “Why you should focus on talent gaps and not funding gaps”. One of the examples they give is about AI Safety where there are people who are ready to donate even more funds, but think there isn’t enough “talent pool”. More posts such as “Working at EA orgs (June 2017), “The world desperately needs AI strategists (June 2017), “Why operations management is the biggest bottleneck in EA” (March 2018), and High-Impact-Careers (Aug 2018), continue to make the case for EA orgs being TC. Even in their recent post, “Key Ideas” (October 2019)–which is mostly recycled from the 2018 article on High-Impact-Careers–they continue to say that the bottleneck to GPR for example, is researchers and operations people⁶.

In Nov 2018, they wrote a post to clarify any misconceptions regarding the understanding of the term TC: “Think twice before talking about Talent gaps”. 80k informs us that EA orgs are not TC in general but are TC by specific skills. Some examples (according to them) being, people capable of Disentanglement Research in Strategy and Policy (FHI, OpenAI, Deepmind), dedicated people in influential government positions etc… This is great, the claim is becoming narrower: EA is TC in specifically X. So what is this X?

Where is the EA specifically TC

There seem to be a list of posts from 80k from which we can gather where EA is specifically TC. They are:

• Surveys (2017, 2018, 2019)
• Bottlenecks in top problem profiles (Shaping AI, Working in an EA orgs, GPR etc…)
• Posts on priority career paths (High Impact Careers, Key-ideas)
• Focused bottleneck posts (for AI strategists and Operations)

The surveys from 2017 to 2019 that informed us that the EA Orgs are TC, provide information on “what sort of talent the EA orgs and EA as a whole would need more of, in the next 5 years?”. This question sounds like a proxy to “Where is EA specifically TC?”. 80k seems to agree with this proxy-approximation of the question as evidenced here⁷ and here⁸. The top 7 results (out of 20 or so) are below:

* High level overview of EA
*** Government and Policy

For the talents that are unclear⁹, I am unable to do anything with them at the moment. For the ones that I have clear examples for, I proceed further.

Another way to arrive at or to supplement this list, is to look at the top problem profiles and check what the bottlenecks are. For example, in the profile on shaping AI (March 2017), we see that 80k calls for people to help in AI Technical research, AI Strategy and Policy, Complimentary roles and, Advocacy and Capacity building. So basically EVERYTHING IN AI except ETG, is TC (it appears). In the problem profile on GPR (July 2018), 80k suggests that they mainly need researchers trained in math, econ, phil etc… Also needed are academic managers and operations staff. A very similar story for working at EA orgs as well.

Is it just me or is EA TC in “general”? Like when researchers, operations people and managers are in shortage at GPR orgs, AI orgs and other EA orgs, then who else is left?

In the post on High Impact Careers (August 2018), 80k suggests the following priority career paths and what they are constrained by:

In brief, we think our list of top problems (AI safety, biorisk, EA, GPR, nuclear security, institutional decision-making) are mainly constrained by research insights, either those that directly solve the issue or insights about policy solutions. — High Impact Careers

In focused bottleneck posts for Operations and AI Strategy just the title already informs how TC the situation is:

• The world desperately needs AI strategists (June 2017)

  Here, other than the title, I didn’t really understand the “desperate need for AI strategists”. Miles expects that “many” jobs would open up in the “future”.

• Why operations management is one of the biggest bottlenecks in effective altruism (March 2018)

  80k updated this post one year later saying that the post is “somewhat out of date”, and that the job market has changed over the last year. That sounds plausible.

In conclusion, the surveys say GRs, ML/AI people, GPR people and movement building are TC (2019). The problem profiles seem to suggest that GPR and AI are completely TC except for ETG (2017,2018). Whereas the High-impact-careers post says that research insights (good researchers) and policy solutions (good policy people) are the most constrained (2018). It appears that there is some discrepancy between different articles–every article doesn’t seem to say the same thing–but we move on with the key message that all these things listed could be potentially TC. But are they really TC though?

The Evidence

GR in GPR

Researchers in GPR are claimed to be constrained. GR’s also stand on top of the survey lists shown before, for 2019. Yet, Open Phil seems to paint a very different picture. For the recent hiring round by Open Phil (started in Feb 2018 and ended in December 2018) they wanted to hire 5 GRs. They report that more than a 100 strong resumes with missions aligned to that of Open Phil were received. 59 of them were selected after remote work tests and went into an interview. Of this, 17 of them were offered a 3 month trial and 5 selected in the end. “Multiple people” they met in this round are claimed to have potential to excel in roles at Open Phil in the future. Open Phil does not seem to feel that there is a lack of skilled people. It appears that they had plenty to choose from and that they have found suitable candidates.

A similar case is observed with EAF. In EAF’s November 2018 hiring round they wanted to hire 1 GR (for grant evaluation) and 1 operations person. Within just 2 weeks, 66 people applied to this EA org which was in a non-hub¹⁰. These 66 trickled down to 10 interviews after work tests, 4 were offered trials and 2 were selected in the end. No TC in GR here either.

Would Open Phil like to hire more GRs? For sure, but they don’t have the capability to deploy such a pool of available talent, they say. They seem to be constrained by something else, something not “talent”.

AI Strategy and Policy

Researchers in AI Strategy and Policy are also claimed to be constrained. The surveys echo the same as well. But Carrick from FHI (Sep 2017) suggests that AI Policy implementation and research work is essentially on hold until Disentanglement Research progresses. And that even “extremely talented people” will not be able to contribute directly until then. Similar to Open Phil, institutional capacity to absorb and utilize more researchers in Strategy is constrained, according to Carrick. It must be noted that this is just one persons view on the matter and that a stronger version of evidence for this would be if several AI orgs agreed with Carrick’s view.

Except for the TC in Disentanglement research (DR)—where there seems to be large demand and if you meet the bar, you will get a job—there seems to be no sign of TC in Strategy and Policy, at the moment.

Once DR progresses, there would be a need for “a lot of AI researchers”, Carrick expects. It’s been 2.5 years since the post by Carrick, and as late as Nov 2018, 80k continues to cite Carrick’s article. This seems to suggest that not much might have changed. I have tried requesting Carrick to write a reboot of his initial post and hopefully he can further clarify the TC or lack there of.

Researchers and Management staff in other EA orgs

The co-founder and board member of Rethink Charity seems to suggest that both senior and junior staff for Rethink Charity and Charity Science were not hard to find, aka not TC.

I’ve certainly had no problem finding junior staff for Rethink Priorities, Rethink Charity, or Charity Science (Note: Rethink Priorities is part of Rethink Charity but both are entirely separate from Charity Science)… and so far we’ve been lucky enough to have enough strong senior staff applications that we’re still finding ourselves turning down really strong applicants we would otherwise really love to hire.—Peter Hurford says in the 2019 survey

The Life You Can Save’s Jon Behar, agrees with Peter. He adds that it’s not the lack of talent but the lack of money to add new staff which is the bottleneck for TLYCS.

Charity Entrepreneurship’s incubation program has grown from 140 applications to ~2000 applications for 15-20 positions since last year. It’s plausibly not TC this year atleast.

Conclusion

An org is TC in Talent X, if it is not able to find “skilled” people despite “hiring actively”. So far we have seen that Open Phil, EAF, Rethink Charity, Charity Science, TLYCS and FHI, are able to find the skilled people they need–except for one concrete example of Disentanglement Research in FHI (and possibly similar institutes). Contrary to the claims from 80k, it appears that several orgs are not TC.

I am really upset with 80k. First it was focusing too much on career capital (CC) and positions like management consulting, and now TC. Getting into the TC debate only opened a Pandora’s box of more issues. Recently, I discovered that their discussion on replaceability is plausibly wrong. They have gone back and forth¹¹ on it in the past and currently have suggested that it depends. They ended up inflating impact associated with people working in EA orgs and have now taken it back. They severely downplayed how competitive it is to get jobs in EA orgs¹². And there are so many cases¹³ of people who feel the same way, not without reason. I traveled with 80k on the CC hype and spent months on identifying positions of “maximum CC”¹⁴. Then I did a 1 year course of Data Science at Coursera. After that I jumped onto the work-at-an-EA-org-because-TC hype and was just about to upskill in statistics and apply for GR positions because they need me.

So many crucial mistakes that cost people like me and others¹³ a lot of time, and the world a “lot of” dollars. And when someone requests one of the members of 80k to not just serve the elite and that perhaps maybe invest in a small conversation with the non-elite EAs to save them years of wasting time, there is no reply.

Thus, I find it very hard to trust the claims listed in 80k. And there are so many of those claims in every post and it’s just impractical to verify each one of them. Rather than relying on the interpretation of English¹⁵ and generalization of advice for everyone, I find EA forums a much easier place to get information from, challenge claims and get responses for (quickly). I found most of the evidence against TC including the Pandora’s box of issues, there. A lot of the successful people from the EA world seem approachable there with chats, comments and AMAs. Recently I was able to chat with Ben West, Aaron Gertler, Peter Hurford, Jon Behar, Jeff Kaufman and Stefan Torges. A bigger celebrity list of people can be seen commenting in posts, such as Carrick Flynn and 80k’s very own Rob Wiblin.

Final message

Caution: Just because an org is not TC, it doesn’t mean that you should reject that org.

Why is this debate so important?

Whether an org is TC or not, has implications on the impact made. The true impact you make when a job is TC at an EA org is (much) higher, than when the job is not TC. A junior GR at GiveWell is expected to move 2.4m$ if the job was TC. The same GR is expected to move only 244k in the case that the hired GR is better than the next-best-candidate by 10% (Not TC). Such is the distinction between being TC and not.

The above example assumes no spillover effects. But is that correct? Why is there no spillover? Should I work in EA or not? How much value do people really get out of working at an EA org? What is best path for my aspiring EA career?

Stay tuned…

STM feedback

Machan,

BTW: http://agent18.github.io/is-ea-bottlenecked-2.html (Uyir a koduthu eluthirkain;80hrs ezhuthirkain, need some meha critical feedback on writing, any potential claim evidence fuckups etc…)

I would pause immediately at the term “talent-constrained”. I don’t understand it. The first step is to describe the claim “X is talent-constrained” in terms of familiar claims so that you can actually test the claim against evidence. Do you mean that there is a low rejection rate for PhD applicants?

A related issue is that, for the same claim, you’re switching between evidence in the form of surveys and evidence in the form of rejection rates. If the claim is about rejection rates, then you either have the numbers or you don’t. If you don’t, you can’t test that claim against a concrete example. If the claim is about surveys, then you’ll have to use the survey. Right now, you’re taking the same claim and mixing multiple kinds of evidence, such as surveys and 80k opinion posts and rejection rates, which left me at the end with no clear answer.

So, split the different kinds of claims: “EA has high rejection rates” and “EA surveys have high percentages saying the words ‘we are talent-constrained’”. You can even make the claims more precise: “Operations manager roles have high rejection rates for candidates with 2 years of similar experience” - notice how you can immediately test that claim given concrete examples.

Contrast that to “In brief, we think our list of top problems … are mainly constrained by research insights”. How do you test that given some data? Imagine if they’d said “Salaries went up by 20% last year but number of open questions solved in published papers went down by 40%”. We can debate whether “number of open questions solved” has been a useful metric but there’s no question that we can test that claim against evidence.

On a different note, a key point that is missing from the analysis, here and elsewhere, is that we talk about a shortage for a given price. We don’t say that there is no supply of onions. We say there is no supply of onions at Rs 10/kg. When the price has eventually risen due to lack of supply, people have even transported onions from other countries to supply them for a profit.

Saying that there is a lack of “talent” or researchers for a given role doesn’t make much sense unless you talk about the current salary. But people are talking as though, no matter the salary, there is not enough talent in the world to do this research. People have in the past moved from country to country and from job to job for higher pay (and other desired characteristics like climate and family members). There are a lot of well-published PhDs and postdocs working on all kinds of other research areas for much less than six-figures and a lot of professors and researchers working for not too much more. Is the claim that they won’t switch for a 2x salary or that they can’t study and catch up on the slightly different field in a few years? If the EA organizations were “desperate” for a particular kind of researcher, did they raise the salary a lot? If they didn’t have enough funds to raise salaries, then aren’t they… “funding-constrained”? Are the two “constraints” actually distinct?

Mission #6: For now, I recommend rewriting the post after splitting the claims till you have narrow claims that are either tested with examples or don’t have any available examples. Check if you see any lingering confusion or ambiguity at the end. One week should be enough time.

Footnotes