The difference between human extinction and a wide range of positive outcomes is dependent upon how technology is developed and implemented. The EA community knows this better than anyone.
But a sustainable process to maximize the positive contributions of technology to human well-being while minimizing the probability of existential and global catastrophic risks requires participation by domain leaders in tech development, policy, academia, and business. This is currently not the case.
One of the ways to remedy this is by targeting future domain leaders. That’s why I’m excited to announce a new EA-aligned organization, Envision, with the goal of imbuing a forward-looking but safety-conscious mindset towards technology in future leaders in tech development, policy, academia, and business by intervening at the college level. The goal is two-fold: convince future domain leaders that technology will play a pivotal role in humanity’s future, and instill in them the careful consideration of technological safety as a core value.
This post will:
- Outline Envision’s current status
- Justify the claims made above
- Outline Envision’s goals and strategy
- Define metrics of success and measurement mechanisms
- Review failure modes
- Explain how EAs can potentially be involved
Current status
As a general outline, Envision will be a global network of student chapters with the goal of imbuing a forward-looking but safety-conscious mindset towards technology in future leaders in tech development, policy, academia and in relevant business leaders.
Envision is currently a 6-month old student group at Princeton, with 15 officers and 91 members, which will hopefully significantly increase with fall recruiting. Envision’s website, although still under construction, provides more information about past events.
Students at MIT, Stanford, Cambridge, and UPenn have tentatively expressed interest in opening a chapter.
The locus of Envision is our Conference, Dec 2-4, which will serve as a focal point for recruiting and kickstart many of the processes and strategies I am about to describe.
Our current official partners include MIRI, Future of Life Institute, Jaan Tallinn, and Sam Altman among others. Confirmed speakers at our conference include Andrew Critch from MIRI, Robin Hanson, Anders Sandberg from FHI, and Ruth Chadwick from the Centre for Economic and Social Aspects of Genomics. MIRI will most likely host a workshop during the conference.
Justification of claims
This section will expand upon the claims made in the introduction.
-
The difference between human extinction and a wide range of positive outcomes is dependent upon how technology is developed and implemented.
- Existential risk is inherent in artificial intelligence, synthetic biology, and nanotechnology, as argued by Nick Bostrom among others.
- Even given that existential risk is avoided, these technologies, in addition to genetic engineering, space technology, fusion, virtual and augmented reality, and neuro-engineering, have a wide range of possible impacts, from the catastrophic to unprecedented advances in human well-being and capabilities.
- Given the rapid pace of technological development,
- Other technologies are likely to arise which will be crucially important to humanity's future, and
- The impacts of the development of these technologies is likely to be shaped by actions in the short to medium-term and will be felt within a century.
- But a sustainable process to minimize the probability of existential and global catastrophic risks while maximizing the positive contributions of technology to human well-being requires participation by domain leaders in tech development, policy, academia, and business.
- The argument is often made that AI is more safely developed by a small group of people, without significant domain leadership involvement. However, the key word is sustainable: the future is unpredictable and new technologies and issues will crop up. It is not sustainable to have to convince the relevant players and assemble a team anew every time. As Nick Beckstead argues, broader and more general interventions, including in education, are important to shaping the far future.
- Furthermore, ‘participation’ is different from ‘exposure’ and is assumed to be positive. A misinformed elite meddling in AI safety is not a desirable outcome; an elite participating in safe technological development is a positive outcome. Participation is far harder to achieve than exposure, but because of how Envision is set up, is achievable.
- Technology does not exist in a vacuum. How it is implemented is as important as how it is developed. The actions taken by governments, academic institutions, and businesses both influence technological development and determine implementation.
- This is currently not the case.
- Although a small number of domain leaders in tech development and policy are now Effective Altruists and/or are engaged with AI and other technologies in a safety-conscious manner, this remains the exception.
- The way to remedy this is by targeting future domain leaders during university.
- College strikes a balance between flexibility and accessibility on the one hand, and intellectual maturity and indentifiability on the other.
- Flexibility: Still sufficiently early that beliefs are in flux as students are exposed to new ways of viewing the world, and the most important career choices still lie in the future. Many smart, socially aware students default to tech, consulting, and finance for lack of prestigious and effective alternatives, which can be more easily avoided pre- than post-decision.
- Intellectual maturity: However, it is late enough such that existential risk, exponential growth, the technical details of technology, and other important concepts can be grasped.
- Accessibility: Future domain leaders are still easily accessible at low cost.
- Identifiability: However, it is late enough that future domain leaders are considerably more readily identifiable and can be specifically reached out to.
- One major disadvantage is the more extended time frame before interventions reap benefits, as it takes time before current students reach the top positions within their domain. This is valid, but there are reasons to think a college intervention now will almost certainly pay off.
- Sub-elites exert significant influence on the culture of domain leaders, so effects will be realized considerably before the future domain leaders that are exposed to Envision assume full control. The impact of the intervention begins increasing from the moment students enter the workforce and is thus not as long-term as it appears.
- The massive impact a successful university-level intervention would have in causing a significant shift in mindsets is likely to justify the longer time until full realization of impact.
- Intervening with future domain leaders also reaps longer rewards: current domain leaders have considerably shorter time left in power.
- Identifiability is an issue; current domain leaders can be far more easily identified than future elites. There will certainly be a lower success rate of identification, but what matters is the absolute number of future domain leaders affected, not the percentage of those affected who become domain leaders. It is thus theoretically possible for all future leaders to be influenced despite a low success rate of identification if enough students are reached.
- College strikes a balance between flexibility and accessibility on the one hand, and intellectual maturity and indentifiability on the other.
- There currently does not exist an organization that does what Envision does.
- There is no mechanism by which college students are exposed in a single context to a balanced view of both the potential and the dangers of rapidly developing disruptive technologies.
- X-Risk organizations and student groups do not appeal to most future domain leaders, who are more excited by the potential of technology to improve the world.
- Tech-specific and issue-specific groups exist, but none are holistically oriented to take into account the bigger picture.
- There is no mechanism for pulling in future policy makers and business leaders into the same space and mindset.
- Although online resources exist, they are not easy to find, and the message is not brought to most future domain leaders. There is thus no easy way to access the information without an existing initial interest or recommendation.
Envision – The Strategy
As a summary, Envision will be a network of student chapters at top unis worldwide with the goal of imbuing a forward-looking but safety-conscious mindset towards technology in future leaders in tech development, policy, academia and in relevant business leaders.
This section provides an overview of Envision’s strategy for accomplishing this.
Goals can be broken down into four separate categories:
- Attract and retain future leaders across the domains relevant to future technology development.
- Convince future leaders that technology will play a pivotal role in humanity’s future and integrate technology safety into their mindset.
- Provide the tools to contribute towards the positive development of technology.
- Providing a pipeline into EA.
Strategies for accomplishing goals
For attracting and retaining future leaders
Leaders must first be identified. For this, we are developing a predictive model identifying key variables current domain leaders had in common during university.
Leaders must then be attracted. There seem to be three factors that have widespread appeal:
- Positive messaging emphasizing the potential of technology while integrating concern for safety. Envision’s official mission is inspiring students to pioneer a brighter future through the responsible and innovative use of rapidly developing disruptive technologies, and a lot of our events focus on the positive applications of technology.
- Domain leadership in the fields they’re interested in. To appeal to this, our strategies are:
-
- Partnering with domain-leading organizations and individuals. Our current list includes MIRI, FLI, Sam Altman, and Jaan Tallinn, and we are in the process of expansion.
- Interfacing with firms on the cutting edge of technology, through inviting them to demo their technologies and visiting their offices and labs.
- Partnering at each university with established domain-leading student groups. At Princeton, our founding location, we are partnered with Entrepreneurship Club.
- Inviting domain-leading speakers.
- Developing prestige to attract the most talented students. This includes:
- emphasizing selectivity in officer and fellow selection, although membership will be open to all
- partnering with prestigious organizations
- including a competitive element
- ensuring sleek design and streamlined operations
- targeted reach-out.
Additionally, we will maintain independence from EA. Although there will be links with EA, partnerships with EA orgs, and many EA members, Envision will remain officially non-affiliated to maintain its appeal to those who would be off-put by EA and maintain the differentiation of its message.
The above strategy is highly subject to change as we learn more about what most attracts future leaders and we will continuously update our strategies in response to new information.
Convince future leaders that technology will play a pivotal role in humanity’s future and integrating technology safety into their mindset.
It’s important to integrate both these aspects and present safety as a standard component of technological development, not a stand-alone add-on as it is often portrayed or interpreted.
This is best accomplished by emphasizing technology’s vast potential while consistently addressing safety concerns. A dichotomy between positive applications and negative risks is to be avoided.
Strategies:
- Provide exposure to companies and labs on the cutting edge of rapidly developing disruptive technologies. This makes clear the speed with which technological development is occurring and how close we already are to revolutionary breakthroughs, and the concrete practical applications and implications.
- Integrate safety as a principle into all events and issues, even for small concerns. This will establish it as an ingrained attitude. To do this, we will ask all speakers invited, firms visited, and other relevant parties to address how to mitigate the potential negative outcomes of their technology, and we will address safety in discussions.
- Host specific safety events in the form of talks by safety-related experts, including in:
- Tech development, ie companies such as DeepMind
- Technical safety, such as MIRI
- Safety considerations, such as FHI and FLI
- Curation of online resources, including papers such as Nick Bostrom’s, major news updates, and overviews of relevant technologies.
- Discussion. This is a crucial element to complement the outside expert views with the inside, socially acceptable peer views. Envision will hopefully have many Effective Altruists, so discussion will be one of the most potent tools for integrating a safety mindset.
It is important to note Envision focuses on technological developments and issues that are not commonly known, which is generally correlated with the longer-term. For example, the ethics of self-driving cars are already widely discussed and thus do not fall within the scope of Envision.
For providing the tools to contribute to the safe development of technology
- Envision Entrepreneurship: A competition for the most innovative application of a rapidly developing disruptive technology for large-scale impact, with prize money. With time, partnering with organizations to provide a more direct route for the winning ideas to be implemented. As with everything, safety considerations will be integrated. Being piloted at the conference.
- Interfacing with organizations that are working on innovative tech development. Includes tech firms and start-ups, VC firms, government agencies, research labs, think tanks, and tech safety research organizations. Exact programming highly dependent on organizations. Those deemed to be negatively contributing to human well-being due to an active disregard for safety considerations will be avoided.
- The purpose of this interfacing is both to provide the means for working on safe technological development and smoothing the path towards domain leadership.
- Will include career-oriented interaction with relevant organizations, including career fairs and in select cases presentations on how to work towards the positive development of technology in different careers.
For providing a talent pipeline into EA
This is quite straightforward: sufficiently involved Envision members will be exposed to EA. We’re collaborating with EA Build, the body coordinating EA Student Chapters, to accomplish this, and will coordinate with individual student chapters at the universities where we open chapters.
However, this exposure will be light, in order to retain those future leaders who are not interested in EA – this is a crucial part of Envision’s value-add.
Planned structure
Envision will be a global network of student chapters initially headquartered at Princeton University. Governance may become more decentralized and distributed across chapters with time, depending upon developments.
There will be an annual main Envision conference, and several smaller technology- and issue-specific conferences hosted by different chapters.
Envision-hosted events will generally be open to the entire student body of the university in question. There will be officers, who run the chapter, and fellows, who are selected by invitation from event attendees and gain several perks such as priority of attendance for events limited in size, and access to more intimate fellow-only discussions with professors and visiting experts.
The exact structure of Envision chapters, while open to change and additions, initially includes:
- Understand: guest speaker series and workshops with university and outside speakers and organizations.
- Interface: Trips to nearby companies and labs developing or working on implementing rapidly developing disruptive technologies, and attendance of nearby relevant conferences.
- Envision Entrepreneurship: Smaller local versions of the conference event. Winners will compete at the main Envision conference or at a separate Envision Entrepreneurship event.
- Discuss: Discussions among members, sometimes including faculty. There will be a more selective subset of Fellows who will discuss more frequently and have more intimate discussions with professors and visiting experts.
Success metrics and measurement
These are incomplete. Many measurements and success criteria are not well defined, several are subject to Goodhart’s Law and are not optimized to measure counter-factual impact, and the details of implementation are missing. Feedback and advice for addressing these issues is welcome.
Goal |
Metric |
Measurement |
Success criterion |
Attracting future leaders |
Success in attraction |
Quality of applicants |
Qualitative - compare to criterion of future leader |
Quantity of applicants – Chapters |
10% of freshman class |
||
Quantity of applicants - Conference |
700 applications |
||
Quality of attractiveness |
Potential applicants not applying |
No talented, potentially interested people not applying. A key issue is how to measure this. |
|
Successful screening process |
Fellows alumni turn into future leaders |
50% become leaders |
|
Accurate selection |
Every year after acceptance, fellows are re-evaluated whether they still match future leader criterion. 75% yearly success rate. |
||
Convince future leaders that technology will play a pivotal role in humanity’s future. |
Successful events |
Pre- and post-event survey - Conference |
Significant update towards understanding power of technology in 85% of cases |
Post-event surveys for each event – Chapters |
70% significant updates to deem an event successful |
||
Overall successful mindset change |
End-of-year survey, more detailed informal check-in for 50% of most involved members by non-central members (to avoid bias) |
Significant positive updates, tangible excitement in 80% of cases |
|
Integrating safety into future leaders’ mindset |
Theoretical updates |
Pre and post conference survey |
70% Significant update towards concern for safety |
Post safety event surveys |
30% major update, 60% minor update to deem event successful |
||
Observed changes |
Alumni concerned with safety |
60% of alumni leaders demonstrate concern |
|
Engagement with safety issues |
End-of-year survey; 50% show increased engagement and concern |
||
Providing the tools to accomplish above |
Career changes |
End of year survey question on career change |
70% update, of which 45% major update (positive) |
Observed changes |
Qualitative; how to measure? |
||
Career advancement |
Opportunities through Envision |
15% of career-related event attendees find opportunity. Based on post event survey, several months after. |
|
Partner meetings |
2 partnerships a year formed. |
||
Provide a talent pipeline |
Envision alumni working for the organizations in question who otherwise would not have |
At least 5 researchers within 10 years who updated through Envision. Based on self-reporting. |
Key terms to define
Positive update
Safety update
Leader
Future leader criteria
Positive career change
Partnerships
Minor versus major update
Metrics to keep track of
Number of applicants
Alumni paths and tech safety concerns
Pre and post conference views on tech and tech safety
Post event updates chapters
End of year major updates and tech safety views - number asked and answers (non-anonymous)
Pre and post conference career plans
Observed career changes
Opportunities reached through career-related events
Partnerships formed through Envision
Failure Modes
Envision faces many challenges, of which I will outline some of the most salient.
Inability to identify future leaders
Our predictive model is designed to counteract this, but success is far from guaranteed. Some potential causes of failure:
- Goodhart’s Law.
- Identifying only the visible manifestations, not the underlying traits.
- Finding there are no common identifiable traits.
Inability to attract future leaders
Once identified, being unable to attract future leaders. Potential causes:
- Factors that attract leaders vary so widely between domains it is not possible to appeal to all.
- Path dependence - mistakes early on and the attracting of the wrong initial set of students permanently places Envision on a path that cannot lead to success. This also applies to individual chapters; if initially badly set up, it would be close to impossible to gain traction at the university in question.
- It is not possible to provide the factors that would attract future domain leaders within the boundaries of what is possible given Envision’s goals.
Failure to instill a safety mindset in future domain leaders
This is quite a broad failure mode which can result from several different causes:
- Resources are not well-provided. This could be due to speakers and firms refusing to address, or inadequately addressing, risks.
- Resources are not accessed. For example, students may simply not show up to safety-specific events and not read resources provided.
- Resources fail in instilling a safety mindset: Even if well-provided and accessed, students may refuse to update. This can be partially mitigated by presenting safety as an integrated component, that has concrete solutions.
- Integration of safety leads to a competitive disadvantage, causing those who do update to have a lower likelihood of achieving domain leadership.
These are but the most likely failure modes that we are most actively addressing. There are more, and we are continuously re-evaluating to ensure we remain aware of failure modes, are not in one, and are not missing any.
How you can be involved
Envision is in its early stages. Strategy is subject to change if another route is deemed more conducive to fulfilling Envision’s goals. Critical feedback, questions, and advice on any part of the value proposition and strategy are encouraged.
If you are a student or are affiliated with a university, further involvement is welcomed. You can apply to the conference and, if you have the time, dedication, and interest, help set up a chapter, in which case you can contact me.
If you are not but think Envision is important and are willing to help, further involvement is welcomed. This is especially true if you have organization-building/management experience, in which case you can advise, or model-building skills (quantitative or qualitative), in which case you can help build our predictive model. If the model is successful, it can be widely applied within EA beyond Envision.
If you are in charge of, employed by, or affiliated with an organization or company that potentially would be interested in partnering with Envision, further involvement is welcomed. If Envision is successful, this is an opportunity to potentially have a significant impact on the future of humanity at low cost. More directly, we also have recruiting and marketing opportunities if these are relevant.
Partnership can be content-based, as in the case of our partnership with MIRI, who are providing a speaker and possibly hosting a workshop; the provision of funding (we currently have some but are far from fully funded to execute on all of our projects); or a combination.
I can be reached through Direct Message or at lrade@princeton.edu.
To summarize, in this post I:
- Introduced Envision, a new EA-aligned organization, and its value proposition.
- Outlined Envision's current status.
- Justified the claims made with regards to its value and effectiveness.
- Outlined Envision's strategy
- Presented our success metrics and measurement mechanisms
- Went through some of the most salient failure modes
- Explained how EAs can potentially be involved
Thank you, Gregory. You raise excellent points. I will address them individually and then alltogether in conclusion.
1) That's correct, we will have to compete with other student groups. So far, our message appears powerful enough to give us a significant advantage, which will help partially compensate for our lack of a track record.
We also don’t necessarily have to compete. The strategy of partnering with other successful student groups (ie Entrepreneurship Club at Princeton; similar organizations at a handful of universities have expressed excitement at partnering with us and helping set up a chapter, although of course excitement does not necessarily equal actual work) appears to be sufficient to allow us to compete at a level that we're able to grow and sustain. Envision is in many ways a welcome addition to entrepreneurship groups so they’ve been very receptive so far to partnering and sharing resources.
2) Excellent point. To counter this, we’re focusing on building up an ecosystem of faculty advisors and partner organizations, which adds both prestige in the competition for student leaders and significantly increases the likelihood of sustainability.
I also think you underestimate the appeal of helping build a new organization, especially one working on something exciting, even if you don’t run it. However, I could be wrong on this.
Lack of investment by later leaders is certainly a problem. However, 1) investment is less important since the organization already exists so far less work is required, 2) continued involvement by a board of alumni will help keep the organization on track, and 3) with faculty advisor buy-in some of the continuity will stem from them. With this combination, a weak leader should not cause the organization to collapse.
3) A good point, and one we had not thought of in detail. A few thoughts: 1) we could just use external validation criteria, eg internships at the most competitive companies, although this is not necessarily indicative of future domain leadership. 2) Breadth is a strong solution to this; the more people we reach in absolute terms, the higher the likelihood we touch future leaders.
In light of your point, do you think it’s worth creating a predictive model at all? It would use up valuable man-power, and you’ve convinced me it would likely have limited impact.
4) This is a good point. Although it seems like having fellows could actually increase attendance at events. In any case, you’ve convinced me to defer a fellow program until at least after the conference before re-considering in light of the new evidence we’ll have gathered.
5) I disagree that the targets are ‘stratospheric’ – although they are optimistic. I also don’t see the problem with the ‘gung-ho start-up ethos’ – it gets quite a lot done. The targets are hard to achieve, and it’s very unlikely we’ll hit all of them, but we’ll try and get pretty far in the process. Failing at achieving optimistic targets but getting quite far towards them in striving is much preferable to achieving unambitious ones and sitting back in satisfaction.
Having made that philosophical objection, I do agree our targets are in some cases probably unrealistic. I’d welcome a more detailed explanation of which ones you think are unrealistic and why, as well as suggestions for more reasonable targets.
6) From my perspective, Envision fills a clear and gaping niche. ‘Interested in science and technology’ is different from ‘interested in the medium-term and long-term future of science and technology and what we can do to pioneer a better future with the tools we have and will have available.’ The differences: more action-oriented; focused on future issues that do not receive much attention on college campuses; more broadly focused on multiple technologies and how they interact; integrating technology and science with ethics, policy, and entrepreneurship.
To use reductio ad absurdum on your argument of there existing separate groups that each touch on an aspect of Envision: there have always been groups for altruistic people and groups for effective people. That does not mean a group for effectively altruistic people can add no value.
I also don’t see much overlap with EA groups – most of our members are not EA, even though many have heard of it. Keep in mind we’re targeting future leaders, in particular those who do not yet have a concern for safety or awareness of the future of technology, to help them learn about it.
A final note – I think you overestimate how many similar student groups exist. We’ve now exhaustively gone through all student organizations at over a dozen universities, and have not yet come across an organization with significant overlap that is run well to the point of making Envision unnecessary. Cambridge is among those with the most potential candidates (although we’ve also had the most interest from students there, in addition to MIT, about starting a chapter).
Finally, as a thought experiment – how many student organizations contain entrepreneurs and policy-makers not in the EA sphere and have Andrew Critch and Robin Hanson as speakers?
There is a lot to be said here. First, to break down your second sentence:
• Hyper-successful: I’m not entirely sure what this means, but I don’t see how Envision succeeding at its goals requires it. We certainly need success as a student group, but I don’t think we need substantially more than what would traditionally qualify as success for a student group, albeit repeated several times (which is certainly harder).
• Significant outside prestige: We need prestige among students, but I don’t see why outside prestige is necessary (if I’m interpreting this correctly). It helps with getting external organizations on board, but prestige is sufficient for this, not necessary – being students excited about the future of tech and an organization with the prospect of hiring opportunities, and in the EA sphere being a student organization with the goal of promoting safety, goes a long way.
• Access to extra-university elites: This is true, but we only need very limited access. Ie a few hours on a weekend to come speak or showcase your technology at a conference that pays for your flight, setting up a recruitment booth in return for providing a (for a company) small amount of money, having your name on a student organization and occasionally speaking to excited students, etc. We’ve so far been pretty successful at getting this since it’s low-cost. To be successful, we don’t need more than this – the more the better, but the acceptable threshold of access to ensure success is quite low. In light of this, looking at some other student groups, I think the amount that fail is less than the vast majority, especially when keeping in mind we’re partnering with organizations like Entrepreneurship Club that have already been successful on most relevant metrics and can advise us, lend us their credibility, and help us with recruiting.
On repeating the same process: Here I agree with you. This is certainly one of the most difficult parts of Envision’s strategy. Even if our success is limited to Princeton, though (and I have high confidence we’ll establish at least a few additional chapters at significant universities), I think the net impact is still sufficient to justify building Envision.
With regards to your second sentence:
• Predicting elite leaders – I agree this is difficult, but as I explained in the point about this, not necessary to success. Casting a wide net at universities most likely to produce future leaders ensures high probability of impacting the correct people. Predicting future leaders would be hugely beneficial, but failure here does not invalidate the value proposition.
• Tempting them to invest their time and attention in you – They need only attend a few events and change their minds. We need officers, but this is only a tiny subset, and I think there’s sufficient message attraction to fill this. Getting people to attend events is not trivial, but certainly doable, and has been done before, including by us.
• Ensuring momentum and resilience – I agree that this is a major challenge. However, as I outlined above in the relevant point, I think building a framework will make this significantly easier.
• Getting a nucleus of highly effective leaders to start parallel groups at other universities – I completely agree that this is extremely difficult and highly competitive. But in my experience the draw of starting a new organization, even as a chapter, is quite high. And again, I think the message is powerful and will in itself attract several such highly effective leaders.
For reasons I have already elaborated, I disagree that we have no edge in principle (interpreting this as a synonym for message – correct me if I’m wrong).
Track record – I agree we don’t yet have this, but neither did any organization (especially student organization) upon founding. And the partnership with existing student organizations quite significantly mitigates this.
On your last paragraph about other means of accomplishing what Envision has in mind:
• I agree on the importance of using existing ecosystems, and I think Envision is doing what you describe. Ie Entrepreneurship Club, a conference, being a student group, running a pitch competition, leveraging existing opinions and resources rather than producing our own. I would argue what Envision is doing is leveraging the existing ecosystem more than a student think-tank producing papers would be.
• The work of the Wilberforce Society is admirable, and we will cer... (read more)