Co-Authors: @Rocket, @Ryan Kidd, @LauraVaughan, @McKennaFitzgerald, @Christian Smith, @Juan Gil, @Henry Sleight
The ML Alignment & Theory Scholars program (MATS) is an education and research mentorship program for researchers entering the field of AI safety. This winter, we held the fifth iteration of the MATS program, in which 63 scholars received mentorship from 20 research mentors. In this post, we motivate and explain the elements of the program, evaluate our impact, and identify areas for improving future programs.
Key details about the Winter Program:
This post is a mix of
Shameless self promotion portion bit:
...LessOnline is in 3 weeks (May 31 – June 2). It's "a festival of writers who are wrong on the internet (by try to be less so)", celebrating truthseeking and blogging.
Early ticket prices end this Monday (aka 3 days from now as of me posting this).
Writers attending include Scott Alexander, Zvi Mowshowitz, Eliezer Yudkowsky, Patrick McKenzie, Agnes Callard, Katja Grace, Kevin Simler, Andy Matuschak, Cremieux Recueil, Duncan Sabien, Joe Carlsmith, Aella, Clara Collier, Alexander Wales, Sarah Constantin, and more.
It'll be a weekend filled with talks, workshops, puzzle-hunts, dance parties, and late-night conversations around the fireside. There's also on-site
This could be a long slog but I think it could be valuable to identify the top ~100 OS libraries and identify their level of resourcing to avoid future attacks like the XZ attack. In general, I think work on hardening systems is an underrated aspect of defending against future highly capable autonomous AI agents.
In this Rational Animations video, we discuss s-risks (risks from astronomical suffering), which involve an astronomical number of beings suffering terribly. Researchers on this topic argue that s-risks have a significant chance of occurring and...
I'm surprised the video doesn't mention cooperative AI and avoiding conflict among transformative AI systems, as this is (apparently) a priority of the Center on Long-Term Risk, one of the main s-risk organizations. See Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda for more details.
Animal Ethics has recently launched Senti, an Ethical AI assistant designed to answer questions related to animal ethics, wild animal suffering, and longtermism. We at Animal Ethics believe that while AI technologies could potentially pose significant risks to animals, ...
Thank you to James Ozden for feedback on this post.
Edit: I added "relatively" to the title to more precisely capture my claim. To be clear, I think CWRs are still underfunded in absolute terms.
In this post I argue that corporate welfare reforms (CWRs)* are relatively...
This is indeed a legitimate concern. We do not have accurate information on the distribution of BCC-approved breeds used in the committments made so far, but I believe that organizations working on and monitoring the committments (possibly the Humane League and CIWF, which publishes the Chicken Track), are likely to have this information. From statements of company's representatives, it seems that the Hubbard breeds are prevailing in Europe, see e.g. this statement: "In Europe, where the issue of breed is more advanced than in the U.S., the Hubbard JA757, ...
Just as the 2022 crypto crash had many downstream effects for effective altruism, so could a future crash in AI stocks have several negative (though hopefully less severe) effects on AI safety.
The most obvious reason AI stocks might crash is that...
Bitcoin is only up around 20% from its peaks in March and November 2021. It seems far riskier in general than just Nvidia (or SMH) when you look over longer time frames. Nvidia has been hit hard in the past, but not as often or usually as hard.
Smaller cap cryptocurrencies are even riskier.
I also think the case for outperformance of crypto in general is much weaker than for AI stocks, and it has gotten weaker as institutional investment has increased, which should increase market efficiency. I think the case for crypto has mostly been greater fool theory (a...
According to some highly authoratitive anecdotal accounts, when a lone crab is placed in a bucket it will crawl out of its own accord but put a pile of crabs in a bucket and they will pull each other down in an attempt to escape, dooming them all. This is a classic illustration...
It's odd that you say the reviewer provides no support for his assertions. It seems to me like the reviewer presents quite a bit of evidence.
For example, in responding to Bregman's claim that male control over female sexuality (and gender inequality more generally) began with the rise of agriculture, Buckner (the reviewer) mentions arranged marriages among the !Kung, a hunter-gatherer society. Buckner also references husbands beating their wives for infidelity among the Kaska, a nomadic foraging society. He also references the Ache, a hunter-gatherer socie...
Introducing Ulysses*, a new app for grantseekers.
We (Austin Chen, Caleb Parikh, and I) built an app! You can test the app out if you’re writing a grant application! You can put in sections of your grant application** and the app will try to give constructive feedback about your applicants. Right now we're focused on the "Track Record" and "Project Goals" section of the application. (The main hope is to save back-and-forth-time between applicants and grantmakers by asking you questions that grantmakers might want to ask.
Austin, Caleb, and I hacked together a quick app as a fun experiment in coworking and LLM apps. We wanted a short project that we could complete in ~a day. Working on it was really fun! We mostly did it for our own edification, but we’d love it if the product is actually useful for at least a few people in the community!
As grantmakers in AI Safety, we’re often thinking about how LLMs will shape the future; the idea for this app came out of brainstorming, “How might we apply LLMs to our own work?”. We reflected on common pitfalls we see in grant applications, and I wrote a very rough checklist/rubric and graded some Manifund/synthetic applications against the rubric. Caleb then generated a small number of few shot prompts by hand and then used LLMs to generate further prompts for different criteria (e.g., concreteness, honesty, and information on past projects) using a “meta-prompting” scheme. Austin set up a simple interface in Streamlit to let grantees paste in parts of their grant proposals. All of our code is open source on Github (but not open weight 😛).***
This is very much a prototype, and everything is very rough, but please let us know what you think! If there’s sufficient interest, we’d be excited about improving...