A lot of policy research seems to be written with an agenda in mind to shape the narrative. And this kind of destroys the point of policy research which is supposed to inform stakeholders and not actively convince or really nudge them.

This might cause polarization in some topics and is in itself, probably snatching legitimacy away from the space.

I have seen similar concerning parallels in the non-profit space, where some third-sector actors endorse/do things which they see as being good but destroys trust in the

This gives me scary unilaterist's curse vibes..

Everyone who seems to be writing policy papers/ doing technical work seems to be keeping generative AI at the back of their mind, when framing their work or impact. 

This narrow-eyed focus on gen AI might almost certainly be net-negative for us- unknowingly or unintentionally ignoring ripple effects of the gen AI boom in other fields (like robotics companies getting more funding leading to more capabilities, and that leads to new types of risks).

And guess who benefits if we do end up getting good evals/standards in place for gen AI? It seems to me companies/investors are clear winners because we have to go back to the drawing board and now advocate for the same kind of stuff for robotics or a different kind of AI use-case/type all while the development/capability cycles keep maturing.

We seem to be in whack-a-mole territory now because of the overton window shifting for investors.

I don't think we have a good answer to what happens after we do auditing of an AI model and find something wrong.

Given that our current understanding of AI's internal workings is at least a generation behind, it's not exactly like we can isolate what mechanism is causing certain behaviours. (Would really appreciate any input here- I see very little to no discussion on this in governance papers; it's almost as if policy folks are oblivious to the technical hurdles which await working groups)

I see way too many people confusing movement with progress in the policy space. 

There can be a lot of drafts becoming bills with still significant room for regulatory capture in the specifics, which will be decided later on. Take risk levels, for instance, which are subjective - lots of legal leeway for companies to exploit. 

Communicating by keeping human rights at the centre of AI Policy discussion is extremely underappreciated.
For e.g., the UN Human Rights chief in 2021 called for a moratorium on the sale and use of artificial intelligence (AI) systems until adequate safeguards are put in place.

Respect for human rights is a well-established central norm; leverage it  

Great post!
Do note that given the context and background, a lot of your peers are probably going to be nudged towards charitable ideas. I would want to be generally mindful that you are doing things that have counterfactual impacts while also taking into account the value of your own time and potential to do good.

I encourage you to also be cognizant of not epistemically taking over other people's world models with something like "AI is going to kill us all" - I think an uncomfortable amount of space inadvertently and unknowingly does this and is one of the key reasons why I never started an EA group at my university.

Also, here is a link if anyone wants to read more on the China AI registry which seems to be based on the model cards paper

Nice summarization! I generally see model registries as a good tool to ensure deployment safety by logging versions of algorithms and tracking spikes in capabilities. I think a feasible way to push this into the current discourse is by setting it in the current algorithmic transparency agenda. 

Potential risks here include who decides what is a new version of a given model. If the nomenclature is left in the hands of companies, it is prone to be misused. Also, the EU AI Act seems to take a risk-based approach, with the different kinds of risks being more or less lines in the sand.

Another important point is what we do with the information we gather from these sources - I think there are "softer"(safety assessments, incident reporting) and "harder"(bans, disabling) ways to go about this. It seems likely to me that governments are going to want to lean into the softer bucket to enable innovation and have some due process kick in. This is probably more true with the US which has always favoured sector-specific regulation.

It's frankly quite concerning that usually technical specifications are only worked on by Working Groups after high-level qualitative goals are set by policymakers- seems to open a can of worms for different interpretations and safety washing.

After talking and working for some time with non-EA organisations in the AI Policy space, I believe that we need to give more credence to the here-and-now of AI safety policy as well to get the attention of policymakers and get our foot in the door. That also gives us space to collaborate with other think tanks and organisations outside of the x-risk space that are proactive and committed to AI policy. Right now, a lot of those people also see X-risks as being fringe and radical(and these are people who are supposed to be on our side).

Governments tend to move slowly, with due process, and in small increments(think, "We are going to first maybe do some risk monitoring, only then auditing"). Policymakers are only visionaries with horizons until the end of their terms(hmm, no surprise). Usually, broad strokes in policy require precedents of a similar size for it to be feasible within a policymakers' agenda and the Overton window. 

Every group that comes to a policy meeting thinks that their agenda item is the most pressing because, by definition, most of the time, contacting and getting meetings with policymakers means that you are proactive and have done your homework.

I want to see more EAs respond to Public Voice Opportunities, for instance- something I rarely hear on the EA forum or via EA channels/material.