16

End of Project Write Up: Effective Altruism Policy Analytics

If this is your first time reading about Effective Altruism Policy Analytics, you can look at our website here for more information: https://eapolicy.wordpress.com/ Effective Altruism Policy Analytics finished its last policy comment for the summer on Saturday, September 12th, 2015. The project ran longer than its initial funding period due... Read More
8

Effective Altruism Policy Analytics Update

If this is your first time reading about Effective Altruism Policy Analytics, you can look at our website here for more information: https://eapolicy.wordpress.com/ Effective Altruism Policy Analytics has now been working for eight weeks, and is more than halfway through its experimental period. Our overall strategy for finding regulations to... Read More
Comment author: John_Maxwell_IV 18 June 2015 08:20:57PM 2 points [-]

What does the "assessment" column mean?

Comment author: Gentzel 20 June 2015 10:58:09PM 1 point [-]

In an early version of the sheet we had multiple columns subjectively assessing things like the replaceability of comments, how high impact an influential comment could be, and our sense of how probable influence was. Each person on the team had their own column for ranking importance.

In the current sheet, these were merged together to make a rough prioritization and remove clutter from the sheet for those who help us. That being said, this prioritization did not take into account our current team ability to produce comments, or the fact that easier comments may be good for feedback. This is why we submitted a low importance comment as a feedback test.

2

Effective Altruism Policy Analytics, cooperative documents and feedback

For those who haven't seen already, Effective Altruism Policy Analytics  made its first post  last week and will still be looking for more assistance going forward.  We are looking for help in: Finding experts Making policy comments Getting feedback Logo design Building a website Searching regulations.gov If you are interested, or... Read More
Comment author: Gentzel 16 June 2015 03:30:40PM 1 point [-]

For those who are interested, this is our current blog: https://eapolicy.wordpress.com/

We will try to keep it fairly updated.

Comment author: egastfriend 14 June 2015 02:52:14AM 0 points [-]

This sounds awesome! Is the idea that policy comments would get regulators to: consider a new policy they weren't previously considering; change their mind about a proposed policy; help back them up politically for something they already want to do; or a combination of these? I'm not sure which type of impact policy comments are best for.

Comment author: Gentzel 15 June 2015 03:48:39PM 2 points [-]

I think it is most likely we will be backing up good policies that some regulators want. New policies are hard, and a lot of requests for comments come in a sort of binary way: "should we implement policy x.1 or x.2?"

Comment author: Peter_Hurford  (EA Profile) 13 June 2015 05:34:59PM 4 points [-]

I second what John Maxwell says -- definitely one of the best things I think you could do is set up a blog and practice some Givewellian transparency where you publish as much as you can. It would be cool for us all to learn from it!

Especially if you speak candidly about learning experiences, mistakes, and failures and how you improve from them.

Comment author: Gentzel 15 June 2015 03:44:42PM 3 points [-]

I currently have a google doc that I have been using to record hours, mistakes, lessons learned, and observations. I do think I should write it up as we make a blog.

Writing down problems has seemed to function in a similar way to rubber ducking though, trying to get certain problems into words can sometimes highlight a solution, and that has been useful.

Comment author: John_Maxwell_IV 13 June 2015 12:27:32AM *  8 points [-]

This sounds like a pretty cool project. Some thoughts:

Do we have evidence, whether quantitative or anecdotal, about the level of impact that policy comments have?


Reading about your spreadsheet reminded me of this post on sequence thinking vs cluster thinking. I think spreadsheets are inherently sequence-ish in nature; a change to a single field can completely alter the result.

As a software developer, I'm intimately familiar with the phenomenon of tiny subtle mistakes causing unexpected catastrophes... writing production software trains a certain paranoia when dealing with computers that's hard to get elsewhere. Sometimes I wonder if most scientific simulation code has subtle errors in it that invalidate the results because the researchers don't have enough programming experience to realize how easy it is to write code that's incorrect and simulations are black boxes where errors don't necessarily manifest themselves immediately & visibly.

One interesting idea might be for each person in your group to try to generate frameworks for evaluating a proposed regulation independently, and meet to compare spreadsheets later. Probably this would work even better with a group that had a diverse set of backgrounds (educationally, politically, etc.)


Because politics is apparently inherently interesting to almost everyone, you might be able to start blogging about whatever regulations you are looking in to and extract free cognitive labor from your readers to provide an additional perspective, critique your research, etc. In fact, maybe there is low-hanging fruit in just redirecting the attention of intelligent policy bloggers to policies that can actually be impacted in the near term instead of just whatever the fashionable discussion topics of the day are, then writing up any consensus they come to as a comment.

(This is a tougher one since there are also downsides to having politics discussions online, which I wrote about previously. Not sure where I currently stand, but leaning towards a blog promoting a high standard of discussion being the best option.)

Broadly my intuition is that some policy recommendations have the potential for 10x or 100x the impact relative to others, so you are better off aiming to produce a small number of high-quality, highly-targeted recommendations than a large number of lower-quality ones, at least in the long run. (Incidentally, are animal issues on your radar?)

Comment author: Gentzel 15 June 2015 03:26:53PM 4 points [-]

I think we will start blogging in a limited capacity about regulations we are seriously considering working on and some that we considered and then dismissed. We probably aren't going to blog about every regulation we look at since there are so many. Some comments are likely to be far more impactful than others, however the comments that are likely to have the most impact are also likely to have slower feedback and no nearby certain deadlines for implementation.

Our current priority list seems to be: -Network early to get expert feedback and assistance -Produce lots of comments early to get feedback and learn how to make influential comments -Focus on high impact comments toward the end of our project trial

To some degree, these priorities can get jumbled by time sensitive opportunities, but as an overall aim, we think this is correct for moving forward.

Animal issues are on our radar, but I have yet to see anything lately relevant to factory farming of cows, pigs, or chickens. We have seen a lot of proposed rules about fisheries and species protection, but didn't have the expertise to go after them yet. If there are experts we could consult on animal issues, they would likely be helpful unless their way of sorting policies into "worth going after" and "not worth it" is the same as ours and nothing new comes up/is noticed.

Comment author: John_Maxwell_IV 13 June 2015 12:27:32AM *  8 points [-]

This sounds like a pretty cool project. Some thoughts:

Do we have evidence, whether quantitative or anecdotal, about the level of impact that policy comments have?


Reading about your spreadsheet reminded me of this post on sequence thinking vs cluster thinking. I think spreadsheets are inherently sequence-ish in nature; a change to a single field can completely alter the result.

As a software developer, I'm intimately familiar with the phenomenon of tiny subtle mistakes causing unexpected catastrophes... writing production software trains a certain paranoia when dealing with computers that's hard to get elsewhere. Sometimes I wonder if most scientific simulation code has subtle errors in it that invalidate the results because the researchers don't have enough programming experience to realize how easy it is to write code that's incorrect and simulations are black boxes where errors don't necessarily manifest themselves immediately & visibly.

One interesting idea might be for each person in your group to try to generate frameworks for evaluating a proposed regulation independently, and meet to compare spreadsheets later. Probably this would work even better with a group that had a diverse set of backgrounds (educationally, politically, etc.)


Because politics is apparently inherently interesting to almost everyone, you might be able to start blogging about whatever regulations you are looking in to and extract free cognitive labor from your readers to provide an additional perspective, critique your research, etc. In fact, maybe there is low-hanging fruit in just redirecting the attention of intelligent policy bloggers to policies that can actually be impacted in the near term instead of just whatever the fashionable discussion topics of the day are, then writing up any consensus they come to as a comment.

(This is a tougher one since there are also downsides to having politics discussions online, which I wrote about previously. Not sure where I currently stand, but leaning towards a blog promoting a high standard of discussion being the best option.)

Broadly my intuition is that some policy recommendations have the potential for 10x or 100x the impact relative to others, so you are better off aiming to produce a small number of high-quality, highly-targeted recommendations than a large number of lower-quality ones, at least in the long run. (Incidentally, are animal issues on your radar?)

Comment author: Gentzel 15 June 2015 02:41:40PM 2 points [-]

As for spreadsheets, we could go through a cluster thinking way of producing estimates, but I am under the impression this would take a lot more time per person, and then when comparing spreadsheets at the end, we'd be finding errors that would have been easier to handle earlier if we worked together earlier and got faster feedback.

There certainly is value to avoiding groupthink though. Overall I do think using multiple sequential techniques could be a rather rigorous way to evaluate something, and make a very good comment, but we are also trying to get useful feedback by making a lot of comments and we are already commenting much slower than we desired to because of outreach.

I would like to do this, but I think it would be useful to have more comment drafting workers if we want to do this a lot.

Comment author: John_Maxwell_IV 13 June 2015 12:27:32AM *  8 points [-]

This sounds like a pretty cool project. Some thoughts:

Do we have evidence, whether quantitative or anecdotal, about the level of impact that policy comments have?


Reading about your spreadsheet reminded me of this post on sequence thinking vs cluster thinking. I think spreadsheets are inherently sequence-ish in nature; a change to a single field can completely alter the result.

As a software developer, I'm intimately familiar with the phenomenon of tiny subtle mistakes causing unexpected catastrophes... writing production software trains a certain paranoia when dealing with computers that's hard to get elsewhere. Sometimes I wonder if most scientific simulation code has subtle errors in it that invalidate the results because the researchers don't have enough programming experience to realize how easy it is to write code that's incorrect and simulations are black boxes where errors don't necessarily manifest themselves immediately & visibly.

One interesting idea might be for each person in your group to try to generate frameworks for evaluating a proposed regulation independently, and meet to compare spreadsheets later. Probably this would work even better with a group that had a diverse set of backgrounds (educationally, politically, etc.)


Because politics is apparently inherently interesting to almost everyone, you might be able to start blogging about whatever regulations you are looking in to and extract free cognitive labor from your readers to provide an additional perspective, critique your research, etc. In fact, maybe there is low-hanging fruit in just redirecting the attention of intelligent policy bloggers to policies that can actually be impacted in the near term instead of just whatever the fashionable discussion topics of the day are, then writing up any consensus they come to as a comment.

(This is a tougher one since there are also downsides to having politics discussions online, which I wrote about previously. Not sure where I currently stand, but leaning towards a blog promoting a high standard of discussion being the best option.)

Broadly my intuition is that some policy recommendations have the potential for 10x or 100x the impact relative to others, so you are better off aiming to produce a small number of high-quality, highly-targeted recommendations than a large number of lower-quality ones, at least in the long run. (Incidentally, are animal issues on your radar?)

Comment author: Gentzel 15 June 2015 01:43:12PM 3 points [-]

Some of these questions require semi-detailed responses, so I will respond with a few different comments. Richard had some examples/anecdotes about the level of impact policy comments could have:

"A good recent example of FDA making major changes as a result of public comment is the Food Safety Modernization Act (FSMA) Produce Safety rule. This rule was extensively changed as a result of feedback from the public, mostly the affected farmers. Some of the comments were merely self-interested lobbying, but some pointed out where FDA's lack of understanding of specific situations would have led to an inefficient regulation. The changes were so extensive that FDA had to publish a reproposal of the rule.

This is a large and complex rule, so you can choose how much detail you want to look into. The overview is here:

http://www.fda.gov/Food/GuidanceRegulation/FSMA/ucm334114.htm

and it has links to further information. The full Federal Register notice is here:

https://www.federalregister.gov/articles/2014/09/29/2014-22447/standards-for-the-growing-harvesting-packing-and-holding-of-produce-for-human-consumption

You can do a text search in the Federal Register document for the phrase "Relevant Comments" to see the FDA responses to comments on various subjects and how those comments changed the regulation.

This is a link to the economic analysis of the reproposal, which has details on the costs and benefits of the changes in the reproposal:

http://www.fda.gov/downloads/AboutFDA/ReportsManualsForms/Reports/EconomicAnalyses/UCM442801.pdf

Another rule where public comments led to major changes and a repropsal is the FSMA Preventative Controls rule:

http://www.fda.gov/Food/GuidanceRegulation/FSMA/ucm334115.htm

However, understanding the issues in that rule requires a bit more specialized knowledge."

View more: Prev | Next