07 August 2009

Did EPA Stack its Ethanol Life-Cycle Analysis Peer Review?

The EPA has just released a set of expert peer review reports under the heading of Renewable Fuel Standard Program (RFS2) Lifecycle Greenhouse Gas (GHG) Analysis. The life-cycle analysis of renewable fuels is a political hot button, for reasons that I have discuss here and here and here. For this post the important thing to be aware of is that the Obama Administration has proposed a life-cycle approach to counting the greenhouse gas emissions from ethanol.

Here I focus on the report titled Lifecycle Greenhouse Gas Emissions due to Increased Biofuel Production: Methods and Approaches to Account for Lifecycle Greenhouse Gas Emissions from Biofuels Production Over Time (PDF). The study was conducted by a consulting firm called ICF International, which has a $21 million contract from EPA for technical assistance. The study concluded overall that:
Most peer reviewers generally agreed that the approach taken by EPA was scientifically objective.
The report notes that there was some difference of opinion among the reviewers about the details of the review. However, despite these technical issues of disagreement, the report asserts that EPA's proposed methodology has passed peer review.

Taking a look at the five experts on the EPA peer review panel shows that three are from environmental organizations that have called for a life-cycle analysis of corn ethanol (WRI, UCS, and TNC) and a fourth is an independent consultant who is primarily working with another environmental advocacy group that has also called for a life-cycle analysis of corn ethanol (ED). The fifth peer reviewer is a former employee of ICF (see CVs appended to report).

So 4 of 5 peer reviewers are employed by advocacy groups with established positions on the topic under review, and these position support that of the Obama Administration's EPA. At a minimum this situation creates a whale of an appearance problem, regardless of the scientific qualifications of the peer reviewers (which I am not at all discussing in this post).

For its part EPA and ICF claim in the report that:
the contractor and EPA determined that there were no direct and substantial COI or appearance of impartiality issues that would have prevented a peer reviewer’s comments from being considered by EPA.
This assertion does not pass a basic credulity test. The committee sure looks like it has members with strong biases falling in one direction and these biases may even rise to the level of a conflict of interest. The fact that the peer review resulted in support for the EPA was in fact conducted by a consulting firm paid tens of millions from EPA adds another layer of bad appearance.

How does an agency avoid such problems in the empanelment of scientific advisors?

Well, they should start by reading our BPC report issued this week (here in PDF), which tackles exactly these sorts of issues.

5 comments:

  1. Roger, this article makes me wonder if the only way to get good scientific discourse on a hot issue like this is to a) identify the people with agendas b) have them hire the best scientists to defend their position c) let them have at it in a public forum (web) where others could comment and ask questions. It seems like this would be more honest and get away from the almost impossible task you lined out in the BPC report of assessing the causes and degrees of people's lack of objectivity.

    ReplyDelete
  2. -1-Hi Sharon F

    What you describe is of course called "political debate"!

    The BPC report described in some detail the operational mechanisms for what I call in THB "science arbitration". That is, if you want to get experts to render their views on questions that can be resolved empirically, then our report outlines how you might go about do this.

    On controversial questions it becomes important to ensure that a panel is appropriately balanced with respect to issues of bias, which happens when scientists have strong views on an issue, and is almost always the case on highly politicized topics.

    EPA could have balanced its peer review committee with a few experts from the biofuels industry or academics on record opposing life-cycle analysis.

    ReplyDelete
  3. I would argue that the problem with the "hired science slingers" in the political debate is that there is no formalized kind of structure for debates on science so that "drive by" knowledge claims are common (not getting to the bottom of the disagreement- climate change is a good example- as we read on blogs).

    The same kind of structured transparent fact finding process that you described in BPC would be much easier if you just a) gave each side the same number of "expert slots", and b) had science mediators who made sure that the debates carried forward until the difference in claims and data were exhausted and could be summarized.

    In my experience, people who are involved in a scientific field have their own points of view and science societal agendas (my buddy is..) and expecting them to be objective is impossible. Expecting them to follow rules about supporting their inferences and holding them to it seems more doable. In my opinion.

    PS I have been reading about the process of "joint fact finding" in dispute resolution so my ideas are probably influenced by that body of work.

    ReplyDelete
  4. Dr. Pielke,

    In skimming your previous posts on Ethanol, it appears this is another area where you are --- indeed -- an “Honest Broker”.

    When evaluating anything coming out of the EPA these days, it might help to remember the political affiliations of the Czar currently lording over both EPA and DOE.

    Click here for my posts on Ethanol.

    ReplyDelete
  5. I am not sure I understand why Dr. Romm would raise the sales ranking of Honest Broker, when Hell and High Water appears to have done less well.
    Certainly his latest post is simply a display of bad manners.

    ReplyDelete