Former Navy tactical jet pilot analyzes the Aerial Use and Effectiveness study

Calwood Fire, Oct. 17, 2020
Calwood Fire northwest of Boulder, Colorado, Oct. 17, 2020. Photo by Ben Nelson, Envision Studio, Boulder, Colorado.

(Note from Bill: I asked Bean Barrett, a former Naval Aviator and frequent contributing writer on Fire Aviation, for his thoughts about the Aerial Firefighting Use and Effectiveness study that was conducted by the U.S. Forest Service during a nine-year period at an annual cost of $1.3 million. The AFUE had very ambitious goals initially when Tom Harbour was the Director of Fire and Aviation for the U.S. Forest Service. “AFUE was initially intended to eventually help answer questions about the size and composition of aviation assets needed by the USFS,” Mr. Harbour told Fire Aviation recently. Bean’s comments are below. It begins with the Government Accountability Office’s request to the USFS to justify spending about half a billion dollars a year for aerial firefighting programs. That was one of the reasons for conducting the study.)


What the Government Accountability Office recommended :

  1. Expand efforts to collect information on aircraft performance and effectiveness to include all types of firefighting aircraft in the federal fleet.
  2. Enhance collaboration between the agencies and with stakeholders in the fire aviation community to help ensure that Agency efforts to identify the number and type of firefighting aircraft they need reflect the input of all stakeholders in the fire aviation community.
  3. Subsequent to the completion of the first two recommendations, update the agency’s strategy documents for providing a national firefighting aircraft fleet to include analysis based on information on aircraft performance and effectiveness and to reflect input from stakeholders throughout the fire aviation community.

The AFUE study says it provide “…empirical information addressing the three GAO recommendations…”

From a program management perspective, enough information is required to inform decisions on inventory objectives, inventory types, and to allow costing out various aircraft and contract options.

At a higher level, when dealing at the programmatic and budget levels, it is necessary to first clearly demonstrate the value of aircraft vs ground crews and equipment or the appropriate mix of aircraft, ground crews, and equipment. One departmental budget buys all. When the value of aircraft can be demonstrated empirically, the desired mix of air and ground capability can be decided. Then, aircraft inventory objectives by type can be determined. AFUE picks up at this point and I’m not sure the larger case for air has been made. We cannot answer the question … In general, how much more efficient are our firefighting efforts when ground crews are augmented with air tanker support?

With respect to GAQ tasking, AFUE has only taken a partial first step in addressing and providing the necessary information for GAO recommendation #1. There is not enough information to determine desired inventory objective, type and mix of types, and cost. I’m not sure it is presented in a form to allow conclusions to be made. As the information was collected across the various LMA’s it does comply with #2. Without fully accomplishing task #1, updating strategy documents is not possible #3.

The Performance definition utilizes two metrics. Interaction percentage [IP] and probability of success [POS]. The definition of those two performance measurements and how they relate to GAO’s recommendation to collect information on “Performance” and Effectiveness” is not clear. I’m not sure Performance and Effectiveness by type aircraft can be adequately determined.

This study is perhaps being too clever by half when it introduces new terms to analyze.

What constitutes satisfactory interaction in Interaction Percentage [IP]? Probability of Success [POS] seems to be conflated with “effectiveness”. What constitutes success? What constitutes “effective”?

Some questions:

If “Each drop was analyzed on the basis of objective and outcome” with respect to use statistics, was the air asset chosen by the IMT or was it used because it was all that was available? If it was a beggars can’t be choosers tasking situation, some of the aircraft-mission paring data might be questioned.

We may have a better idea of what groups of aircraft are tasked for which missions but not why a particular group is being chosen/used. The statistics clearly show helos flying over 80% of the missions. Is that a reflection of IMT choice, the practicality of using what was available, the preponderance of water as a payload, or perhaps it is related to the ratio of small fires to large fires? There is insufficient information to determine why helos flew over 80% of the sorties.

How much of IP and POS was a function of drop accuracy? Which aircraft were the most accurate? What missions are more accuracy dependent?

How much of the IP and POS statistics are more dependent on payload than aircraft type?

Fire is the target. It is why aircraft are flying. Is classification of fire into IA, EA, and Large as defined in the study sufficient to sort aircraft assignment and results by type? EA appears to be a small gap filler between IA and Large Fire.

Why are there no time dependent metrics? From a program analysis perspective, many aircraft distinguishing characteristics are time related. Speed, range, endurance, reload-return, and IA response come to mind.

If this were a commercial airline study, I would expect to see some “bottom line” characteristic similar to Revenue/Seat-Mile. Why isn’t there something like cost/payload gallon-mile/hr for air tankers?

From a commercial and military perspective, it is more about payload delivered and what it takes to get it there. Aircraft are mission dependent variables. Why wasn’t the AFUE study based on mission requirements? What payload needed to be delivered, what quantity, what coverage level, how fast, how accurately, and for how long? Given something like these parameters, the number and type of aircraft can be determined.

A problem?

Figure 10. With LAT’s approximately 45% of drops have a positive outcome. For VLAT’s 50%.

Drop Outcomes, AFUE
Figure 10, Drop Outcomes, AFUE.

Figure 13. LATs have a 67-68% Probability of Success. VLATs have a 70% Probability of Success.

Figure 13, Probability of Success, AFUE
Figure 13, Probability of Success, AFUE

Someone smarter than me needs to explain this discrepancy. Evidently a positive outcome is not necessary to qualify as a success.

Conclusion:

The original GAO requirement was for information on “performance” and “effectiveness” to support determination of what constituted an adequate aircraft firefighting fleet. From a program manager’s standpoint, this study does not provide the necessary information to determine aircraft inventory objectives or mix of types.

AFUE made an assumption — they assumed that what we are doing today with aircraft is mostly correct. This going-in assumption resulted in missing an examination of “Are we doing it the right way? Are we using the right assets? Or, are we just doing the best we can and getting by with what we have?” The answers to these questions are necessary to inform the data collection process and objectives. It is highly likely that the AFUE data more closely reflects firefighting that is making do with what it has, as opposed to selecting the best aircraft for the job.

Australia has done a far better job at justifying use of firefighting aircraft and examining aircraft effectiveness and performance.

Going to your e-mail questions.

  • Is it useful? Aircraft types are grouped so nothing can be determined about specific aircraft except in a most general nature based on Tanker Type Categories.
  • Adequate comparison? Data is insufficient or entirely missing.
  • Future program decision support? Not useful in determining aircraft inventory and types for a national air firefighting fleet. Also not adequate to support program decisions because it never answered the “why” various types were chosen for various missions.
  • Cost metric. As mentioned in my missive, it’s about payloads required/delivered. That is a better sorting function than Tanker Type. Aircraft Type and #’s are dependent variables of payload and payload related requirements.
  • Aircraft Mix going forward…

My familiarity is somewhat limited. I would like to think I understand some of aviation based on my 31 years of military and some commercial experience but since I’m on the board of directors of a 98 square mile fire protection district in Colorado (the district with the community at highest assessed risk of loss to wildfire in the whole state of Colorado) my immediate day-to-day scope is much more focused on things like IA, what it takes for our department to keep small fires small, and our citizens safe.

In Colorado it is the small fire protection districts that are primarily responsible for IA.

If something like 95-96% of our wildfires fires are contained then the approximately 95-96 % containment was probably due to effective IA. Taking a couple more percent off of the escaped fire percentage is far cheaper than fighting a large fire.

Maybe more unique to Colorado, but almost every district could use more staff for IA (primarily volunteer departments) and larger budgets for equipment. Neither one is in the cards. We need close air support to overcome the obstacles of marginally accessible terrain and shortage of staff. Small, accurate, and timely air support comes to mind.

Like a Marine gunny I knew once observed — when I need close air support I don’t want a B-52, I usually want something like an A-10 with a gun.

We are short of big lakes around here so scoopers are out. Dipping/snorkeling helos come to mind to directly support the small crews that are responsible for getting the initial attack going. SEAT’s reload-return times are too long.

Somewhere in the bean counting world there is a very good reason a LAT or VLAT cannot be made available for every 12-20 man module/ crew working an IA. Locally, we need more helos. Who owns the helos isn’t important. It is the response time and cycle time that counts.

We need the bigger tankers for the 4 or 5% of the fires that do escape.

If I had to do a rough guess on the aviation mix, I’d start on the ground floor with what might be considered “first principles” and look at the targets list… (how many fires, how big, where, how often), then figure out what needs to be dropped on the various targets and generally, where the targets are and if the missions are direct support or indirect support. Determine payload type, quantity, and necessary accuracy. Then, look at the best aviation mix to deliver the required payloads.


Bean Barrett

  • 31 years as a Naval Aviator flying tactical jets, some heavy time, and a little prop time.
  • Flew for Continental Airlines for two years.
  • Aviation Safety Officer and mishap investigator.
  • Staff time at the Navy Strike Warfare Center/ included work on close air support training.
  • Navy sponsor on Navy Staff for some air and space programs. Total annual obligation authority $1B.
  • Presently on our local Colorado fire protection district board of directors. Have worked with our fire department for over 10 years.
  • Lived in Colorado’s front range tinderbox since 1979. Have seen several of our front range big fires up close and personal. Evacuated for one.

Typos, let us know, and please keep in mind the commenting ground rules before you post a comment.

4 thoughts on “Former Navy tactical jet pilot analyzes the Aerial Use and Effectiveness study”

  1. Helicopters with water sources close by is the key for cost effective initial attack. Put a water tender with a pumpkin on every initial response with the closest helicopter, or attach one to the contract. Not the answer for every situation, but for wui fires, water and tenders should be readily available.

    On another note pertaining to air tankers, large air tankers are for large fires, in my observation, because not many units are going to send large air tankers on initial attack.

    1. Ive written a whole conceptual paper on how we can use tankers more efficiently and i call it the fire bridge cool it retard it then doze it.

      Fires produce btu and water absorbs, LC95 has no established btu/lb rating

      Fire ground hydraulics is what we need to start teaching people

  2. Keep air tankers loaded with water as it will not set up like retardant. Hit any fire when it starts and ask questions later. Canadian air tankers once said CALL US FIRST AND THE FIRES WILL BE PUT OUT. No one seems interested in putting out fires when small as only large fires provide big money for everyone.

  3. “Maybe more unique to Colorado, but almost every district could use more staff for IA (primarily volunteer departments) and larger budgets for equipment. Neither one is in the cards. We need close air support to overcome the obstacles of marginally accessible terrain and shortage of staff. Small, accurate, and timely air support comes to mind”.

    This is true for all volunteer departments I have observed. That statement applies to federal agencies and their local land management units as well. There are simply not enough Initial Attack resources, strategically located, to respond to a fire and stop it. But isn’t that the goal?

    There are areas in the western United States where more helicopters could improve the success of IA, but they lack a water source, or an adequate water source is too far from the fire, or alternate water sources (portable tanks & water tenders) take too long to set up or are in short supply.

    The debate for use of aerial resources to fight fires is mostly a numbers game. Those who are familiar with firefighting are aware that the debate is also about they type of resources used. I imagine a fire season without air tankers, or helicopters to support ground operations would provide more definitive data and visual evidence of their value for firefighting.

Comments are closed.