© Janet Hastings | Dreamstime.comGetting it wrong (which happens if the objective of the survey/census is not well articulated) can produce results that greatly misrepresent risk.  The two types can be thought of as subject focussed and hazard focussed surveys.

Let me give you a couple of examples.

Survey type 1 – Subject focussed.

Objective:  to test a hypothesis related to kangaroo diet. An appropriate survey design here would be to survey (or census) all the kangaroos in the region of interest.

Survey type 2 – Hazard focussed

Objective: to determine the number of kangaroo collisions during summer on some stretch of road.  Here, a census of kangaroos would use huge resources, and may produce biased results (not all the kangaroos surveyed go near the road in question, let alone get into an altercation with a car).© Dirkr | Dreamstime.com

Here we require a hazard focussed survey frame, i.e. we would survey (or census) the stretch of road(s).

These examples may be trivial, but a confusion between subject and hazard focussed surveys is a real and overlooked issue in many environmental surveys, particularly those involving rare or cryptic species.

The traditional approach in ecology is to undertake behavioural/biological surveys – to determine features of the species, or their interaction with their environment.  These surveys are usually designed to maximise the observer’s likelihood to actually observe the species.  The more individuals observed, the more data you can collect on morphology, behaviour etc etc etc.

However, in sustainable planning we are interested in determining

• The likelihood of an impact (how many critters will be hit by cars, lose their breeding ground, fly into a turbine, etc?)
• The consequence of an impact (this is done through population viability analysis if the population is geographically isolated, or through various harvest rates for species that range further). 

Determining the likelihood of the impact is akin to determining the impact of a new road on local kangaroo population.  We must focus our surveys on the hazard, not the subject.  This involves stepping away from traditional subject based surveys.

If this is not done, than you may have a wonderful data set filled with observations of species behaviour down in that valley over there, but no insight at all into the risk posed by the road up on the ridge here.  Without that insight, management is impossible and compliance is reliant on subjective assessment. 



I particularly agree that there is a tendency towards cooks in natural resource management in Australia - driven by a need to be transparent and tick the boxes.  However, this is a fallacy, as true transparency and repeatability only comes if the chef (or practitioner) can adequately communicate and defend why they used a certain ingredient in the mix and why it was added in just that way.

I am often frustrated by being asked to provide a cookbook, or a toolkit for analysis and decision making in NRM.  It is so much more important to understand your ‘ingredients’ - complexity, logic, multiple-criteria techniques, than having a black box recipe handed to you.

And just to labour the food idea, I suppose, for me it comes down to ‘give a man a fish and he’ll eat for a day, teach him to fish and he’ll eat forever’.

As consultants, it is imperative to not just give a recipe for a single decision, but to enable our clients to understand and value the decision making process, so that better decisions are made in the future in lots of different circumstances.

I still remember a professor who spent six months and an excessive amount of grant money building more elaborate experimental rigs.  He was testing a theory about the settling patterns of small particles in a fluid flow, and did not get the answer he had expected.  For those keeping score, the laws of physics remained unimpressed with the new, expensive rigs and kept doing what they’d always done.

But decision making biases can be more insidious than that, especially in complex systems wiith high levels of uncertainty, like ecology or economic modelling/budget forecasting. 

It’s true that good physical and stochastic models can go a long way to lower uncertainty, and multicriteria decision analysis can help in sorting and ranking a wide range of factors, simple cognitive bias can trump the most well planned decision framework.

The first step in eliminating these biases is the ability is to understand their existance.  The link below is a beautifully rendered study guide to a range of cognitive biases.  A couple of my “favourites”

• The texas sharp shooter fallacy: The fallacy of selecting or adjusting an hypothesis after the data has been collected, which makes it impossible to fairly test the hypothesis.  The name refers to the analogy of shooting a bunch of bullets into a wall, drawing a circle around a closely clustered set and declaring that was your target.
• The zero-risk bias: The tendency to try to reduce a small risk to zero over acheiving a greater reduction in a larger risk.
• Planning fallacy: The tendency to underestimate task completion times (honestly, who hasn’t commited this one?)

Hope you find it useful.

Cognitive Biases - A Visual Study Guide by the Royal Society of Account Planning

It frequently falls to us to make an argument, based on a survey. To do this, we must first live with the data for a bit, getting a feel for its quirks and shortcomings along with its strengths. Unfortunately for all applied statisticians, rarely do we get the pleasure of translating data collected from our own design, where we have attempted to control all the confounding factors.

There is a growing school, driven amongst computer scientists and engineers for non-parametric studies of datasets that are generated without control- often called “data dredging.” It is the sometime harrowing and risky process of looking for patterns post-hoc, and then asserting a p value to their strength.

There are lots of discussions floating about out there regarding the relevance of a p value on a dredged pattern, usually along the lines of “well, given ‘something’ has to happen…” And they are valid. But I came across another concern that has left me perplexed.

We all know about the difficulty in interpreting cross tables, particularly 2X2, the staple of demographics.

But, dredging might make this even harder. Given a survey design, the interpretation of a contingency, or cross table, is relatively easy. You know which factor was controlled, and how the subjects were chosen. With a dredged set, how was the table propagated?

Let us presume that we have a 3X2 table. If we selected the data using a query that fixed the row totals, we could analyse it has two multinomials. Easy. If we fixed the column totals, we have three binomials. A different test but easy enough. If we extracted a fixed number of records (the grand total) and propagated the table, then each response cell becomes a Poisson variable. That’s more complicated, but doable. What about if there is another factor we didn’t think of? Something that might inflate the variance…….What I now have is a list of at least three different ways to analyse the same table, and in the case of a dredged set, no meaningful way to choose between them.

If anyone has any ideas, I’d love to hear them. If anyone feels like writing some theory for us foot soldiers, that will guide us when there is no data model.

And bemoaning dredging as a practice isn’t helpful…it is here to stay. We just need a rudder to help steer it.

Take a look at these feeds from the Australian:

Mathematics Students in Serious Decline

Equation for maths warns of disaster

I was teaching at University when we as a society ripped these students off, replacing core problem solving with vapid histories and philosophies. Such a change had a lot to do with the academia of mathematics becoming disconnected with the application of its theory. And this is where the problem lies.

I’ll give you an example. I sought to take a Graduate Certificate in Applied Statistics recently. Not because I wanted another piece of paper, but because I am looking for some more knowledge to protect myself and my clients from stupid misapplications of theory. I couldn’t find one that was more than a simple training course in certain statistical software packages. Needless to say, I haven’t enrolled. I need, as do all analysts more than the mindless application of a software package.

The need arose when the tried and proven ANOVA test failed on me. ANOVA is the draught horse of multiple testing applications. I had a test returning p values that experience tells were way too small. The underlying data, was violating a number of assumptions of the ANOVA test, and I could find a transform that would fix it. I still haven’t found a transform, and have had to move on without that test, making my story that much longer as I now have to justify the use of “unorthodox” testing procedures.

This is not the first time that I’ve come across underlying short-comings in standard procedures. My argument comes from the fact that often I have to go right back to 1930’s papers by deities like Fisher, or early works by Tukey to find underlying mechanics and discussions. Far too many papers simply state the software package they used, and the outcomes, never addressing whether the package should have been applied in the first place.

I wonder how often it is that an assumption has been made regarding validity, and never checked.

The conference is attended by over 700 delegates from all areas of the clean energy sector, and promises some interesting discussion and insight.

We’ll be there throughout, and are sponsoring a break time on Tuesday, so drop down to the exhibition hall and have a coffee on us.

“However, it is worth keeping in mind that the median price is not the same as thing as the average price. It is simply the middle sale price when all property sales are arranged chronologically.”

Um, no actually.  For a given month, the middle sale price when sales are arranged chonologically (in time) would be the price received around the 15th of the month (assuming an even sales rate through the month).

For the record, here are the definitions you need.  When we talk “average” we may mean one of three measures.

• The median is the middle value, when prices are arranged in order from lowest to highest.
• The mean (most commonly just called the average) is just the sum of all the prices, divided by the number of sales.
• The mode is the most common sale price. 

In many cases, these three measures are very similar, but not always.

It’s worth noting that the mean is highly susceptible to outliers - a one off $10 million dollar property sale will inflate the mean price, but leave the median less affected.  This is why the median is a more stable measure of things like house prices, which are likely to have a number of small outliers (very low or very high prices).

Now that’s settled, the rest of the article is worth reading - it discusses why understanding the drivers behind changes in these measures is so important.  For example, if a jump in median house prices reflects a drive by investors moving on high end properties, it does not necessarily translate to making a killing on selling a low end property to a young first buyer market.

This is important to think about, but the first step is to understand the basics of what the measures actually mean.

At 7:51am this morning on ABC Melbourne, as part of the news broadcast, it was mentioned that today’s forest fire index rating for the state is about 47 http://bit.ly/6cFYSt

Given that this is generated from at least five numbers (representing the five fire forecast regions), this tells us next to nothing about the actual risk.

Let’s assume that of the five regions four of them have a fire index of 20 (negligible) whilst the remaining region has an index of 150 (comparable to Ash Wednesday).  This gives the STATE a fire index average of 46.  However for that one region with a Code Red – Catastrophic rating, any fire that starts will be devastating, but the state as a whole will be fine.

While it is impossible to announce the fire index rating for each and every city, town and locality in the state, giving a state-wide ‘average’ index is pointless.  The state was divided into the five fire forecast regions because they were identified as having very separate fire risks; shouldn’t they be reported individually?

In a weather report there are eight capital cities which get mentioned, and to give the current forecast for all eight takes less than a minute when done expeditiously.  Surely it would take less time again to broadcast the fire index ratings for each of the five regions individually.

Averages are incredibly useful tools when used properly, I myself use them many times each day.  The forest fire index should NOT be given as a state wide average, it masks the true risk to the state.  There once was a man who drowned crossing a river that had an average depth of 3 feet…

http://www.flawofaverages.com/



Related: More on Models, interpretation and black swans (http://symbolix.squarespace.com/blog/2009/1/29/dont-bash-the-statistician-black-boxes-and-black-swans.html)

The roadmap we ended up with looked a lot like that for an old-world exploratory expedition (see below). This was the only way I could ensure, with reasonable confidence, the quality of the potential outcomes for the project. It involved identifying the desired goal (Gulf of Carpentaria), the potential routes and risks (Dry land up the middle, with not much water), and most importantly, the current status.

It occurred to me that the quickest, quickest way to get lost on a (data) mining expedition, was to not know where you were initially. That is, to get lost it’s best to start lost. Given you are going to be traversing uncharted territory in most mining applications, it suddenly becomes really important to make sure you start out from where you thought you were starting out from. Otherwise you can get a very expensive, random walk out into the aether, because all the assumptions behind your planned route map will be nonsense.

It also highlighted for me how crucial it is to treat the technology with great care. To continue the analogy, if you start of your GPS way point path at the wrong spot, the little handheld unit will be dead certain you are at the Sydney Harbour Bridge, oblivious to the large rock and red dust in front of you. Just so, a poorly fired analytic will tell you with absolute confidence that you should change the Call Centre settings….but if the initial data source wasn’t what you thought it was, you might find yourself standing in a wind blown desert wondering what went wrong.

I was interested to read about the chief drugs advisor to the UK government being sacked for speaking out against the government’s drug policy. Professor David Nutt argued that cannabis had been reclassified as a higher risk drug against scientific evidence and was promptly sacked for commenting on policy. 

He is quoted as saying: “So this government has made the law and then it has gone back to the advisory council and said ‘could you find some evidence to support our decision?’  Now we’ve said ‘no, we will stick to science, our scientific guns, we will produce the evidence and if you go and legislate inappropriately, we will continue to point out the evidence does not support you’.” (http://bit.ly/VUJpW)

This quote echoed a number of conversations that took place at the Enivironment Institute of Australia and New Zealand Conference last week.  In particular, there was a lot of frustration at groups and individuals that don’t want a development in their locality (for whatever cultural, social or personal reason) and who build (often spurious) scientific arguments to validate their opinion. 

Considerable time and effort is often spent in putting together good, unbiased scientific evidence to address the “scientific” concerns, only to be ignored, deflected and demeaned, because it does not support the community sentiment.

Too often this situation decends into a “my expert is better than your expert” skirmish, and noone wins.  And you may be surprised, but as a scientific consultant, my livelihood is dependant on me giving my clients the best evidence-based answer, not the one they want.  And it’s a kick in the guts to go into those situations in good faith and wanting a good scientific debate, only to have my scientific integrity brought into question. 

And you know what I reckon?

I think that it is actually valid for a community to stand up and say, “we have heard the evidence, we accept the value of the proposal but we don’t want it because it will make my backyard ugly/change the nature of our community/change the road I take to work”.  Politicians should stand up and say “that is a fair and reasonable scientific argument, but parents in my constituency just don’t want another thing to worry about, no matter the risk, so we will go against the scientific advice for now.”

It’s ok to have social, political, religious or emotional responses, and for these to be taken into account.  But they cannot be taken into account, if they are not out on the table.

Just don’t use science as a cover up.  Thanks.