data vs dogma

Give a man a fish....

2010-07-07T23:40:29Z

Unfortunately I missed David Snowden’s recent ‘Making sense of complexity workshop’ but I was impressed with this response by the guys at the River Restoration Centre (http://bit.ly/98LbhP) which speaks about the difference between a cook who follows a recipe and a chief, who understands their ingredients and mixes them with style and flair to create a unique result.

I particularly agree that there is a tendency towards cooks in natural resource management in Australia - driven by a need to be transparent and tick the boxes. However, this is a fallacy, as true transparency and repeatability only comes if the chef (or practitioner) can adequately communicate and defend why they used a certain ingredient in the mix and why it was added in just that way.

I am often frustrated by being asked to provide a cookbook, or a toolkit for analysis and decision making in NRM. It is so much more important to understand your ‘ingredients’ - complexity, logic, multiple-criteria techniques, than having a black box recipe handed to you.

And just to labour the food idea, I suppose, for me it comes down to ‘give a man a fish and he’ll eat for a day, teach him to fish and he’ll eat forever’.

As consultants, it is imperative to not just give a recipe for a single decision, but to enable our clients to understand and value the decision making process, so that better decisions are made in the future in lots of different circumstances.

Cognitive Biases and decision making

2010-05-18T00:08:56Z

I’m sure anyone who works in the field of complex decision making is aware of the impact of personal bias.

I still remember a professor who spent six months and an excessive amount of grant money building more elaborate experimental rigs. He was testing a theory about the settling patterns of small particles in a fluid flow, and did not get the answer he had expected. For those keeping score, the laws of physics remained unimpressed with the new, expensive rigs and kept doing what they’d always done.

But decision making biases can be more insidious than that, especially in complex systems wiith high levels of uncertainty, like ecology or economic modelling/budget forecasting.

It’s true that good physical and stochastic models can go a long way to lower uncertainty, and multicriteria decision analysis can help in sorting and ranking a wide range of factors, simple cognitive bias can trump the most well planned decision framework.

The first step in eliminating these biases is the ability is to understand their existance. The link below is a beautifully rendered study guide to a range of cognitive biases. A couple of my “favourites”

The texas sharp shooter fallacy: The fallacy of selecting or adjusting an hypothesis after the data has been collected, which makes it impossible to fairly test the hypothesis. The name refers to the analogy of shooting a bunch of bullets into a wall, drawing a circle around a closely clustered set and declaring that was your target.
The zero-risk bias: The tendency to try to reduce a small risk to zero over acheiving a greater reduction in a larger risk.
Planning fallacy: The tendency to underestimate task completion times (honestly, who hasn’t commited this one?)

Hope you find it useful.

Cognitive Biases - A Visual Study Guide by the Royal Society of Account Planning

Introducing ANSA - A toolkit for analytics

2010-04-26T06:55:34Z

There is an old tale about Rolls Royce mechanics, in the golden era of motoring deep last century. The story goes that the first job an apprentice was given was to forge their own tools. Hand crafted tools were the best fit for the hand crafted Rolls’ and the best way to gain expertise.

For years now Symbolix have been working on delivering the silver bullet of analytics - a Toolkit. We have spoken with clients, delivered countless flowcharts and decision matrices along this journey.

One of the problems we’ve always faced is that powerful tools have steep learning curves. This is also the problem with every other tool kit out there. Most of our tools (software, processes and methods) have been hand crafted to some degree by our consultants, so we can be confident that we will find the best tool and apply it in the best way..

…but how could we provide our clients with the kind of analysis power we can access? How can we communicate the range and depth of models available (above and beyond your standard hypothesis tests)?

We have struggled at Symbolix to provide tools that are correct and generate insight, with a full duty of care attached to protect you the decision maker.

We now proudly introduce the ANSA - a toolkit for applied analytics.

ANSA (ANalytics Services & Applications) is based upon the concept of Software as a Service (SaaS), but has some benefits over such an approach.

Symbolix will continue to apply and supply the most powerful tool and techniques to your application. ANSA allows us to provide pay-per-use analytics engines, along with our traditional consulting services.

ANSA consists of suites of analysis tools, codes and engines, all with the usual duty of care promise. You are now free to choose your level of service to meet your budget and needs – from a complete custom analysis, through to sending us your database and receiving the model’s output in the mail.

You give us the data (or the problem, the expert opinions etc), we’ll give you the ANSA.

No more learning curve, no more stress, just a complete decision support service.

We’d love to know what you think, and if there are any additions you’d like to see…you can search our ANSA listing here

Of Drunkards and Lamp-posts

2010-04-07T00:09:56Z

It has been said of statistics that they” are often used as a drunkard uses a lamp post, more for support than illumination.”

It frequently falls to us to make an argument, based on a survey. To do this, we must first live with the data for a bit, getting a feel for its quirks and shortcomings along with its strengths. Unfortunately for all applied statisticians, rarely do we get the pleasure of translating data collected from our own design, where we have attempted to control all the confounding factors.

There is a growing school, driven amongst computer scientists and engineers for non-parametric studies of datasets that are generated without control- often called “data dredging.” It is the sometime harrowing and risky process of looking for patterns post-hoc, and then asserting a p value to their strength.

There are lots of discussions floating about out there regarding the relevance of a p value on a dredged pattern, usually along the lines of “well, given ‘something’ has to happen…” And they are valid. But I came across another concern that has left me perplexed.

We all know about the difficulty in interpreting cross tables, particularly 2X2, the staple of demographics.

But, dredging might make this even harder. Given a survey design, the interpretation of a contingency, or cross table, is relatively easy. You know which factor was controlled, and how the subjects were chosen. With a dredged set, how was the table propagated?

Let us presume that we have a 3X2 table. If we selected the data using a query that fixed the row totals, we could analyse it has two multinomials. Easy. If we fixed the column totals, we have three binomials. A different test but easy enough. If we extracted a fixed number of records (the grand total) and propagated the table, then each response cell becomes a Poisson variable. That’s more complicated, but doable. What about if there is another factor we didn’t think of? Something that might inflate the variance…….What I now have is a list of at least three different ways to analyse the same table, and in the case of a dredged set, no meaningful way to choose between them.

If anyone has any ideas, I’d love to hear them. If anyone feels like writing some theory for us foot soldiers, that will guide us when there is no data model.

And bemoaning dredging as a practice isn’t helpful…it is here to stay. We just need a rudder to help steer it.

It might just be worse than that

2010-03-09T23:39:07Z

I rarely get to blog here, so I am taking the opportunity to point out something above the general decline in mathematics students - the losing of knowledge.

Take a look at these feeds from the Australian:

Mathematics Students in Serious Decline

Equation for maths warns of disaster

I was teaching at University when we as a society ripped these students off, replacing core problem solving with vapid histories and philosophies. Such a change had a lot to do with the academia of mathematics becoming disconnected with the application of its theory. And this is where the problem lies.

I’ll give you an example. I sought to take a Graduate Certificate in Applied Statistics recently. Not because I wanted another piece of paper, but because I am looking for some more knowledge to protect myself and my clients from stupid misapplications of theory. I couldn’t find one that was more than a simple training course in certain statistical software packages. Needless to say, I haven’t enrolled. I need, as do all analysts more than the mindless application of a software package.

The need arose when the tried and proven ANOVA test failed on me. ANOVA is the draught horse of multiple testing applications. I had a test returning p values that experience tells were way too small. The underlying data, was violating a number of assumptions of the ANOVA test, and I could find a transform that would fix it. I still haven’t found a transform, and have had to move on without that test, making my story that much longer as I now have to justify the use of “unorthodox” testing procedures.

This is not the first time that I’ve come across underlying short-comings in standard procedures. My argument comes from the fact that often I have to go right back to 1930’s papers by deities like Fisher, or early works by Tukey to find underlying mechanics and discussions. Far too many papers simply state the software package they used, and the outcomes, never addressing whether the package should have been applied in the first place.

I wonder how often it is that an assumption has been made regarding validity, and never checked.

Join us for a cuppa at the Clean Energy Council Conference

2010-02-23T05:58:17Z

Symbolix is proud to be a sponsor of the upcoming Clean Energy Council National Conference. It’s being held in Adelaide from the 3rd-5th May 2010.

The conference is attended by over 700 delegates from all areas of the clean energy sector, and promises some interesting discussion and insight.

We’ll be there throughout, and are sponsoring a break time on Tuesday, so drop down to the exhibition hall and have a coffee on us.

The median stripped bare? Well....

2010-01-31T22:20:54Z

The Age newspaper today published an article analysing differences in the way different market research companies report the median selling price for different suburbs. This is an important point to discuss, but I was not concerned by the analysis as much as this definition of median:

“However, it is worth keeping in mind that the median price is not the same as thing as the average price. It is simply the middle sale price when all property sales are arranged chronologically.”

Um, no actually. For a given month, the middle sale price when sales are arranged chonologically (in time) would be the price received around the 15th of the month (assuming an even sales rate through the month).

For the record, here are the definitions you need. When we talk “average” we may mean one of three measures.

The median is the middle value, when prices are arranged in order from lowest to highest.
The mean (most commonly just called the average) is just the sum of all the prices, divided by the number of sales.
The mode is the most common sale price.

In many cases, these three measures are very similar, but not always.

It’s worth noting that the mean is highly susceptible to outliers - a one off $10 million dollar property sale will inflate the mean price, but leave the median less affected. This is why the median is a more stable measure of things like house prices, which are likely to have a number of small outliers (very low or very high prices).

Now that’s settled, the rest of the article is worth reading - it discusses why understanding the drivers behind changes in these measures is so important. For example, if a jump in median house prices reflects a drive by investors moving on high end properties, it does not necessarily translate to making a killing on selling a low end property to a young first buyer market.

This is important to think about, but the first step is to understand the basics of what the measures actually mean.

The hidden danger of averages

2009-12-15T22:38:20Z

At 7:51am this morning on ABC Melbourne, as part of the news broadcast, it was mentioned that today’s forest fire index rating for the state is about 47 http://bit.ly/6cFYSt

Given that this is generated from at least five numbers (representing the five fire forecast regions), this tells us next to nothing about the actual risk.

Let’s assume that of the five regions four of them have a fire index of 20 (negligible) whilst the remaining region has an index of 150 (comparable to Ash Wednesday). This gives the STATE a fire index average of 46. However for that one region with a Code Red – Catastrophic rating, any fire that starts will be devastating, but the state as a whole will be fine.

While it is impossible to announce the fire index rating for each and every city, town and locality in the state, giving a state-wide ‘average’ index is pointless. The state was divided into the five fire forecast regions because they were identified as having very separate fire risks; shouldn’t they be reported individually?

In a weather report there are eight capital cities which get mentioned, and to give the current forecast for all eight takes less than a minute when done expeditiously. Surely it would take less time again to broadcast the fire index ratings for each of the five regions individually.

Averages are incredibly useful tools when used properly, I myself use them many times each day. The forest fire index should NOT be given as a state wide average, it masks the true risk to the state. There once was a man who drowned crossing a river that had an average depth of 3 feet…

http://www.flawofaverages.com/

24 hour time is NOT decimal

2009-12-09T05:12:39Z

If I had to do this only once in my data crunching life I could excuse that. However this is now the third time in a year that I’ve had to perform this same miraculous transformation of time.

When recording data, I love it when people think to use a 24hr time format, as this saves a lot of messing around trying to work out if it’s meant to be 6AM or 6PM, it’s somewhat more difficult to get 6 and 18 confused. However…there’s a worrying trend starting to appear in the data that crosses my desk. If you make a note of something at quarter past two in the afternoon, please, for the love of data crunching monkeys EVERYWHERE, write it as “14:15”. That magical extra dot which transforms a decimal point into a colon saves many hours of headaches when checking data prior to analysis. “14.15” and “14:15” are very different values when you’re working with times.

If you insist on using decimal points in time, please us a Julian Date format instead of a normal 12 or 24 hour format. Julian Dates are MEANT to have decimal places, and data crunchers should be just as happy with them as normal times.

Wandering in the (data) Wilderness

2009-11-18T04:06:54Z

I was working on a proposal for a significant firm to look at employing some pretty hefty business analytics. I was (naturally) laying out a project plan, and noticed I was spending a lot of time in a risk mitigation phase of determining the underlying quality of the data sources.

The roadmap we ended up with looked a lot like that for an old-world exploratory expedition (see below). This was the only way I could ensure, with reasonable confidence, the quality of the potential outcomes for the project. It involved identifying the desired goal (Gulf of Carpentaria), the potential routes and risks (Dry land up the middle, with not much water), and most importantly, the current status.

It occurred to me that the quickest, quickest way to get lost on a (data) mining expedition, was to not know where you were initially. That is, to get lost it’s best to start lost. Given you are going to be traversing uncharted territory in most mining applications, it suddenly becomes really important to make sure you start out from where you thought you were starting out from. Otherwise you can get a very expensive, random walk out into the aether, because all the assumptions behind your planned route map will be nonsense.

It also highlighted for me how crucial it is to treat the technology with great care. To continue the analogy, if you start of your GPS way point path at the wrong spot, the little handheld unit will be dead certain you are at the Sydney Harbour Bridge, oblivious to the large rock and red dust in front of you. Just so, a poorly fired analytic will tell you with absolute confidence that you should change the Call Centre settings….but if the initial data source wasn’t what you thought it was, you might find yourself standing in a wind blown desert wondering what went wrong.