data vs dogma

It might just be worse than that

2010-03-09T23:39:07Z

I rarely get to blog here, so I am taking the opportunity to point out something above the general decline in mathematics students - the losing of knowledge.

Take a look at these feeds from the Australian:

Mathematics Students in Serious Decline

Equation for maths warns of disaster

I was teaching at University when we as a society ripped these students off, replacing core problem solving with vapid histories and philosophies. Such a change had a lot to do with the academia of mathematics becoming disconnected with the application of its theory. And this is where the problem lies.

I’ll give you an example. I sought to take a Graduate Certificate in Applied Statistics recently. Not because I wanted another piece of paper, but because I am looking for some more knowledge to protect myself and my clients from stupid misapplications of theory. I couldn’t find one that was more than a simple training course in certain statistical software packages. Needless to say, I haven’t enrolled. I need, as do all analysts more than the mindless application of a software package.

The need arose when the tried and proven ANOVA test failed on me. ANOVA is the draught horse of multiple testing applications. I had a test returning p values that experience tells were way too small. The underlying data, was violating a number of assumptions of the ANOVA test, and I could find a transform that would fix it. I still haven’t found a transform, and have had to move on without that test, making my story that much longer as I now have to justify the use of “unorthodox” testing procedures.

This is not the first time that I’ve come across underlying short-comings in standard procedures. My argument comes from the fact that often I have to go right back to 1930’s papers by deities like Fisher, or early works by Tukey to find underlying mechanics and discussions. Far too many papers simply state the software package they used, and the outcomes, never addressing whether the package should have been applied in the first place.

I wonder how often it is that an assumption has been made regarding validity, and never checked.

Join us for a cuppa at the Clean Energy Council Conference

2010-02-23T05:58:17Z

Symbolix is proud to be a sponsor of the upcoming Clean Energy Council National Conference. It’s being held in Adelaide from the 3rd-5th May 2010.

The conference is attended by over 700 delegates from all areas of the clean energy sector, and promises some interesting discussion and insight.

We’ll be there throughout, and are sponsoring a break time on Tuesday, so drop down to the exhibition hall and have a coffee on us.

The median stripped bare? Well....

2010-01-31T22:20:54Z

The Age newspaper today published an article analysing differences in the way different market research companies report the median selling price for different suburbs. This is an important point to discuss, but I was not concerned by the analysis as much as this definition of median:

“However, it is worth keeping in mind that the median price is not the same as thing as the average price. It is simply the middle sale price when all property sales are arranged chronologically.”

Um, no actually. For a given month, the middle sale price when sales are arranged chonologically (in time) would be the price received around the 15th of the month (assuming an even sales rate through the month).

For the record, here are the definitions you need. When we talk “average” we may mean one of three measures.

The median is the middle value, when prices are arranged in order from lowest to highest.
The mean (most commonly just called the average) is just the sum of all the prices, divided by the number of sales.
The mode is the most common sale price.

In many cases, these three measures are very similar, but not always.

It’s worth noting that the mean is highly susceptible to outliers - a one off $10 million dollar property sale will inflate the mean price, but leave the median less affected. This is why the median is a more stable measure of things like house prices, which are likely to have a number of small outliers (very low or very high prices).

Now that’s settled, the rest of the article is worth reading - it discusses why understanding the drivers behind changes in these measures is so important. For example, if a jump in median house prices reflects a drive by investors moving on high end properties, it does not necessarily translate to making a killing on selling a low end property to a young first buyer market.

This is important to think about, but the first step is to understand the basics of what the measures actually mean.

The hidden danger of averages

2009-12-15T22:38:20Z

At 7:51am this morning on ABC Melbourne, as part of the news broadcast, it was mentioned that today’s forest fire index rating for the state is about 47 http://bit.ly/6cFYSt

Given that this is generated from at least five numbers (representing the five fire forecast regions), this tells us next to nothing about the actual risk.

Let’s assume that of the five regions four of them have a fire index of 20 (negligible) whilst the remaining region has an index of 150 (comparable to Ash Wednesday). This gives the STATE a fire index average of 46. However for that one region with a Code Red – Catastrophic rating, any fire that starts will be devastating, but the state as a whole will be fine.

While it is impossible to announce the fire index rating for each and every city, town and locality in the state, giving a state-wide ‘average’ index is pointless. The state was divided into the five fire forecast regions because they were identified as having very separate fire risks; shouldn’t they be reported individually?

In a weather report there are eight capital cities which get mentioned, and to give the current forecast for all eight takes less than a minute when done expeditiously. Surely it would take less time again to broadcast the fire index ratings for each of the five regions individually.

Averages are incredibly useful tools when used properly, I myself use them many times each day. The forest fire index should NOT be given as a state wide average, it masks the true risk to the state. There once was a man who drowned crossing a river that had an average depth of 3 feet…

http://www.flawofaverages.com/

24 hour time is NOT decimal

2009-12-09T05:12:39Z

If I had to do this only once in my data crunching life I could excuse that. However this is now the third time in a year that I’ve had to perform this same miraculous transformation of time.

When recording data, I love it when people think to use a 24hr time format, as this saves a lot of messing around trying to work out if it’s meant to be 6AM or 6PM, it’s somewhat more difficult to get 6 and 18 confused. However…there’s a worrying trend starting to appear in the data that crosses my desk. If you make a note of something at quarter past two in the afternoon, please, for the love of data crunching monkeys EVERYWHERE, write it as “14:15”. That magical extra dot which transforms a decimal point into a colon saves many hours of headaches when checking data prior to analysis. “14.15” and “14:15” are very different values when you’re working with times.

If you insist on using decimal points in time, please us a Julian Date format instead of a normal 12 or 24 hour format. Julian Dates are MEANT to have decimal places, and data crunchers should be just as happy with them as normal times.

Wandering in the (data) Wilderness

2009-11-18T04:06:54Z

I was working on a proposal for a significant firm to look at employing some pretty hefty business analytics. I was (naturally) laying out a project plan, and noticed I was spending a lot of time in a risk mitigation phase of determining the underlying quality of the data sources.

The roadmap we ended up with looked a lot like that for an old-world exploratory expedition (see below). This was the only way I could ensure, with reasonable confidence, the quality of the potential outcomes for the project. It involved identifying the desired goal (Gulf of Carpentaria), the potential routes and risks (Dry land up the middle, with not much water), and most importantly, the current status.

It occurred to me that the quickest, quickest way to get lost on a (data) mining expedition, was to not know where you were initially. That is, to get lost it’s best to start lost. Given you are going to be traversing uncharted territory in most mining applications, it suddenly becomes really important to make sure you start out from where you thought you were starting out from. Otherwise you can get a very expensive, random walk out into the aether, because all the assumptions behind your planned route map will be nonsense.

It also highlighted for me how crucial it is to treat the technology with great care. To continue the analogy, if you start of your GPS way point path at the wrong spot, the little handheld unit will be dead certain you are at the Sydney Harbour Bridge, oblivious to the large rock and red dust in front of you. Just so, a poorly fired analytic will tell you with absolute confidence that you should change the Call Centre settings….but if the initial data source wasn’t what you thought it was, you might find yourself standing in a wind blown desert wondering what went wrong.

Its ok to have feelings, just don't blame science.

2009-11-02T04:35:21Z

I’m all for evidence-based everything, which is no surprise I’m sure.

I was interested to read about the chief drugs advisor to the UK government being sacked for speaking out against the government’s drug policy. Professor David Nutt argued that cannabis had been reclassified as a higher risk drug against scientific evidence and was promptly sacked for commenting on policy.

He is quoted as saying: “So this government has made the law and then it has gone back to the advisory council and said ‘could you find some evidence to support our decision?’ Now we’ve said ‘no, we will stick to science, our scientific guns, we will produce the evidence and if you go and legislate inappropriately, we will continue to point out the evidence does not support you’.” (http://bit.ly/VUJpW)

This quote echoed a number of conversations that took place at the Enivironment Institute of Australia and New Zealand Conference last week. In particular, there was a lot of frustration at groups and individuals that don’t want a development in their locality (for whatever cultural, social or personal reason) and who build (often spurious) scientific arguments to validate their opinion.

Considerable time and effort is often spent in putting together good, unbiased scientific evidence to address the “scientific” concerns, only to be ignored, deflected and demeaned, because it does not support the community sentiment.

Too often this situation decends into a “my expert is better than your expert” skirmish, and noone wins. And you may be surprised, but as a scientific consultant, my livelihood is dependant on me giving my clients the best evidence-based answer, not the one they want. And it’s a kick in the guts to go into those situations in good faith and wanting a good scientific debate, only to have my scientific integrity brought into question.

And you know what I reckon?

I think that it is actually valid for a community to stand up and say, “we have heard the evidence, we accept the value of the proposal but we don’t want it because it will make my backyard ugly/change the nature of our community/change the road I take to work”. Politicians should stand up and say “that is a fair and reasonable scientific argument, but parents in my constituency just don’t want another thing to worry about, no matter the risk, so we will go against the scientific advice for now.”

It’s ok to have social, political, religious or emotional responses, and for these to be taken into account. But they cannot be taken into account, if they are not out on the table.

Just don’t use science as a cover up. Thanks.

What exactly are you asking? Thoughts from EIANZ

2009-10-30T00:24:56Z

Good policy is ok, but how do we implement policy that we are also able to monitor and evaluate over time? How do we know that the activities we undertake on the ground will have the impact we expect?

This is a complex issue, especially in the evironmental sector. Here turning good policy into good practice involves a clever mix of

community engagement
science
trust (i.e. avoiding the pitfalls of the ‘my expert’s better than your expert syndrome’)

and utimately, it requires policy that actually can be translated into action, monitored, evaluated and improved.

This was a topic of much discussion at last week’s Environment Institute of Australia and New Zealand (EIANZ) conference in Canberra. There were a plethora of examples, workshops and ideas (stayed tuned, and I’ll post a link when the presentations go online).

For our part, we argued that good practice is impossible unless policy and investment is planned with a view to the evidence and monitoring that will be gathered to track the effectiveness of the activity. Incorporating statistical survey design is one such approach.

Traditionally survey design is done just prior to going out in the field. However, the first, most critical aspect of good survey design is having a well defined question to answer. This, we argue, is where policy comes in. During planning (whether planning a compliance or an investment program) we need to consider the long and medium term resource outcomes we which to achieve but ALSO asking what are the most appropriate metrics that can be monitored on the ground to track our progress towards goals.

This involves considering the size of the effect we are looking for and the power and the significance we can acheive through monitoring.

But enough from me - you can download the full presentation here

A confession

2009-10-29T02:24:38Z

I do confess that I read textbooks. Mostly they are obscure methods and applications, but occassionaly they are graduate or college level, and even introductory. I do this because I need to keep up with new tricks, but also because there is something special in trying to teach a newbie analytics.

I mean, it is all about creating the right mental image so they can see the data shape and flow, and so instinctively recognise how to work with it. And introductory text are full of simplififed, yet correct, views of the world of data.

So, what is my confession? I normally gloss over the sections about “descriptive statistics” (i.e using stats to describe the overall “feel” of the data - 1 in 5 kids skip breakfast, etc).

They never interested me. I am a more “inferential modelling” type of fella (i.e. using hypothesis testing and other hefty tools to infer insight and generate correlations and predictions about a data set).

If it isn’t predicting an outcome, or telling me how to do things better or faster, why would I care? I should acknowledge that this often gets me into trouble with the Communications section, who are charged with creating the visualisations and stories around the inferences I find.

But I had a moment the other day when I was challenged on why I had made a choice of design. My answer had to do with the underlying nature of the data set…yep. Descriptive Statistics held the key.

So now, as well as sending my models down to Communications to have them make the pretty visuals for the output, now I speak to them before I model too, to see if the descriptive stuff holds any secrets that make the modelling and data mining more efficient, or more appropriate.

This blog was written without duress. No inference should be made about my continuuing support for the Communications department and the wonderful work they do.

Mmmm humble pie……

Don't squeeze all your numbers into the one black box

2009-09-29T02:00:47Z

Spreadsheet packages store, visualise and manipulate data, typically in the form of numbers. In my role as a “Data Cruncher” I spend a lot of time delicately pushing, pulling, twisting and squeezing numbers, and a lot of my work is done right there in those spreadsheet packages.

Once I know what information I need to extract from the data, I set about doing what I do best…turning data into information. After putting it through a series of checks to ensure that the data actually contains the information I’m after, I get to have what I call fun.

There is an almost limitless number of ways that data can be “crunched”, this also means that there’s no typical way to deal with the data, every job is unique. For me this has the added bonus that for many jobs I require customised tools to handle the data. The bonus comes from creating these customised tools, I get to develop tools and methods which ensure the data is handled and analysed correctly and efficiently.

Custom built tools allow for a complete understanding of exactly where every piece of data starts off, where it goes and what happens to it along the way to turning it into information. The greatest disadvantage that a data cruncher, like myself, can have is to work with a “black box” tool, one in which you have no way of knowing what happens between one end of the system and the other.

If you treat the numbers poorly the information can come out the other end bruised and battered, sometimes without any obvious signs of abuse. The tools I build to handle your numbers have nice, clear perspex cases on them…meaning I can see what’s happening to your data, and stop it getting bruised.

Spreadsheets are a good platform for data crunching, however with an expert user driving them coupled to completely customised software they become transparent, efficient data crunching powerhouses.