Last night when I arrived home I found a survey waiting for me in my inbox. Being a man who likes statistics, I tend to fill out surveys1 whenever I can. For the most part, the surveys I’ve completed have been well constructed and relatively enjoyable to fill out. I can’t say that last night’s survey lived up to my previous experience.
Let me explain why.
While most of the questions were well constructed and presented in a manner which removed any ambiguity about what was being asked, several missed the mark (in my opinion). For example, consider the following:
Here the surveyee was asked to identify which of the following online classifieds/listings websites, if any, are well described by each of the following statements? Please check all that apply.
The trouble was not in the question. The trouble was also not in the data that were intended to be collected. The trouble was in how the data were collected – and thus what data were actually collected.
Consider the first potential response: Houses and condos available for sale near to where I live.
In this particular case I was not aware if Kijiji or Craigslist offered house/condo sales listings, even though I had used both sites. I had never visited PropertyGuys.com, so I was unaware if they listed house/condo sales. I assumed they did based on their name, but I didn’t really know having never used the site. I knew that MLS/Realtor.ca offered this type of listing.
So the question became how do I respond?
By checking MLS/Realtor.ca, I’d be indicating my belief that MLS/Realtor.ca is well described by the statement Houses and condos available for sale near to where I live. Making this selection meant that I clearly couldn’t select None of these as a potential response. So far, no problem.
But how to deal with Kijiji, PropertyGuys.com, and Craigslist? By not toggling them, I’d be indicating that they could not be well described by the statement Houses and condos available for sale near to where I live. But this wasn’t how I should have responded. In fact, I should have responded to each with a Don’t know because I truly didn’t know they could be well described by the statement. This was an issue. I couldn’t respond in this manner. The Don’t know toggle was a global response. That is, I could select Don’t know if in fact I didn’t know for all of the four sites listed. But this wasn’t the case in which I found myself. I didn’t know for three of the sites, whereas the fourth site I knew.
So what did this mean? It meant that the data that were collected weren’t actually answering the question the surveyor likely wanted to answer. I could be wrong. Perhaps they were only concerned about the global Don’t know responses and those responses where the surveyee thought a particular site was well described. Perhaps the surveyors didn’t care about sites that weren’t well described.
Or perhaps the data I provided were incomplete and possibly misleading.
Being the surveyee, I’ll probably never know. What I do know is that this – at least for me – is a reminder why data shouldn’t just be collected without ample thought. It’s important to recognize why data are being collected, what purpose they serve, what question(s) they are trying to answer. Knowing these things should help ensure how the data are collected is appropriate to the task at hand.
1 Assuming I have time and they appear to come from a reputable company2.