Are our surveys scientific? Are they representative? Are the numbers reliable?
Some assume that because this survey is not administered via telephone or via snail mail, it is not scientifically representative of the community and that, therefore, one should be skeptical of all results.
First, skepticism of surveys is a good idea for a variety of reasons. They are often used to push an agenda someone has. They can be poorly designed because the people creating them cannot look past their own biases. Sometimes, they fail to extend beyond their own bubble of contacts. Many prefer to listen to people, web sites, news, and TV shows that agree with their views most of the time. When these groups/sources run a survey, they are often receiving mostly the opinions of their audience, and that audience tends to agree with them.
We are aware of these limitations and challenges. Can we rise above them? Perhaps. We will continuously try. So, please do look at what we do with skepticism. Point out to us things we can do to improve. Help us extend beyond our circle of contacts to include those we don’t know. Help us create something that is fair to people of different perspectives.
We write this page to address the question of whether the public hospital district survey is representative or not and, for that matter, whether any survey we conduct is representative.
Keep in mind that we will do another hospital district survey closer to the election. We will compare the results of that survey to the actual election results in order to gauge how representative we are.
Years ago, a traditional survey team would obtain a list of phone numbers. Calls were made to a selection of random numbers from that list. A query was, then, made to whomever answered in order to see if the person matched the desired demographics. If so, the survey proceeded with the hopes that the person would finish the survey. To get 400 respondents who matched the desired demographics it might have required several thousand phone calls. The effectiveness of this technique has plummeted with the advent of caller id and cell phones. Most people now screen their calls. Companies have taken to routing through local numbers; but, even this is no longer effective. Accessing a comprehensive list of cell phone numbers is difficult. Do you get some calls where there is a ring and then they hang up when you answer? One reason for this is someone calls every possible number to discover which numbers are valid. Then, they sell the list.
One can go with in-person interviews. But, these too have their limitations. People will go to the other side of the street, if they see someone with a clipboard. If they are with someone and get stopped, they may modify the answers they give in order to be in harmony with what they think their friend might want to hear. In-person surveys are also limited in that they often miss the working, the young, and the very old. They are more difficult to match the needed demographics than many of the other methods of surveying.
Snail mail. This can be an effective method. One can obtain the names and snail mail addresses of voters from the election board. One is prohibited from using this source for commercial purposes. Politicians exempt themselves, and one can use the list to conduct polling regarding political issues or to campaign. You will notice that this method is rarely used, though. There are a couple of reasons. Cost is one. Second, it misses those who are not registered to vote.
Another important aspect of surveying is to make sure that those being polled are not self-selecting but are, instead, a random selection from the population on whom the survey is focused.
Notice, that not all kinds are being sampled.
Are only those who share a similar opinion taking the survey, and are we failing to gain the opinions of those who think differently? If the answer to this is “yes,” then the results will not be representative. To protect against this, statisticians say we must survey a random sample. One way of doing this is for the survey maker to choose who will take the survey. As long as that choice is made using some truly random method, we clear the first hurdle. But, we must also persuade each of these randomly-selected participants to actually participate and to give honest answers. When dealing with objects or animals, this is relatively easy to do. But, when dealing with people, it is very difficult. We might randomly select a group of people; but, if most of the people who think a certain way end up refusing to participate, then we still have an unrepresentative result. This happens frequently when dealing with controversial subjects where the survey taker doesn’t trust those giving the survey to keep the information private and secure.
Every survey has potential pitfalls. There is no one method that is without challenges.
We allow anyone who wants to take our survey to do so. This puts us at risk of not having a random sample and, as a result, not accurately measuring the population.
So who does not take our surveys?
This matters if these groups do not share the opinions in equal proportion to those who are taking the surveys. For example, if we ask questions about rentals and only ask the landlords, then we will not be representative of the opinions of renters — at least not until we increase the number of renters taking our surveys.
We solicit survey participants via email, Facebook, our web site, press releases to the local media, posters at the post offices and other frequented locations, and via other similar methods. We, then, ask demographic questions of those taking the surveys and compare them to census data and any other known measure that we can. If we have a close match to the known demographics, then we know we are representative of those demographics. We can verify this using a statistical technique with our data that is called “bootstrapping.” We can randomly sample our own data a thousand times or ten thousand times and then compare results. If our data falls within an acceptable standard of deviation, then we know we can have confidence that our sample is random, even though we are not the ones who selected the survey participants. Both criteria must be met. We must match known demographics and must fall within the normal range of our bootstrapped data. This does not eliminate the problem of not sampling an adequate number of people across all demographics.
The point of this is that we can demonstrate a scientifically valid sample even if people are choosing on their own whether to take a survey or not. There are several criteria that much be met, but it is doable.
Most of those receiving our email invitations open it to see what it is about. If they are not interested, they ignore it. This is no different than someone hanging up the phone if they don’t want to take a survey or are not interested in the subject matter. It is no different than someone throwing away snail mail they don’t want to answer or walking on the other side of the street from someone holding a clipboard.
One might worry that only those who support a cause take a survey. The best way to counter this is to make sure that the questions are fair and neutral, and that the operation doing the survey has a good reputation. Everyone has to start somewhere. We have started with a topic that people care about. We have a vote coming up that will allow us to compare our results to an actual vote of the people. As more people participate, the results will be more of a reflection of the overall perspectives of the community. We hope to build our reputation and acceptance of this technique for engaging the public. Accurate results mean better decisions by policy makers and more understanding in the community of each other’s perspectives.
In the case of the hospital district survey, we have over 740 people who have taken the first survey. That is over 17% of the island’s voting age population. We expect that, as we touch on other topics, more people will engage. Over time, we think that people will learn they can trust us to protect their privacy and to be fair in not only our questions but also how we use the data. Again, that can only be demonstrated by actions and over time. Thank you for participating in this first survey. You have helped to start some valuable discussion, and you have given information that is important for us all to learn.
A final thing we want to address in this article is timing. Surveys are snapshots of what people are thinking at the time they take the survey. Later, they may gain more information on the subject or events may happen that cause a person to change their mind. Part of what is good about having these surveys is that they may start information-gathering, thought-provoking discussions in the community.