I'm afraid that's just not the case fela.fela fan said:
I used to work in the survey business. There are indeed "scientific" ways to go about surveying that can and do produce remarkably accurate results.
AND, your surveyor can also tell you exactly the degree to which those results will be replicated by conducting the same study and, hence, the level to which those results are to be trusted as a true reflection of the opinions of the "population" as a whole.
It involves careful "sampling" of the "population" (which could be a "representative" sample of the whole population aged between 15 and 64, for example, or a "representative" sample another "population", say Han Chinese women, living in Hong Kong, aged 18 - 35, who have purchased headache pills in the last six months.
Sampling is a precise science which takes into account population distribution by a wide variety of factors and often involves identifying and selecting various "data points" (geographical locations,) to ensure that a representative sample is surveyed.
Sample sizes are important, but usually, a sample of n = 900 is large enough (if properly selected,) to provide a representative sample that will produce results at a certain "confidence" level (usually 95%).
The questionnaire must be very carefully designed and the questions, and their phrasing, are critical (for example, a question about opinions on "abortion" will elicit a different response from the same group of people as the same question about "termination" will).
The selection, phrasing and order of the questions is, of course, an art and a science, but there're enough academic books out there devoted solely to the subject of questionnaire design, length, phrasing, ordering, etc. to ensure that anyone worth their salt sticks within certain existing "rules" and conventions. Questionnaire design also (should) takes into account how comparisons will be made with previous surveys to ensure that apples and apples or oranges and oranges are involved - matching fruit an' all that.
The "timing" of the survey is critical too and many variables are factored in to ensure consistency (is it a major holiday period for example, when certain "types" of peeps will be away and would be missed by the survey: has there been any recent "event" that could influence peeps responses: and so on). All of these things are accounted for and "controlled" for as tightly as possible.
Once all is ready, a "field test" is conducted and analysed to ensure that any unforseen problems with the questionnaire are identified and corrected and the survey is then conducted in its entirity.
Once the raw data is back, some of the respondents are contacted again (usually 20%) to ensure that the data has "integrity" (that the fieldworkers are not just sitting and filling in the forms themselves.
Once the data integrity is confirmed, the data is entered into a computer and then "logic checked" to ensure consistency (has any respondent contradicted themselves by answering similar questions with wildly different answers, for example).
If any problems are detected, these are fixed and additional respondents may be surveyed to ensure the required sample size is reached.
Once this is all done, the data can be "manipulated" to identify the results we are seeking to discover. Data manipulation can be as simple as cross referencing tables (to see if peeps who eat beef also eat chicken, or under 25 y/o women buy more cosmetics than over 65 y/o women, for example,) or can be excruciatingly complex involving running any number of statistical analyses, such as multiple regression analysis, corellation analysis, structural equation modelling, etc. depending upon what it is you're interested in discovering.
Once the data has been analysed and "milked" for as much info' as possible, the results are then "interpreted" and presented first in a "topline" presentation and then in a later detailed report. Interpretation of data is, once again, an art and a science and a skilled and experienced researcher will be able to provide much added value and insight through drawing upon their expertise and experience.
Assuming that the surveyor has the requisite knowledge and expertise in research design and the survey is carried out properly in every aspect (sampling, questionnaire design, fieldwork, back-checks, data entry, data analysis and reporting) then surveying is a truly scientific and extremely accurate predictor.
As an example. A properly conducted survey with a relatively small sample size (say n = 900,) can predict future results, within certain limits (margin of error,) at a confidence level of 95%.
So, what we can say is that if a "random" sample of all UK registered voters is surveyed, we can be 95% confident that replicating the same study will produce exactly the same result within a certain "margin of error" (usually somewhere between 2% and 5%).
We can predict with 95% certainty that ANY random sample of UK registered voters will give the same results to within say 3%. This study can be replicated again and again and again and (unless the world has changed - which again can be measured and accounted for,) we are statistically certain that if this study is conducted randomly 100 times, that 95 times out of a hundred the results will be the same (within + or - 3%).
And THAT means that if you surveyed EVERY registered voter in the UK you can be 95% certain that the results will be exactly the same as your "sampled" survey produced (within a margin of error of + or - 3%).
Most "commercial" surveys (y'know, the stuff to find out what washing powder people prefer, or whatever,) are conducted at the 95% confidence level and have a margin of error of plus or minus 3%.
Some surveys (long term, in-depth studies for medical reasons, etc.) can squeeze the confidence level up to 98% or higher and squezze the margin of error down to 2% or lower.
BTW fela, this isn't really aimed (only) at you. There've always been alot of people on these boards dispariging surveys, but actually, conducted "scientifically", a survey of a "representatve sample" of the "population" will always produce exceedingly accurate results, within some pretty fine tolerances.
As far as surveys go, the evidence is clear - properly conducted, they work and are remarkably accurate.
It is, of course, important that such surveys are conducted "transparently" and that the entire methodology and process is open to question - from the sampling to the questionnaire design, fieldwork, data entry, analysis, etc, etc. 'Tis the way of science. The results may be challenged, but the numbers will speak for themsleves and if the survey has been properly conducted, the results will stand, will be beyond reproach AND will be replicable.
(N.B. I am, of course, talking about "quantitative" research here - surveying, as opposed to "qualitative" research [focus groups, in-depth one-on-one interviews, etc.], which, although "scientific" is, by definition, not as accurate as surveys. Qualitative research is often used after a quantitative study to "add flesh to the bones" of the numbers, to provide deeper insights and to explore in-depth any particular issues raised by the quantitative study or any other areas of interest revealed in the survey.)
So there!
Woof
not that i want to derail or anything over a tounge in cheek comment, but the CNN poll is, or is it not, to gauge peoples opinion on Charlie Sheens (the actor) claims about a government cover up?
