 |
| Image Source: https://www.sd61.bc.ca/ |
At the time I had a few parents bring this report to my attention, asking me to look into the survey results, as they felt that the released results were misleading, or rather that the proposed policy changes did not appear to line up with the opinions of most parents (in their view).
Initially, my first thought was "Everything is probably legitimate and it's great to see some evidence based decision making on the part of the school board" ... but within minutes of reading the press release, I began to wonder if this press release was the result of a rushed ad hoc job on the part of the school district, or if they were purposely trying to twist the evidence to support a pre-determined policy. Let's hope that it was the former and nothing sinister is at work here.
The full press release can be found here for those interested.
Early in the results, SD 61 claims that their survey had a 70% response rate. (starting page 12 of this document)
- 3450 to the parent survey
- 418 to the student survey
Yielding a total of 3868 respondents. 70% response rate, pretty good! (mind you no back ground as to the methodology as to how these surveys were distributed).
The next bit is the geographical distribution of the respondents to the parent survey, I have created a little table (below) to demonstrate the breakdown including the relative frequencies as well as comparing these relative frequencies to the relative frequency of the population in each municipality and school aged children. Keep in mind, given limited access to data (and time on my part ... this is only a blog!) we are looking at different years for each of these points -- but we should expect the relative frequencies to stay relatively constant over this short time period:
The first thing we should notice is that despite the earlier claim of having 3450 parent respondents, we only have a geographical break down of 3168 surveys, meaning 282 (8.2%) of the respondents were not included or dropped for what ever reason.
The second thing we should notice is the large discrepancy between the relative frequencies of the municipalities - specifically, Saanich is grossly under represented while Oak Bay is grossly over represented. (The two yellow rows are identified as such as there are no true relative frequencies to compare these too).
Here is a bar chart of the above table -- because pictures are nice too.
So, page one (truthfully page 13 if you're following along) of the report so far - some questions raised, but perhaps nothing too misleading so far -- Let's explore the reported results of the survey.
The first thing to note is that these reported results are only from the parent survey, the details of the student survey are not presented. Again pay attention to the numbers here.
The first question is along the lines of enrollment priorities. specifically, should siblings have priority (in order to keep siblings together at same school) or should catchment students have priority? meaning siblings may be split between schools.
The results (as presented in the report) are as follows:
Based off of this - it seems as if a strong majority of respondents (almost 61%) support catchment school enrollment over keeping siblings together. But pay attention to the total respondents ... only 2971. Turns out 469 respondents "skipped" this question for one reason or another.
The big question then - is it important that 469 respondents skipped this question? should these 469 respondents be dropped from the results? or should we include these 469 as perhaps a "no opinion" category? Let's include these skipped responses and see how our relative frequencies change.
We see the same situation here - The question being, who should have class priority each year, non-catchment returning students or new catchment students -- the reported results (omitting the skipped) say that almost 58% support returning non-catchment students. If we include the 469 who skipped the question, less than 50% support returning non-catchment students having priority, a sudden change in results if we are interested in majority support.
The second thing we should notice is the large discrepancy between the relative frequencies of the municipalities - specifically, Saanich is grossly under represented while Oak Bay is grossly over represented. (The two yellow rows are identified as such as there are no true relative frequencies to compare these too).
Here is a bar chart of the above table -- because pictures are nice too.
So, page one (truthfully page 13 if you're following along) of the report so far - some questions raised, but perhaps nothing too misleading so far -- Let's explore the reported results of the survey.
The first thing to note is that these reported results are only from the parent survey, the details of the student survey are not presented. Again pay attention to the numbers here.
The first question is along the lines of enrollment priorities. specifically, should siblings have priority (in order to keep siblings together at same school) or should catchment students have priority? meaning siblings may be split between schools.
The results (as presented in the report) are as follows:
The big question then - is it important that 469 respondents skipped this question? should these 469 respondents be dropped from the results? or should we include these 469 as perhaps a "no opinion" category? Let's include these skipped responses and see how our relative frequencies change.
If we include these skipped responses -- now only 52.65% of respondents support catchment over siblings ... not such a loud statement anymore! Additionally, with the 469 skipped included, we only have 3450 respondents. What happened to the other 10 respondents?
Sadly as we move through the other responses we see a similar trend. we have 469 respondents who decided to skip a question, and were just completely omitted as a result.
Again, total responses are only 3440, not the 3450 stated as total respondents. 10 are still missing.
The final reported question is asking about what should happen when students finish at their current school and are set to transfer to their middle or high school. Specifically, should out of catchment students simply follow the school path (certain elementary schools feed into specific middle schools etc.) or should these students be required to go to their catchment school, and have to apply to follow the school path.
The interesting part of this question is that we only have 2914 responses, with no information about skipped responses or any hint about the other 536 omitted respondents. Do we assume that 536 (15.5%) of respondents skipped this question? Or did these 536 spoil their response, circling both answers? unfortunately, we have no insight, only that 2914 answered this question as opposed to 2971 which answered the prior two.
Sadly, it is reports and lazy statistics like this which are supporting government policy. The school board has already met and revised enrollment priorities based off of these results.
The biggest question I have is this. If there are so many errors in this 'polished' press release, can we trust any of the methodology or process used in determining any of these results? remember, all that is being presented in the above is summary statistics - we have to trust that the individuals who put this report together properly sorted, compiled and calculated these statistics properly. Given the issues, I have just raised - I have my doubts.
What are your thoughts on this? feel free to comment below.
Sadly, it is reports and lazy statistics like this which are supporting government policy. The school board has already met and revised enrollment priorities based off of these results.
The biggest question I have is this. If there are so many errors in this 'polished' press release, can we trust any of the methodology or process used in determining any of these results? remember, all that is being presented in the above is summary statistics - we have to trust that the individuals who put this report together properly sorted, compiled and calculated these statistics properly. Given the issues, I have just raised - I have my doubts.
What are your thoughts on this? feel free to comment below.
Keith, oversampling one area (therefore under-sampling in another) is only an issue if the responses are not weighted. Since there appears to be a good chunk of observations for each area appropriate weights can be calculated, therefore I don't quite take an issue with this. Secondly, if respondents did not have an opinion, they may be coded as "skipped." If this is the case, I think we would want to know what percentage of respondents favour one option over the other conditional on having an opinion. These seem pretty common practice, perhaps more so than the average reader is aware of.
ReplyDeleteJeff, I agree with all you stated above - the problem is due to a lack of a methodology (I understand this is only a briefing paper and press release) we are left to make our own assumptions.
ReplyDelete(A) there is no further mention of geographical breakdown of responses - my understanding is that all responses are then aggregated for the final results, which may be misleading based on family size and dynamics in Oak Bay Vs Saanich for example.
(B) Your right about skipped results! My concern is the nature of this being a political document - For example, might you want to report your relative frequencies with or without the skipped results to change the magnitude of support which you may or may not have for a given policy?
Great points for sure though Jeff, Thanks for the comment.