If transparency in our government is to be achieved, it will depend in no small measure upon an educated public. If the public is not educated about the decisions being made about their future then transparency is not the whole issue that faces us. If the public does not have sufficient understanding of the proposed actions, then attempts to gauge their opinion, their choice of options or their understanding of the consequences is made tenuous. If we are misled as was the case regarding the capabilities of TNR to meet the needs of Wichita and were misled about the ability of the vendor Friends of Feral Felines KS to carry out the ambiguously defined mission, then transparency has not, and will not be, achieved. We find it difficult to take the city seriously when they say they wish to have greater transparency if this is what transparency looks like.
Wichita Animal Services claims that the outcomes of two web based surveys gave them the necessary information regarding public opinion to pursue TNR as a method for dealing with unowned cats in our city. In this article I will show why that assertion is unjustified.
The city’s use of web based surveys to gauge public opinion on controversial topics should be discouraged and results from such surveys should not be considered, de facto, as substantial proof of a representative opinion. At the very least web based surveys should not be put forth as the Voice of the People without scrutinizing the results of the survey. The use of loaded questions as seen in question 16 (How do you feel about the idea of Trap-Neuter-Vaccinate-Return – which is a program that makes sure the cats stop having kittens, they are vaccinated against rabies and returned to the same spot so other cats don’t move in.) is a way to lead respondents to a conclusion about TNR that does not meet the reality, as citizens have already seen even before this proposal has been voted on. This erodes transparency by presenting the results as something arrived honestly when in fact the response was manipulated. This question was followed by a selection of specific responses which guaranteed to weight the opinion in favor of that held by Animal Services. The specific responses are viewable from the link below ‘survey results’. This is not the impartiality the Chief of Police ordered AC to operate under.
Another thing that puzzled me was why, if the opinion of Wichitans was sought, did the survey use an open text field for zip code? Normally you would use a drop down constrained to the zip codes Wichita has. Or you limit the input area to 5 numeric values. I’m not sure this bit means much but it did allow some none zip code data in and I think it shows the level of thought that went into the survey design.
Let’s take a closer at the “Feral Cat” survey presented to the public for 14 days from March 27, 2017 through April 10, 2017.
This KSN news story about the survey says “By about 5 p.m., it already had over 800 responses from the community. ” By 11:55PM it had 1406 which, given the manner in which it was advertised and the widely dispersed locations of respondents, is a bit surprising. Ultimately word of mouth distributed among a network of individuals, like a Facebook group, an email group or members of a chat may have played a greater role in informing people of its existence than its presence on a wichita.gov alone. But were these responses entered by random citizens or was it more organized?
To answer this question I’ve taken the time between subsequent response start times for all 2124 response (this includes those outside of Wichita) and shown how on the first day the difference between each response is less than 30 seconds and the majority are 10-20 seconds apart for the majority of the day. After Day 1 the time between responses grows and the length of time to complete a response grows. This is indicative of brigading and is one reason the city should not rely blindly on these types of surveys. You can view the first 30 responses here to get an idea of what I’m referring to. The entire 2124 can be seen here.
I feel that a note about my data is in order for this next section. The question I have is this: Given the location specific nature of this issue does a result that that treats Wichita as homogeneous whole, with each zip code as being equal in regards to their actual experience of this issue and the possible consequences of this policy really provide a clear guidance for policy making? For example, if a zip code that is relatively free of cats dominates the result set, skewing (my assertion) the results for all Wichita is that really representative of all Wichita? If an area of the city that is seeing the worst impacts of free roaming cat populations, unabated by Animal Control and the conscience of cat fanciers, would they think releasing cats back to their neighborhood be great as the sole alternative? Personally, I think you have to get a representative sample from each zip code and then gauge the results from that to see if the city will be served by the proposal.
The population of Wichita in 2017 (when this survey was posted) was 390,591 according to Census data. I used zip code populations published by Zip-codes.com which relies on postal delivery information in order to make population estimates with regards to specific zip codes. If a zip code spans 2 cities both populations are counted. If the City of Wichita has more specific data that would be the preferred source for population estimates but my search did not turn up an alternative source.
I am also relying for sample size metrics on this site. An expert for what constitutes a valid sample size would be the preferred method but I have no funds available to access such an expert. I did not compare this to what is published by SurveyMonkey. My reason is that SurveyMonkey is a for profit survey provider and they may not have an impartial metric for determining this. I glanced at their recommendations but made no note of them nor did I compare them with what was presented by CheckMarket.com. You may wish to do so.
For a 2% margin of error and a 95% confidence level, the required sample rate is 2387 for our population size. We had 2124 responses but as is shown not all of these were from the city of Wichita. In order to achieve this sample size 11,935 would need to be invited from Wichita to take the online survey which assumes that only 20% of those (2387) would actually take the time to do the survey. Surveys appear on Wichita.gov and are shared on various social media sites to facilitate exposure to the population of Wichita. It’s questionable if random visitors to City sites would generate the kind of response rates that were recorded.
If all 2124 respondents were residents of Wichita our margin of error would be 2.12% with a confidence level of 95%. The charted results presented by Animal Services to the public and to their advisory board did not remove non-Wichita zip codes, opting instead to use all responses even though they were attempting to capture the will of Wichita citizens and even though extraneous data was obviously collected.
Removing out of area zip codes from the results we find representation by 27 different Wichita zip codes, as entered by respondents, to be as follows:
The graph indicates that some zip codes are better represented than others but, as will be shown, still below what is needed to be called ‘representative’. What if some of those zip codes were added by respondents living outside of Wichita to appear as though they were residents? How could we make an educated guess about that?
The use of geo-location by IP address is ubiquitous today. Getting directions to a place from where you are currently located is something that almost all of us present have done at one time or another and is an example of this technology. Another example is when google search results display local businesses.
When you allow a site to know your location, in addition to providing you with say, accurate directions, that information is stored in a database and becomes an asset which can be sold to other interests wanting to offer location based services or fraud detection services. One of the most accurate, fee based databases is maintained by a company named Maxmind.
There were 1973 unique IP Addresses in the survey, location data was available for 1877 and of these IPs 1245 (60%) were located in Wichita, 728 were from locations outside of the city. 96 locations were unknown. Duplicate IPs were not significant, most were 2 entries from the same IP as would be the case for a shared home computer. Only one stood out as there were 10 entries from the same computer owned by the City of Wichita. If this were testing then this data should have been removed but it was not. 66% of the respondents were in Wichita. When broken down by zip code the veracity of this survey results becomes more questionable. Also it is important to note that the location services show where these IPs are currently located not where they were when the survey was first filled out.
Seen in this way it becomes clear that not all areas of the city were represented in the survey. In fact far from it. Some zip codes were not represented at all. Here is a map of zip codes contained in Districts and here is a way to find your district. Of the zip codes that did respond the numbers fall short of representative samples from those locales.
| Zip Code | Population | Responses | % of Zip Pop | Responses needed |
| 67202 | 1393 | 8 | 0.57 | 882 |
| 67203 | 31287 | 237 | 0.76 | 2230 |
| 67204 | 21194 | 81 | 0.38 | 2157 |
| 67205 | 16811 | 62 | 0.37 | 2101 |
| 67206 | 15186 | 79 | 0.47 | 2074 |
| 67207 | 26978 | 98 | 0.36 | 2205 |
| 67208 | 18161 | 139 | 0.77 | 2121 |
| 67209 | 13654 | 54 | 0.40 | 2042 |
| 67210 | 10339 | 24 | 0.23 | 1949 |
| 67211 | 20644 | 112 | 0.54 | 2151 |
| 67212 | 44707 | 234 | 0.52 | 2279 |
| 67213 | 21368 | 106 | 0.50 | 2159 |
| 67214 | 16542 | 46 | 0.28 | 2097 |
| 67215 | 5708 | 38 | 0.67 | 1691 |
| 67216 | 24041 | 80 | 0.33 | 2183 |
| 67217 | 30186 | 120 | 0.40 | 2225 |
| 67218 | 22772 | 104 | 0.46 | 2173 |
| 67219 | 12121 | 34 | 0.28 | 2005 |
| 67220 | 14214 | 33 | 0.23 | 2055 |
| 67221 | 0 | 1 | 0 | 0 |
| 67223 | 513 | 2 | 0.39 | 423 |
| 67226 | 17757 | 91 | 0.51 | 2116 |
| 67227 | 344 | 3 | 0.87 | 301 |
| 67228 | 2033 | 8 | 0.39 | 1102 |
| 67230 | 9959 | 48 | 0.48 | 1935 |
| 67235 | 11446 | 69 | 0.60 | 1985 |
The percentage of responses per zip code do not meet the necessary minimums to assure that the entire population of that zip code has been reached, let alone the entirety of Wichita. The number of responses required to meet a 95% confidence level with a 2% margin of error is shown in the right most column and when compared to what was actually received it becomes clear that what was gathered cannot be argued as valid for a Voice of The People survey. When one considers that a shocking 25% of Wichita citizens lack access to basic internet services the disparity grows clearer. The survey should not have been the sole justification for the direction taken by Animal Control.
In this article my hope was to show how an uncritical reliance on web based surveys to capture the will of Wichita citizens is not possible. IT is pretty simple to sway surveys by posing questions and providing the responses. These surveys are even less likely to come close to approximating the general will of our citizenry, especially when the subject is on a hot button topic such as how we deal with unwanted pets. Attempts to shape opinion by using loaded questions and canned responses offers a more compelling reason to believe that these surveys are anything more than propaganda for an unproven technique of animal control.
For a broader study of survey based opinions on TNR see The Road to TNR: Examining Trap-Neuter-Return Through the Lens of Our Evolving Ethics