August 6, 2006
Tariq Ahmed's put together an interesting survey trying to understand what features people want to see in the next version of ColdFusion (ColdFusion 8). Tariq just released the results of the survey, along with the raw data from SurveyMonkey.
I was initially surprised at the results Tariq published on his blog, as what I saw there didn't match up with what I'd been reading in various blogs, mailing lists, or forums (for example, Image Manipulation came in at number 16, something I know to be one of the top requested CF featues for several versions now). Wondering what the raw data looked like, I downloaded Tariq's spreadsheet.
I tried posting my questions/findings directly on Tariq's blog, but after my first comment, the comment system there ate my next attempt (I'm guessing it was too long). In any case, apologies Tariq for making a separate post...
I don't claim to be a statistician by any stretch of the imagination, but I did have an undergrad class or two in statistics and survey design, and two things didn't seem quite right to me.
First, the question style used is somehting known as a Likert-type question. It's a question used to measure attitudes toward a particular construct:
1 - Strongly Disagree
2 - Disagree
3 - Neither Disagree or Agree
4 - Agree
5 - Strongly Agree
Technically, this is known as the Likert-scale, however I'm referring to it here as Likert-type because Likert-scale questions are always used with multiple related items, and the survey here deals with single, unrelated items (individual features).
In a survey using Likert-type questions like this, a response of neutral shouldn't be "scored" as meaning that someone wants a feature more than they don't. If you are assigning a value of 3 to a neutral response, and using that to calculate a mean score, you are making that response seem as if a respondent wants the feature more than they don't (which isn't the case). One of the big issues in using a Likert-type question in this type of survey is that the responses are ordinal because it can't be assumed that people filling out the survey will perceive the difference between adjacent options as being the same. In other words, the value assigned to them (1-5) is positional, not representative of an actual mathematical value.
One common way Likert-type questions are scored/analyzed is to combine all of the like/agree responses into one group and all dislike/don't agree answers into another, discarding all of the neutral answers. Since the question we're trying to answer here is what features are most important to people for CF 8, it seems to make sense to just combine the want/really want, and rank the items from there. This answers the question "what do people want to see" vs. what are people ambivalent about, or what aren't people interested in seeing (which you could extrapolate from the survey as well. If I order just based on what people voted that they want to see, here's the order or preference I ended up with. Out of 317 respondants, here's what people wanted (ordered by the number of people indicating "want it" or "really want it":
243 Image Manipulation
233 Improved Crash Prevention
232 IDE/Hooks into better interactive debugging
229 Access to Server Internals/Metrics
222 Add PDF Forms Support
216 Give us an IDE with CFC introspection
215 Improve CFDocument
211 AJAX Abilities
210 Control logging at the application level
209 Improvements to large file parsing
209 Create/Extract/Manage ZIP Files
202 Tools to debug remoting and gateways
196 Add native support for MS Office formats
190 Strong Integration w/Exchange (Mail/Calendar/Etc...)
189 Threading/Thread Management & Protection
186 Add native support for RSS
184 Add cache control at the app level
184 Manage many CF Servers from a single console
181 Enhanced flash forms using Flex 2
175 Improved Reg-Ex Support
173 Upload to a Variable instead of Disk
168 Add granular control of DB and content caches
156 Faster validation abilities (E.g < validator Flex's borrow or , validValues="x,y,z"> 154 Integrated CF8+Flex FDS Install
153 CFAdmin Tools for creating Event Gateways
149 Add non-DOM based XML Parsing
149 Strong IMAP Integration
144 Make CFC Instances Serializable
143 Plug-In Architecture
141 CF Desktop Plugin to support drag and drop upload
137 CF to CF Communications via AMF (vs XML)
132 Add CSS/HTML pod layout support
128 OO Abilities: Constructors
119 Add refactoring to an IDE
116 Ability to deploy customized+scripted CF Installations
115 3rd Party Updates
114 Web Interface Hooks into JRUN Configuration
113 Static Methods in CFCs
113 Sandbox Security Enhancements
109 OO Abilities: Interfaces
107 Integration of Breeze Functionality
98 Tomcat & JBOSS Support
94 OO Abilities: Function Overloading
90 SNMP Gateway Service
77 Add accessibility 508 checking to an IDE
72 OO Abilities: Destructors
58 Mustang Support
56 Trend to be more Java like
55 E4X Support
If you really want to use the mean for each question, you should get away from a Likert-type question with a neutral value, and just use a scale of 1-5 with one being no interest and 5 being strong interest.
What all of this really illustrates is just how difficult it can be to design surveys and interpret their results. I really wish tools like SurveyMonkey would do more to help reduce ambiguity in surveys, as well as provide better analysis tools for the data generated.