SOC 341-01/51 (Kuechler)
Fall 1997
This assignment is based on chapter exercise #5 for chapter 8 of the textbook (p.306-8). Since you had a choice of picking
among these questions (or do additional ones to receive bonus points), I will discuss all questions. However, you need to
refer to the textbook pages for the original wording. Unfortunately, the questions are not numbered. The numbers here refer
to the sequence in which they are presented in exercise #5.
1. The main problem is that - given the population (=readership of a computer publication) - we cannot be sure that everyone is familiar with the "object" (PostScript Level 2 printer). Personally, I consider myself fairly knowledgeable in the area, but I have a vague idea only what a level 2 (in contrast to level 1) postscript printer is.
The easiest fix would be to add a response category "not familiar with product". Another way to improve the question is to add a brief description of what a PostScript Level 2 printer is. This, however, is tricky because such a description (before the actual question) must be short and it may be difficult to come up with one that is technically correct and makes sense to all respondents.
2. This is blatantly leading questions full of loaded terms (nonviolent, wanton, magnificent). In addition, the word wanton may be not in everyone vocabulary - several student reported that they had to look it up in a dictionary. This is manipulative, start to finish. Even eliminating the most loaded terms leaves a question with an implicit assumption, that commercial fishermen destroy thousands of marine mammals. To fix this one, at least three separate questions are needed:
Full credit if simpler fix at least eliminated the most obvious loaded terms.
3. Frankly, I don't find anything fundamentally wrong with this question. The response scale is adequate; whether or not to include a middle (neutral) category is subject to much debate in the (survey research) profession, there is no clear-cut rule. Adding a "don't know", however, does not make any sense here given that the population consists of college faculty and that respondents are asked about their current job - a topic they must have an opinion about. So, adding a neutral category could be done, adding a "don't know" category should not be done. One could also argue to replace "somewhat" by "mostly", but, again, this is a matter of taste and no strict rule applies.
My main objection is the stilted wording in both the introductory question and the (two) specific items listed here. The same can be said shorter and simpler:
"How satisfied are you with each of the following at your current institution?
a. freedom to teach what you like
b. time you have to work with students as advisor or mentor"
4. Single-family density is real estate and/or planners' jargon. It means building single-family homes in contrast to apartment houses (which requires less land and therefore make housing more affordable). Since this comes from a community survey (addressed to a general population), it is an inappropriate question because there is a high likelihood that people do not understand the question. Matter of fact, quite a few students misunderstood the question.
Apparently, this was taken from a larger list of statements (a statement battery) usually presented in form of a matrix question. Since we don't know the lead-in to this battery it is difficult to provide a fix without creating a possible double-barreled problem:
"Instead of single-family homes more apartment houses should be built to keep housing affordable."
One could argue that this mixes two questions; one about the preference between apartment houses and single-family homes, the other about the positive effect of building apartment houses.
5. The reasonably familiar wording in the question stem suggests a yes/no response scale (problem 1). It is also leading and
vague because "reasonably" can cover a wide range of individual interpretation. Fix: How familiar are you .....
In addition, the response scale must be changed to include "not familiar" - possibly replacing "not too familiar".
An explicit skip instruction is preferable to an if clause in the actual question. Since it is possible that a respondent is familiar with the code of ethics of more than organization (there is considerably overlap in the memberships of these three organizations), the question is triple-barreled. A respondent may agree with the code of ethics for one organization, but disagree with the code of another. No room to state such a view. So, a second matrix question is needed. -- "No opinion" is not a substantive answer and should not appear in the middle of the response scale. The middle category should be defined such that respondent who agree with part of the code and disagree with other parts can express this opinion; "ambivalent" is a possibility given a population of professionals who would certainly understand this word; "in part agree/in part disagree" is another option.
The question stem of the third part seems to indicate a "check all that apply" format. More importantly, the list of reasons seems incomplete and the limited choices offered do not even make sense. What kind of "political pressures" would keep people from "not using" the codes. And the third reason offered is similarly vague. This last part is so bad, that so quick fix comes to mind.
6. Vagueness all over. No specific time frame -- neither with respect to observation period on part of the faculty nor with respect to the period over which student performance would have changed. Also, unclear categories (what exactly is meant by "organization" in this context). Basically, this question is not answerable. The possible defense is that the question tries to address faculty perceptions. And faculty may have such perceptions and express them as "pseudo-facts" - but I hope that most faculty would refused to answer such a bad question. It is legitimate to ascertain perceptions and such data may be important indicators of something like "campus climate", but the appearance should be avoided that actual facts (about student performance) are gathered. Too bad to fix.
7. Apparently, an excerpt from a larger item battery. The response scale is acceptable; but see discussion above about the illusion of a middle category. The second item does not fit the format/response scale. One agree/disagree with general normative statements (about people or oneself should do). However, statements about actual behavior are either "true" or false", they "apply" to a respondent or may "not apply". In addition, the second statement is badly worded. Taken at face value, it is a truism that "I would probably not be able to stop most people from driving drunk". Well, certainly not most people, but how about most of my friends and relatives?
A real fix would require to see the larger context of the item battery (question). Simply switching away from the "I" format
does alter the contents of the question, but it is a first step:
"Even if one tries, it is often impossible to stop someone from driving drunk."