Math 217—Statistical
Consulting
Andrew Gelman is a
well-respected applied statistician at
January 25, 2008
Rindskopf’s Rules for Statistical Consulting
Our statistical consulting mini-symposium yesterday was great. I
wish we’d been able to video it. There was lively discussion of the connections
between statistical consulting and research, and the different aspects of
consulting in academic, corporate, and legal environments. I’ll be posting
everyone’s slides. Here's David Rindskopf's
contribution:
Rindskopf’s Rules for Statistical Consulting
Some of these rules are universal, while others apply only in
particular situations: Informal academic consulting, formal academic
consulting, or professional consulting. Hopefully the context will be apparent
for each rule.
Communication with the Client:
(1) In the beginning, mostly (i) listen
and (ii) ask questions that guide the discussion.
(2) Your biggest task is to get the client to discuss the research
aims clearly; next is design, then measurement, and finally statistical
analysis.
(3) Don’t give recommendations until you know what the problem is.
Premature evaluation of a consulting situation is a nasty disease with
unpleasant consequences.
(4) Don’t believe the client about what the problem is. Example:
If the client starts by asking “How do I do a Hotelling’s
T?” (or any other procedure), never believe (without
strong evidence) that he/she really needs to do a Hotelling’s
T.
Exception: If a person stops you in the hall and says “Have you
got a minute?” and asks how to do Hotelling’s T, tell
them and hope they’ll go away quickly and not be able to find you later. (I’ve
had this happen, and if I ask enough questions I inevitably find that it’s the
wrong test, answers the wrong question, and is for the wrong type of data.)
Adapting to the Client and His/Her
Field
(5) Assess the client’s level of knowledge of measurement,
research design, and statistics, and talk at an appropriate level. Make
adjustments as you gain more information about your client.
(6) Sometimes the “best” or “right” statistical procedure isn’t
really the best for a particular situation. The client may not be able to do a
complicated analysis, or understand and write up the results correctly.
Journals may reject papers with newer methods (I know it’s hard to believe, but
it happens in many substantive journals). In these cases you have to be
prepared to do more “traditional” analyses, or use methods that closely
approximate the “right” ones. (Turning lemons into lemonade: Use this as an
opportunity to write a tutorial for the best journal in their field. The next
study can then use this method.) A similar perspective is represented in the
report of the APA Task Force on Statistical Significance; see their report:
Wilkinson, L., & APA Task Force on Statistical Inference. (1999).
Statistical methods in psychology journals: Guidelines and explanations.
American Psychologist, 54, 594-604.
Professionalism (and self-protection)
(7) If you MUST do the right (complicated) analysis, be prepared
to do it, write a few tutorial paragraphs on it for the journal (and the
client), and write up the results section.
(8) Your goal is to solve your client’s problems, not to
criticize. You can gently note issues that might prevent you from giving as
complete a solution as desired. Corollary: Your purpose is NOT to show how
brilliant you are; keep your ego in check.