A Transparent Process? Questioning the Survey

I spent a year or so working for the influential Dr. Dean Hoge at the Catholic University of America’s Life Cycle Institute. We were working on a major project for 5 Protestant denominations on why pastors leave local church ministry. (The project became this book; they thanked me in it somewhere.) I learned a lot, but not being a credentialed sociologist, my contributions would always be at the level of Research Assistant, and I was lucky to get that. It was fun. I learned a lot about ecclesiology, which has always been an interest. But there was no way that I would have been allowed to implement a survey, or analyze the data, or evaluate and publish its results. Everything I did was tightly scrutinized. The project was directed by a professor of sociology, considered an expert, with many published books indicating his research qualifications, and co-directed by a graduate student who was in the writing phase of her doctorate in sociology. The rest of us just did what we were told.

So imagine my surprise when a study purporting to be major is undertaken by students in the Liturgy Department of the College of St. Benedict/ St. John’s University, with a Social Psychologist from the CSB/SJU faculty as their Professional Consultant, and the hardly-disinterested Rev. Anthony Ruff, OSB, as their advisor.

Here are the credentials of the Project Manager, according to the PrayTell website:

Chase M. Becker is a Nebraska native and current editorial assistant for Pray Tell. Chase holds a B.A. in Philosophy from Saint Gregory the Great Seminary in Seward, Nebraska and is currently pursuing an M.A. in Liturgical Studies from Saint John’s School of Theology•Seminary in Collegeville, Minnesota. Chase currently serves as a Liturgical Designer and Consultant and has earned awards in historic preservation from Heritage Nebraska and Preservation North Dakota.

The sympathetic-leaning PrayTell commentariat have already weighed in on the likelihood of sampling errors, particularly considering the enormous amount of self-selection that occurred. Self-selecting dioceses participated. Self-selecting clergy from these dioceses participated. And as Jeffrey mentions below, we are given very little of the data.

Before we take this survey with any seriousness, why don’t we stop and ask for a little transparency.
What did the surveys look like? Did they indicate Collegeville as their source or destination, thus influencing participation? Which dioceses were involved and why? Were there any outside scientists involved in the process? Was any outside social scientist consulted for independent evaluation of the procedures?

What degree of professionally credible skill was involved in producing this survey? How involved was the in-house scientific consultant? Did she design the survey? Did she evaluate the results or train the liturgy students to a high level of analytical skill before they processed the results? What corrections were made to the raw data?

23 Replies to “A Transparent Process? Questioning the Survey”

  1. If you're suggesting that CARA should be commissioned by ICEL to survey English-speaking priests across the world from Oceania through Australia, Africa, and Britain on the way to North America, I'd say bring it on.

    I suspect you'll get zero traction from Vox Clara, the CDWDS, and ICEL. So good luck getting it in gear, there. So much for the institution taking the lead on seeking the truth.

    Meanwhile, why not ask the German-speaking bishops why they deep-sixed OCF2 auf Deutsch after six weeks?

    Todd

  2. She's not saying anything about a new survey, Todd. She's questioning the objectivity and competence of the people who did this one.

  3. I see that. But it dodges the question, doesn't it? Is MR3 a good development for English-speaking Catholics or not?

    Assuming I grant that a SJC-CMAA joint venture with CARA would be more persuasive, is there enough in this survey to suggest the Church undertake a serious study of the problem? Or is it enough for MR3-proponents to call foul to call the whole thing off?

    My own view might be to suggest that any criticism of the survey is unduly biased as well. There always seems to be an excuse. The survey was biased against non-ideologues who love MR3. And those who voted thumbs-down to MR3 were biased by poor implementation. There's always an excuse to be blind, eh?

    Todd

  4. Todd, these are wild pitches. My questions are about real, verifiable matters. Can you speak to them?

  5. "There's always an excuse to be blind, eh?"

    Which is why, for all practical intents and purposes, all these surveys are useless. One survey shows that a small subset of Catholics like the current translation; another shows a small subset of Catholics don't like it. Both small subsets can discount the results of the survey they don't like based on the methodological limitations of said survey. Well, as interesting as this situation is (yawn), it doesn't actually change anything.

    Then again, people somehow don't seem happy unless they're complaining about something. What a paradox. 🙂

  6. Is MR3 a good development for English-speaking Catholics or not?

    Why is a survey is a good way to answer this question?

  7. It's not. The good news is that while the new translation is not perfect, the chummy translation it replaces is not coming back.

  8. Sure can. I think CARA is a far better conductor of polling. They have an established credibility.

    But the methodology used by the SJU people was sound. My question to you stands: has enough concern been expressed to warrant, in your opinion, a serious survey not just in the US, but in all of English-speaking Catholicism? And can you offer evidence your criticism of the message bearers isn't just an ad hominem distraction?

    Suggesting that places like India or South Africa might be having more difficulty with MR3 is far from a wild pitch. It goes against our whitebread Ameri-centric view of the universe. But it is a serious pastoral matter.

    Todd

  9. Honest surveys publish their data, those that don't publish their data should be ignored. This survey, by not publishing its data, is making demands. Honest surveys don't make demands, they make appeals.

  10. There were news stories last December reporting that CARA found 70% support for the new translation among Catholic adults who attend Mass. It's sad that some disgruntled priests are unable to hear what the Spirit is saying to the People of God.

  11. Golly, this post is kind of an over the top ad hominem attack, isn't it?
    it sure seems liike Kathy Pluth doesn't like the message so she's shooting the messenger.
    At least I find her comments a bit angry and irrational.
    No survey is perfect, all survey has selection bias because not everyone fills it out. But wouldn't it be better to look calmly at what this survey might be telling us, even if it's not 100% accurate?
    What's driving Pluth's response, really?

  12. I have yet to hear a criticism of the new translation from any of the priests I spend time with. Maybe I should begin asking them explicitly. What I did notice was the change of opinion of the person I knew who was most adamently against the new . Before it came out, he would say that the only good thing about it is the restoration of the triple "through my fault" and an opportunity for obedience. Now, on many weeks, he tells me after Mass something that struck him from the translation, finishing it up with a comment, something like, "They did a pretty good job." Now I wouldn't be surprised if there are more criticisms among the presbyterate than the laity–they live and breath liturgy, after all. There are, indeed, legitimate areas for improvement! I myself am not completely satisfied. What I hope, however, is that any revisions of the translation are not in the direction of the unpalatable 1998 translation, or of the banal 1973 translation. This needs to be a decisive turning point, even if it is later improved.

  13. Tom, in failing to address Kathy's specific questions and points but instead accusing her of angry, irrational and personal attack, you appear to indulge what you ascribe to her.

  14. But wouldn't it be better to look calmly at what this survey might be telling us, even if it's not 100% accurate?

    If they release ALL their data and methodology, then maybe. Otherwise this is like saying, "wouldn't it be better to look at what this survey might be telling us about Iran's WMD program, even if it's not 100% accurate?"

  15. I have already admitted that I have never cared for vernacular liturgy (it arrived in the Western Church only in the second half of the 20th century, with no tradition behind it), and have no difficulty finding Latin in both the NO and the Roman Rite. An Oratorian priest (who was well-versed in both the NO and the RR in Latin) said in 2011 that the corrected translation made celebrating Mass in English very much like celebrating a Latin Mass.

    The Todds of this world are so immersed in their 1970s "liturgical" milieu that there is no hope for them. Things have moved on since then, but throwing in a bit of Gregorian chant and classical polyphony into an eclectic mix is anti-liturgical, as any real liturgist (as opposed to the self-styled ones who have established a veritable cottage industry in the USA) will testify.

  16. Todd, if you get a moment, I'd like your thoughts on my response "On Changing Hymn Texts." Thanks

    John

  17. Up to a point, Lord Copper. But real liturgical scholars can be counted on the fingers of one hand, and most of those who 'create' liturgies (yes, despite everything, there are those at PTB who claim to do just that) are not liturgists at all, merely parish music directors who have taken what amounts to a correspondence course in 'liturgy' from a liberal institution. In a UK context I can only think of one who fits the bill, Paul Inwood, recently given the heave-ho from Portsmouth diocese, whose baleful influence will take decades to eradicate.

  18. "Suggesting that places like India or South Africa might be having more difficulty with MR3 is far from a wild pitch. It goes against our whitebread Ameri-centric view of the universe. But it is a serious pastoral matter."

    Those poor, benighted foreigners. They have difficulty understanding complex things, don'tcha know.

  19. I didn't suggest the cottage industry is the product of liturgically faithful and sensitive liturgical and musical scholars, Lord C. The individual you mention is a case in point – he's many things but none of those; and he's made a comfortable living from the "liturgy" business, not least through his late Diocesan appointment and transatlantic connections.

Comments are closed.