The “Gen17” survey is deeply flawed

 Email this story

Print

4 Comments

Although the “Gen17” survey is, of course, a notable achievement in many respects, it is also deeply flawed, and its flaws and deficiencies ought to be pointed out in order to improve any future community survey of this kind…writes Bill Rubinstein.

Professor Bill Rubinstein

The only proper way to conduct such a survey is to obtain a comprehensive master list of all Jews in the communities being surveyed  ̶  in this case, Melbourne and Sydney  ̶  and then draw up a random sample of names on these lists, a genuinely random list obtained from a table of random numbers or generated by a computer. This is, of course, the method adopted in all surveys of voting intentions, and is the reason why a truly random sample of only 1500-2000 registered voters, drawn from the millions of registered voters, will produce a result normally within a few per cent of the actual outcome.

Does such a master list exist? It certainly did for Victoria in 1988 when I examined it, and was maintained by the Jewish Welfare Society  (now JewishCare), and at the time listed the names of 41,276 persons in Victoria who were Jewish. This communal register, as it was known, was used for fundraising purposes and was revised on a frequent basis, eliminating those who died or moved away, and adding new arrivals.  It is difficult to believe that such a master list does not still exist, or that Sydney does not also maintain a similar list.

Using a random sample of this type is not, however, what the “Gen17” survey has done. Apparently, it has simply requested that all local Jews answer its online questionnaire, and reported on the responses of those who did.  According to ECAJ President Anton Block, who launched the survey, “Census data and objective records were studied and used as a baseline to ensure that the Gen17 survey results were appropriately weighted  to exclude any skew of the data towards any demographic or other group.”  Despite this caveat, the methodology it employed was inappropriate if not invalid. Simply reporting on the replies of voluntary respondents is exactly what was done in probably the most notoriously inaccurate poll in history, that conducted by the American weekly TheLiterary Digest during the 1936 presidential election. A popular magazine of the time, it succeeded in obtaining no fewer than 2.4 million responses as to voting intentions at that election, and confidently predicted that Republican candidate Alfred M. Landon would easily defeat President Franklin D. Roosevelt, who was running for re-election.  Of course, the very opposite happened: FDR trounced Landon, and was re-elected with a record majority.  At the same election, the newly founded Gallup Poll employed the valid method of surveying voters, and predicted that Roosevelt would win decisively.  Largely because of TheLiterary Digest fiasco, that journal’s method of conducting surveys was abandoned.

Unfortunately, however, the Gen17 survey has used TheLiterary Digest technique.  While its 8600 respondents are superficially a very impressive number, its results are reliable only insofar as they represent a truly random sample of the Jewish community, without any biases. “Appropriately weighting” its results will not correct the absence of randomness, and almost nothing can be inferred about background or attitudes of those it has not polled.  Any adjustment by assuming that those it has polled will be similar to those it has not polled will be valid only if these two samples are similar, for which it has no evidence.

There are a number of other basic problems as well.  My information is that at least one respondent (not me) voted several times, in order to advantage the viewpoint with which he is associated.  There were apparently no proper checks about this, and nor, indeed, whether respondents were actually Jewish, however defined,  and entitled to vote. The major problem with the survey, however, is that the background and attitudes of those who did not participate in the survey  ̶  90 per cent of the community  ̶  are unknown, and simply cannot be inferred  from the responses of those who actually did vote. Moreover, it is highly unlikely that those who did participate represent anything like a random sample of the whole community, but is a group hallmarked by clear biases. According to Stephen Chipkin, the results showed that “our community is becoming more secular and less observant,” but this result may have simply been found because more secular and less observant Jews were more likely to participate in an internet survey of this kind, which, it is reasonable to suggest, was more likely to be answered by younger, university-educated Jews than others. We do, however, have some clues about the orientation of the community.  Of the 52 synagogues in Melbourne, 47 are Orthodox, three Liberal/Progressive, one Masorti, and one “humanist.” Apparently about 70 per cent of students now at a Jewish day school in Melbourne attend one of the five schools which are Orthodox in orientation, 11 per cent the Liberal-founded school (King David), and 19 per cent the two secular schools (Bialik and Sholem Aleichem). These figures are only indicative, but one wonders whether they have been fully taken into account in the survey. If there is one thing we know about contemporary Jewish identity, it is that the number of very religious Jews has greatly increased, not decreased.

Several of the findings in the survey strike me as being, frankly, unbelievable. Thirty-two per cent of respondents stated that the Israeli government should negotiate with Hamas, officially regarded as a terrorist body by all Western governments, with which no one in the Israeli mainstream would negotiate until it changes its charter that calls for the destruction of Israel. Similarly, 37 per cent of respondents  stated that non-Jewish people suffer from discrimination in Israel, a position here associated only with the extreme left. The sheer implausibility of both of these “findings” strongly suggest either that their respondents are a deeply unrepresentative sample of the Jewish community, or that the poll has been abused by multiple votes from a minority.

The survey’s controversial findings about intermarriage rates also seem questionable. It found that 33 per cent of those whose first marriage took place during 2010-2017 were married to non-Jews. The survey does not, however, present the actual number of such persons, and we have no way of knowing what share  ̶  possibly a very small one  ̶  of the total number of marriages by Jews in this period it has counted, for instance by comparing this with figures from Jewish marriage celebrants.  It did note that only one “Modern Orthodox” Jew in 21 married out, a group which may well be undercounted by the survey’s methodology.  Nor did it ascertain how many non-Jewish spouses subsequently converted to Judaism.

In the 1980s and 1990s I conducted many surveys for the Australian Institute of Jewish Affairs, and I am well aware of how difficult and time-consuming this is, and how little the community supports such surveys financially. My sole aim in offering this critique is to improve any future studies of this kind.

Emeritus Professor Bill Rubinstein taught at Deakin University and at the University of Wales.