Not All Countries Are Analytically Equal

By Robert Higgs
Mar. 28, 2013

Economists and other social scientists have a long history of conducting analyses based on cross-sectional international data. Sometimes these studies examine a handful of countries; sometimes they examine scores of countries. The studies with the larger samples are, it seems, generally viewed as more solidly based than those with smaller samples.

It is common among economists for objections to particular studies to take the form, "But what about country X? It certainly did not follow the pattern your model predicts and your data display." The belief runs deep that only a study that can account for every country's data under the same explanatory setup, whether theoretical or econometric, deserves much respect. Economists seek general explanations, and those that fit every country are commonly regarded as the most general.

All of this seems reasonable as long as we do not spend much time pondering the analytical comparability of the countries. However, when we pause to consider how much countries differ from one another in a great variety of dimensions, we may well begin to wonder whether it makes sense to fit every member of a large group of countries into the same framework of analysis.

Simon Kuznets, who probably did more than any other economist in modern times to systematize the measurements necessary for cross-sectional international analyses and to carry out many such analyses, justified the use of nation states as units of analysis by arguing that each such state has unique discretion and importance in setting the policies, laws, and other institutions that form the context of incentives and constraints in which economic actors must operate; hence nation states are better units of analyses than, say, demographic groups, regions, or cities scattered across different states. Even if we concede Kuznets's point, however, it remains true that nation states differ in a host of ways.

Perhaps the most important dimension of difference is the sheer size of population or economic output. Approximately 200 nation states currently exist. Data compiled by organizations such as the United Nations, the World Bank, and the International Monetary Fund are made available to researchers in convenient forms on a regular basis, and hence these data serve hundreds or perhaps thousands of researchers as grist for their analytical mills. Not uncommonly the researcher employs the data for every country for which pertinent data are available in the standard compilations.

In econometric studies, it is common for the individual country observations to enter the analysis with equal weight, regardless of the differences in the countries’ size. So, for example, one commonly sees analyses for every country that belongs to the Organization for Economic Cooperation and Development (OECD) or for every country in Latin America. Within the former group, tiny countries such as Estonia, Iceland, and Luxembourg may enter the analysis on equal terms with large countries such as France, Germany, Japan, and the United States. Within the latter group, small countries such as Belize, Honduras, and Uruguay may enter the analysis on equal terms with giants such as Brazil and Mexico.

In statistical analyses, the upshot is that outliers from the calculated central tendencies receive the same weight in the summary statistics (e.g., the coefficient of determination in a regression equation) whether they happen to be Iceland or Germany, whether they happen to be Belize or Brazil. Thus do international cross-sectional statistical studies in effect suppress the identities of the observational units as if they did not matter.

But surely we cannot have equal confidence in the findings of two studies, one of them with Italy and France as the outliers and another with Luxembourg and Denmark as the outliers. Likewise in Latin America, we might well have less satisfaction with a study whose overall pattern cannot account for Brazil and Argentina than with one whose overall pattern cannot account for El Salvador and Belize.

An old colleague and dear friend of mine from days gone by, Morris D. Morris, was a student of Indian economic development. I recall his telling me more than forty years ago that despite the dreary pace of overall Indian development, one saw much stronger performance in Gujarat, and that this region alone had a population comparable to the largest western European states. Morris noted that if Gujarat were listed as a separate country in the international data, its income would place it somewhere in the middle, rather than near the bottom, where India as a whole ranked in those days.

Many of the world's great cities—Mumbai, Shanghai, Sao Paulo, and Mexico City, for example—have populations and economic outputs that place them orders of magnitude above many of the world's smaller countries. And often, as in China today, particular regions may have income levels and rates of economic growth far out of line with those of huge backwater regions where mass poverty and slow rates of growth prevail. Almost all geographically large countries harbor substantial interregional differences, especially in the Third World.

The foregoing commentary is not intended to warrant a conclusion that cross-sectional international studies are worthless. Indeed, they can be revealing and instructive. Yet it behooves everyone who deals with them to bear in mind the incomparability of their units of analysis in population, total output, and other important variables, as well as the internal variation that marks all but the very smallest countries.

After all, you can fit a lot of Denmarks, Omans, and St. Lucias into any one of the world's economic and demographic giants. By treating all of these units of analysis on the same basis, we risk overlooking the truly important patterns because we have implicitly assigned excessive weight to the little fellows.