This article gives a formal definition of a lognormal family of probability distributions on the set of symmetric positive definite (SPD) matrices, seen as a matrix-variate extension of the univariate lognormal family of distributions. Two forms of this distribution are obtained as the large sample limiting distribution via the central limit theorem of two types of geometric averages of i.i.d. SPD matrices: the log-Euclidean average and the canonical geometric average. These averages correspond to two different geometries imposed on the set of SPD matrices. The limiting distributions of these averages are used to provide large-sample confidence regions and two-sample tests for the corresponding population means. The methods are illustrated on a voxelwise analysis of diffusion tensor imaging data, permitting a comparison between the various average types from the point of view of their sampling variability.