Living in poverty places children at very high risk for problems across a variety of domains, including schooling, behavioral regulation, and health. Aspects of cognitive functioning, such as information processing, may underlie these kinds of problems. How might poverty affect the brain functions underlying these cognitive processes? Here, we address this question by observing and analyzing repeated measures of brain development of young children between five months and four years of age from economically diverse backgrounds (n = 77). In doing so, we have the opportunity to observe changes in brain growth as children begin to experience the effects of poverty. These children underwent MRI scanning, with subjects completing between 1 and 7 scans longitudinally. Two hundred and three MRI scans were divided into different tissue types using a novel image processing algorithm specifically designed to analyze brain data from young infants. Total gray, white, and cerebral (summation of total gray and white matter) volumes were examined along with volumes of the frontal, parietal, temporal, and occipital lobes. Infants from low-income families had lower volumes of gray matter, tissue critical for processing of information and execution of actions. These differences were found for both the frontal and parietal lobes. No differences were detected in white matter, temporal lobe volumes, or occipital lobe volumes. In addition, differences in brain growth were found to vary with socioeconomic status (SES), with children from lower-income households having slower trajectories of growth during infancy and early childhood. Volumetric differences were associated with the emergence of disruptive behavioral problems.