White flight:
The phenomenon of upper and middle class whites moving out of cities and into the suburbs is known as white flight. The opposite of white flight is gentrification, a process in which wealthy whites move back into an urban area, displacing the current residents and rapidly driving up the cost of living so that the previous residents are forced to move. Both practices have been extensively documented by students of demographics and urban development. White flight, in particular, has negative connotations, especially for those left behind in the suddenly impoverished neighborhood.
White flight began on a large scale after the Second World War, when African-Americans began to try to establish homes in America's cities. Many of these men and women were starting to enter the middle class themselves, with good jobs, education, and community values. In many cases, however, racism led whites to attempt to force blacks out. When this proved unsuccessful, the white population moved to the suburbs, establishing new and primarily racially homogeneous communities. This practice was termed "white flight," and resulted in class and racial segregation in many American cities.
http://www.wisegeek.com/what-is-white-flight.htm
(there's some shocking comments after this article, btw)