Most Southerners have a relationship with the Confederate flag that has nothing whatsoever to do with slavery.
In some cases, it was changed to reflect the centennial of the Civil War, which, if you recall, abolished slavery over a hundred and forty years ago.
There's more at the Atlantic's blog about what Henke calls the "revolting history" of the Confederacy, but I'll let it pass. The point is that the evil of the Confederate flag is a fairly recent discovery—it was no problem for the "Dukes Of Hazzard" to have a car called the General Lee with a Confederate flag on the roof from 1979 to 1986, but it did become a problem in the movie remake in 2005.
In the meantime, if you want actual symbols of slavery, remember that Saudi Arabia only abolished slavery, (officially) in 1962, and they still treat migrant labor like slaves. If you want a symbol of slavery, which still exists throughout the Muslim world, what about all those people dressed in headscarves and burnooses? I have it on good authority that Saudis appear in public wearing white sheets; Saudi women dress in black, but wear Klan-style hoods.
(Apparently the anti-Klan laws intended, not unreasonably, to prevent people from appearing in public in masks, do not apply to Muslim ladies.)
So if you're looking for symbols of slavery and hate, check out the Arab and Muslim symbols that are becoming increasingly common in Western society, not the symbols of "good ol' boys, never meanin' no harm."