The issue of skin color played a significant role during slavery, but more so in the years following emancipation.  For more info, check out influences and here.