There are a few sectors in which Americans could still feel the pain of higher levies longer term.