The opulent dwelling is steps away from the iconic Kennedy family compound, and was owned by Swift in 2012, when she enjoyed a short romance with Robert F. Kennedy Jr.’s son Conor.