Legitimate but slightly sneaky fix for the issue of racing for addition / removal. An alternate fix would be a multimap.