You were a servant at the Phantomhive estate as you were invited there to work for Ciel and protect him and the estate. (You could be an OC or any of the cannon servants in this scenario honestly.)