What a strange time when what we called the left was not on the left, the right was not on the right, and the centre was not on the centre. However, this is how politics should be understood and work. Politics must be all encompassing, for everyone, for the common good.
The political spectrum is important, it is useful for classifying and determining our political positions. However, it cannot be rigid. Each corner of the political spectrum is not rigid and completely separate from the others. Hence why there are many models of this.
People can be on the 'left' or on the 'right' in looking at certain things. Too many variables are needed to truly determine an individual's political position. Most people see other people's political positions only superficially, they only see what they see, just on the surface.
So, it shouldn't matter if people accuse you of being leftist or rightist. Even so, it is still potentially dangerous. I personally avoid any political labeling of myself. It is easier for me to refute accusations than to defend my political position. People change, so do I.