They were cagey at first but Victor did say that Xander was the son of his brother Titus, to whom he was not close. Beyond the mention of the manslaughter/assault case when Xander/Daniel/Brady were teens/kids (let's not ask, ok? LOL), and the oblique mentions of Xander's doing dirty work for Victor, there has been no explanation as to why Victor dislikes Xander or treats him like pond scum.
This whole thing could be far more interesting if we knew WHY Victor disliked him so, or if he were actually the OTHER Alexander, aka Alex, son of Justin/Anjelica/Adrienne, or even a hitherto-unmentioned brother of Justin's, to increase the family connection.