Mary was supposed to stay with Zechariah and Elizabeth as a handmaiden. With Zechariah position, Jesus would have been brought up in the main Jewish temple. Making it easy for him to spread his teachings.
But Mary had to flee from there due to Elizabeth's jealousy. Joseph was the last option to protect Mary from being stoned to death. He chose to dump her before he changed his mind, due to a devine intervention.
Instead of being brought up at the top of the social ladder, Jesus lived at the bottom of it. And with a very difficult life. Being born in a barn full of animals. Sleeping in a food trough. His family abandoning him when he was 12.
Still single by age 30 as it was the parents responsible for arranging their children's marriage. Plus his family did not believe in him nor support him in any way. Even when they knew the local leaders were looking to kill Jesus.