Wait, can you still get back to your bases in the old galaxy? If so, how to you get back to the new galaxy (if you want to given there's no real difference between galaxies).
Yep. Teleporters work from galaxy to galaxy. Normally, your list of places to go just consists of places that you've been before. If you go to the Anomaly station, though, there's a huge teleporter that can dial up many, many bases that people have built. If you don't want to mess with the whole drive-to-the-center thing, you could simply use that to teleport to a base in whatever galaxy you like.
Your ship follows you wherever you go. Your freighter can be summoned to different galaxies, too, as long as its warp drive can deal with the color star you are at.
{Lore-wise, this makes absolutely no sense at all.}
There IS a difference between galaxies, by the way, it's just not much of one. Some have more paradise planets, some have more dead planets, and some have more severe storms.