The Black Tower’s Fishing Hole and the White Tower’s Ore Wall is a factor that could influence one’s decision of what tower to buy, but it doesn’t have much effect on each tower’s usefulness in the long term, so I’m just going to ignore it.
Since almost everything else about the towers is identical, the choice between the two towers mainly falls down to location. The Black Tower sits right next to the Town of Right and Wrong, and has one-way teleportation to Topple Town, allowing you to reach both settlements easily. The White Tower, on the other hand, has one-way teleportation to Right and Wrong, but is a decent distance away from Topple Town, which makes getting to Topple difficult to do quickly without paying 5k at Right and Wrong.
Due to this, I would choose the Black Tower, which I actually did use before I bought my house on Matumada. The Black Tower’s location is very good, especially when you want to grab a Skip Ticket quickly for the Otherworldly Tower. It’s also near some locations with higher level enemies if you want to fight them, and you can just teleport to Topple if you want to go somewhere near that location.