I'm not familiar with Fort Victoria but I do know where it is. As C-Camper says, it's not far from downtown (legislature, museum, Empress Hotel, etc). There is one much closer, can't recall the name as I type, right on the inner harbour roughly across from those features I just identified. Although I've not stayed there, one feature that makes me think my wife and I will spend a few days there art some point is the small boat-bus working the harbour that you can take to cross the harbour to the downtown features.
Where we have stayed is Weir's Beach, generally east of Victoria (and of Fort Victoria) and right on water. We like it there because it is small and fits our "do nothing while we're there" approach. Mind you, when we were there last spring I spent quite a bit of time observing a pair of nesting swans.
Ciao,
Doug