I have heard three reasons why practicing the hand form on the "left side" might be inadvisable. I did not hear these reasons from people who spoke with clear "authority" or certain knowledge, and so I have filed them away in the mental corner I reserve for things waiting further clarification. Two of these reasons would seem to apply to learning and practicing the sword and saber forms using the left hand. On the other hand, I have also heard some "authorities" specifically recommend learning and practicing certain non-traditional forms to both sides.
I once heard Yang Zhenduo respond to a question about practicing the sword form with the left hand. From what I recall, his answer was something like the following. "In China, we learn and practice the sword in groups. In groups, it would be impractical and even dangerous for some to practice in one direction and others to practice in the opposite direction at the same time. That is our tradition. In America, you like to innovate. Maybe you will figure out a way to have some people practice with the right hand and others with the left."
What I took from Yang Zhenduo's statement was that there was no theoretical barrier to practicing with the left hand, but that it was impractical in the classroom setting.
I personally think you can do what you want in private practice, but that most people that stay with Tai Chi will eventually want to practice along with others. There are also some links between the sword and saber forms and the hand form that might be obscured if you practice them to different "sides". But then, I am right handed and so maybe biased.