You are receiving some very good advice and most of it makes some sense for your scenario, however I have a slightly different theory for what is ailing you.
I honestly believe the problem is in your footwork, as was mine.
Your stance sounds weak, almost like your back hip, the one furthest from your opponent, is not in proper alignment to allow you to root properly.
I think this because this sounds very nearly like the same issue I was corrected on recently by my instructor in Yang style TCC and since that day I have found more stability in my stance no matter what direction I am pushed from.
If you don't root properly, you will easily be offset by anyone if they find the proper angle to push against your body. This is why we root firmly and solidly, so that we cannot be offset no matter what angle the energy comes in from.
Once you are able to stand firmly with a solid root that connects your energy to the center of the earth and the top of the heavens, then you will be able to stand confidently against any opponent, no matter what martial style they espouse, and exchange full and empty with them for as long as you'd like.
I'm not sure, but I believe it was Yang Cheng Fu who said that if you have a defect in your form, look to your legs. It was one of the Yangs, anywho.
That's the best advice I can give you. After having done pushing hands of a different sort for quite some time, and thinking I was really doing something, it was brought to my attention, rather dramatically, by my instructor that my stance was only really good for bracing against forward and backward incoming energy, but as soon as energy came in from the side I was easily dispatched.
Does this sound familiar to you? It is what it sounds like is happening to you during your encounters. As long as you accept the energy directly, head on, you are fine, but once you offset the energy out to the side your opponent simply redirects it back towards you and you collapse.
This would be due to an incorrect placement of your back hip, as I said.
Your Certified Coach or Center Director would be the best person to get the details from, certainly, but in a nutshell it sounds as if you are not properly tucking in your hips (as in preperatory position, roll them forward, that stays true throughout the entire form and in any stance practice) and almost certainly it sounds as if you are not opening the kua properly. Also, the correct alignment of your knee with your toes in both legs is crucial, they must be aligned in the same direction in order for you to have a firm stance, along with the kua open and the hips properly rolled.
My Yang Cheng Fu Center Director proved this to me very easily, by tossing me about like a rag doll when I did this wrong, and has since worked long and diligently with me and my fellow students at our Center to correct this flaw in our stances, and now I have begun the long, long journey towards real push hands along with my fellow students.
We have also been told to "invest in loss" and expect to keep doing so for a long, long time until we get this concept, and many others, correct.
I consider us extremely fortunate to have such a good instructor.
Footwork, that's the key here. If you can't stand solidly, you cannot do anything else correctly in tai chi, or anything else for that matter.
Look to your legs, I believe that is where you will find the fault you are feeling, as did I.
Good luck, and let me know if this works for you. Or, if I'm completely wrong, not a far fetched scenario, please let us know what you did that worked for you to correct the situation.
[This message has been edited by Bamenwubu (edited 11-22-2004).]