This is another misperception (for anything so popular, there are bound to be many misperceptions), especially for people with no prior exposure to traditional Chinese martial arts. tai chi is first and foremost a martial art, and there’s never been the slightest doubt about this among high-level traditional Chinese martial artists (see the Bay Area Wing Chun Association link). In fact the legendary Chen style Grand Master Chen Fake (1887-1957) comes from a family that use their fighting capabilities to run security services (in the days without modern weapons). The misperception mostly comes from confusing the training of tai chi with its applications. It is also due to the fact that most modern teachings of tai chi focus on training for its health benefits, and seldom address its martial arts applications. And yes, don’t try this at home by yourself, as it’s probably true that onlytop tai chi artists know how to utilize it for fighting.
On the other hand, one of the attractions of tai chi as a martial art is its principle of leveraging the opponent’s force in fighting to achieve the goal of, to quote from one tai chi classic, “using 4 ounces [of force] to defeat 1000 pounds [of force]”. As another tai chi book says, “What’s the significance of defeating your opponent if it requires that you are stronger and bigger than he is?” And the magic of tai chi lies in defeating your opponents irrespective of their size or power, for you are always using their own power to defeat them.