The main factor was really the size of the fire -- the total area it covered. Building fires typically start with a small fire -- say a burning cigarette on a stack of papers -- which gradually spreads through a larger area. In that situation, the fire is most intense where it has the most fuel (stuff that can burn), and it significantly weakens the support structure only at those most intense points. If a fire starts in the northwest corner of a skyscraper floor, by the time the fire reaches the southeast corner, the starting fire at the starting point will have burned through most of the fuel, and the fire will not be as intense. The result is the fire doesn't put maximum strain on the total support structure all at once. It strains different parts of the support structure in turn, over time.
In the case of the World Trade Center, the burning jet fuel spread the fire across several floors in a matter of seconds. This massive fire put exceptional strain on the structure at nearly all points on those floors.
Additionally, the report suggests that the force of the collision removed much of the fire-resistant material sprayed on the steel, making the structure more susceptible to heat damage.
The heat expanded, twisted and buckled the steel support structure, gradually reducing the building's stability. Any number of things could have happened during this period. For example, connections between vertical columns and floor trusses probably broke, dropping sections of floor on lower levels and breaking connections between the core and the perimeter wall, possibly causing columns along the perimeter to buckle outward. Every broken connection or buckled length of steel added to the force acting on connected steel segments, until the entire structure was weakened to the point that it couldn't hold the upper section of the building.
When this happened, the top part of each building collapsed onto the lower part of the building. Essentially, this was like dropping a 20-story building on top of another building. Before the crash, this upper structure exerted a constant downward force -- its weight -- on the superstructure below. Obviously, the lower superstructure was strong enough to support this weight. But when the columns collapsed, the upper part of the building started moving -- the downward force of gravity accelerated it. The momentum of an object -- the quantity of its motion -- is equal to its mass multiplied by its velocity. So when you increase the velocity of an object with a set mass, you increase its momentum. This increases the total force that the object can exert on another object.
To understand how this works, think of a hammer. Resting in your hand, it doesn't hurt you at all. But if you drop it on your foot, it can do some damage. Similarly, if you swing the hammer forward, you can apply enough force to drive nails into a wall.
When the upper structure of each tower fell down, its velocity -- and therefore its momentum -- increased sharply. This greater momentum resulted in an impact force that exceeded the structural integrity of the columns immediately underneath the destroyed area. Those support columns gave way, and the whole mass fell on the floors even farther down. In this way, the force of the falling building structure broke apart the superstructure underneath, crushing the building from the top, one floor at a time.
To put it another way, the potential energy of the building mass, the energy of position it had due to its height and the pull of gravity, was converted into kinetic energy, or energy of motion (the report puts the total potential energy for WTC 1 at 4*10^11 joules). This is the same basic principle that professional demolition blasters use to bring down unoccupied buildings.
WTC 2, the second tower hit, actually collapsed before WTC 1. This was most likely due to two different factors. First, WTC 2 probably suffered greater immediate damage -- the second plane to hit was going faster than the first. Secondly, the plane that hit WTC 2 crashed lower on the building than the plane that hit WTC 1. Consequently, the strained support columns in WTC 2 had a greater load pressing down on them than the strained columns in WTC 1, so it would make sense that they reached the buckling point more quickly.
While the towers' support structure ultimately couldn't withstand the raging fire, it was strong enough to save thousands of people's lives. Around 99 percent of the people below the impact in each tower were able to evacuate before the buildings collapsed. If the towers hadn't been built with redundant structural stability, the death toll would have easily been in the tens of thousands.