Charcoal is carbon. (See this Question of the Day for details on how charcoal is made.) Activated charcoal is charcoal that has been treated with oxygen to open up millions of tiny pores between the carbon atoms. According to Encylopedia Britannica:
The use of special manufacturing techniques results in highly porous charcoals that have surface areas of 300-2,000 square metres per gram. These so-called active, or activated, charcoals are widely used to adsorb odorous or coloured substances from gases or liquids.
The word adsorb is important here. When a material adsorbs something, it attaches to it by chemical attraction. The huge surface area of activated charcoal gives it countless bonding sites. When certain chemicals pass next to the carbon surface, they attach to the surface and are trapped.
Activated charcoal is good at trapping other carbon-based impurities ("organic" chemicals), as well as things like chlorine. Many other chemicals are not attracted to carbon at all -- sodium, nitrates, etc. -- so they pass right through. This means that an activated charcoal filter will remove certain impurities while ignoring others. It also means that, once all of the bonding sites are filled, an activated charcoal filter stops working. At that point you must replace the filter.
Check out the next page for more links.