If you put helium in a balloon and let go of the balloon, the balloon rises until it pops. When it pops, the helium that escapes has no reason to stop -- it just keeps going and leaks out into space. Therefore, there is very little helium in the atmosphere at any given time.
The helium that is in the atmosphere comes from alpha particles emitted by radioactive decay (see How Nuclear Radiation Works for details on alpha decay). In places that have a lot of uranium ore, natural gas tends to contain high concentrations of helium (up to 7 percent). This makes sense, since the decay of uranium emits lots of alpha particles and a natural gas pocket tends to be a sealed container underground.
Helium is cryogenically distilled out of natural gas to produce the helium we put in balloons.
Here are some interesting links: