Visible light is only part of the electromagnetic spectrum. Electromagnetic radiation is not monochromatic -- it's made up of a range of different wavelengths, and therefore energy levels. (See How Light Works for a good discussion of the electromagnetic spectrum.)
Light can be separated into different wavelengths, which we can see in the form of a rainbow. Since the light that hits our cell has photons of a wide range of energies, it turns out that some of them won't have enough energy to alter an electron-hole pair. They'll simply pass through the cell as if it were transparent. Still other photons have too much energy. Only a certain amount of energy, measured in electron volts (eV) and defined by our cell material (about 1.1 eV for crystalline silicon), is required to knock an electron loose. We call this the band gap energy of a material. If a photon has more energy than the required amount, then the extra energy is lost. (That is, unless a photon has twice the required energy, and can create more than one electron-hole pair, but this effect is not significant.) These two effects alone can account for the loss of about 70 percent of the radiation energy incident on our cell.
Why can't we choose a material with a really low band gap, so we can use more of the photons? Unfortunately, our band gap also determines the strength (voltage) of our electric field, and if it's too low, then what we make up in extra current (by absorbing more photons), we lose by having a small voltage. Remember that power is voltage times current. The optimal band gap, balancing these two effects, is around 1.4 eV for a cell made from a single material.
We have other losses as well. Our electrons have to flow from one side of the cell to the other through an external circuit. We can cover the bottom with a metal, allowing for good conduction, but if we completely cover the top, then photons can't get through the opaque conductor and we lose all of our current (in some cells, transparent conductors are used on the top surface, but not in all). If we put our contacts only at the sides of our cell, then the electrons have to travel an extremely long distance to reach the contacts. Remember, silicon is a semiconductor -- it's not nearly as good as a metal for transporting current. Its internal resistance (called series resistance) is fairly high, and high resistance means high losses. To minimize these losses, cells are typically covered by a metallic contact grid that shortens the distance that electrons have to travel while covering only a small part of the cell surface. Even so, some photons are blocked by the grid, which can't be too small or else its own resistance will be too high.
Now that we know how a solar cell operates, let's see what it takes to power a house with the technology.