As you ascend to appropriate altitudes for training (5000-8500ft / 1600-2600m is generally considered to be ideal), the amount of oxygen that is carried by the hemoglobin in the blood is reduced, resulting in less oxygen being delivered to working muscles. In endurance events that rely on the availability of large amounts of oxygen (generally, an event whose duration is longer than about two minutes), a decrease in delivery results in below-normal performance. However, this is true only in low-speed endurance events, such as running, swimming, or rowing. (In high speed endurance events, such as speed skating or cycling, the lessened air resistance experienced at altitude is more beneficial than is the loss of aerobic power and performance is actually better at moderate altitude.)
With time spent training at altitude, the body makes some physiological adjustments and performance at altitude will improve over a period of several weeks, as does sea level performance for some time after coming down from altitude.