Where to start?
If your load sucks down a lot of current - it will crowbar (suck enough power to lower the voltage when first turned on) the voltage supply to the Axe - not forever, just long enough to reset or cause problems. The supply impedance (like resistor in series with your battery or power supply, measured in ohms) will drop some voltage internally - high current and the voltage droops for a moment - but that moment can cause problems. You engage the load (like a servo) and it is a dead short until it starts to move - that short resets the processor. (you get around that by isolating the 'axe from the load with a diode and cap - to keep the voltage up to the axe, during the momentary droop)
High speed noise, is different (frequency in EEspeak ) versus low speed. Like "noise" on the supply line - very fast transitions . . . A simple plate capacitor - like aluminum foil, separated by glass plates is amazingly good for high frequency (very fast droop that will reset the picaxe) (nano-microsecond ranges). Big plates of conductive metal in close proximity can't be beat for high frequency capacitors> the capacitor construction enters into it.
In any event brush motors cause high frequency noise - you put a very low inductance (lets DC pass and shorts out AC) capacitor across the brushes as close to the motor as you can get - a .01-.1 micro farad ceramic cap (low impedance) directly across the brushes.
Servos, or anything pulling down lots of current requires a low frequency solution - a lot of storage to supply the load quickly and not suck down the power supply. Then you usually add a diode to isolate the axe from the momentary low power supply, with a capacitor to supply energy while the power supply is low - in that case the capacitor is very large in size and the impedance doesn't count for much.
That's the easy stuff - if you use analog parts and an analog system the rules change again. The pcb wire traces can drop enough voltage to cause the analog parts to react to that drop. So the heavy loads get connected close to the battery or power supply with their own wire leads and are bypassed with capacitors.
Voltage regulators help - but that only takes care of the supply side - your grounds can also cause problems and those are solved with bypass capacitors and/or circuit layout.
My 2 breadboards have switches that allow me to run at 4.5 or 3 volts - at 3 volts I see how to improve the layout and improve bypassing.
During the "vacuum toobe era" we use a single point ground for all signals (usually a big bolt on the chassis with lots of wires - lugs coming from it) or a very heavy piece of wire (like #14 since we were dealing with micro amps).
Those solderless breadboards? They also need to be bypassed and layed out so that heavy currents to ground and power don't share the same wires with sensitive signals - use a separate jumper. (and they don't work at over 50 MHZ if your aim is to build a RF transmitter)