Does my level of fitness affect how fast I use my air?

Yes it does.  The more cardiovascular fitness you have, the easier it is for you to breathe.  Those who are unfit and become out of breath with even mild exertion will have trouble diving.  When we are out of breath we breathe more deeply and more often in order to compensate and this will use up a tank of air in a fraction of the time a fit person may use theirs.  Divers who do not possess a good level of cardiovascular fitness will not be able to enjoy scuba diving beyond the very basics.