From personal experience it is the pressure that determines the acceleration of the balloon. A small, taut balloon will take off faster than a stretched balloon.
Mind you the bigger the balloon the further it will travel and it will continue to increase in speed while it is still pushing air out the back.
