The opening sequence presses the young man to have a child. Obviously this would involve making love to a woman, and thus to a degree distancing the speaker from his beloved. The specific woman he chooses doesn't appear until later--though earlier than 127 iirc--but from very early on the speaker is aware of the risk that the youth will come to prefer the love of a woman to his.
See for example number 20:
A woman's face with nature's own hand painted,
Hast thou, the master mistress of my passion;
A woman's gentle heart, but not acquainted
With shifting change, as is false women's fashion:
An eye more bright than theirs, less false in rolling,
Gilding the object whereupon it gazeth;
A man in hue all hues in his controlling,
Which steals men's eyes and women's souls amazeth.
And for a woman wert thou first created;
Till Nature, as she wrought thee, fell a-doting,
And by addition me of thee defeated,
By adding one thing to my purpose nothing.
But since she prick'd thee out for women's pleasure,
Mine be thy love and thy love's use their treasure.
