Note that Nephi said "the voice of the Lord came unto my father" (v. 5). Who is the prophet? Lehi is still the prophet. The voice of the Lord does not come to Nephi and say, "It is time to go. Get your father and family and depart." The voice of the Lord came unto the prophet Lehi and told him that it was time to enter the ship. Verse 6 begins with "on the morrow." The Lord said it was time, and they went the next day. They obeyed immediately as daunting as that might have seemed.
Book of Mormon Central, "Has the Location of Nephi’s Bountiful Been Discovered? (1 Nephi 17:5)," KnoWhy 259 (January 19, 2017).