When you are invited to join a party, the invitation is valid for 90 seconds of real-world time. If you do not choose "Join" within this period, you will not be able to join the party and the invite will be canceled. If this happens, either approach the person who invited you directly to ask for another invitation, or select "Seek Party" from the Party menu and wait to be invited to another party.