Not only would you be making no progress whilst the trays are off, but the potential is worse than just your teeth moving back. Since they have already been moved from their initial positions during treatment, it's also possible that they might move in unanticipated ways. At best you'll be setting treatment back such that you'll need to wear your current trays for some extra period of time (or probably the previous tray, since you would only just have been moving forward). But it's also possible you might throw things off so much as to require new impressions, and delay wiating for new trays.
Just wearing them at night may be a bit less risky. But I still think that you are being extremely silly to risk compromising your treatment in this way.
It seems as if Invisalign might have been a poor choice for you if something so trivial as a few days away is sufficient for you to feel unable to comply with the necessary aligner wear
There truly is no reason that you could not wear your trays in the normal way for the duration of this trip, removing them only for meals, as usual.
Do discuss with your
orthodontist.