We can travel to any US territory as long as you don't transfer anywhere abroad.

US Territories include:

• Puerto Rico
• US Virgin Island
• Guam
• Northern Mariana Islands
• American Samoa
• US Minor Outlaying Islands

Only issue is that USMOI are basically uninhabited or military bases. Guantanamo Bay is also US territory but well, travel there even as a Citizen isn't the easiest. I doubt they'd buy the argument of pilgrimage to the last known Blockbuster.

Basically travel to US Territories is DOMESTIC travel, and doesn't count as departure or entry. Not any different than flying to Hawaii or Alaska.