Okay, because I am a local trivia dork, here goes...
"Gaspee" has two "e's." But that's a minor point.
The White Horse Tavern in Newport is the oldest tavern building in America. The building began as a tavern, and is currently a tavern, but has not always been a tavern. So it can't claim to be "the oldest continuously operating tavern in America." I think that claim to fame belongs to the Union Oyster House in Boston, though I am not sure.
Another fun fact: The Newport Mercury, currently one of Newport's two alternative weeklies, is one of (if not the) oldest newspaper in America. It began in 1758 and the only time it ceased publication was while Newport was occupied by the British during the revolution. The Hartford Courant also claims this distinction.
The Newport County CVB has a whole list of "Newport firsts" here.