The Brown Square House, now the Garrison Inn, is a historic pair of rowhouses at 11 Brown Square in Newburyport, Massachusetts. Built in 1809–10, they form the largest surviving brick building from the Federal period in the city. The building was added to the National Register of Historic Places in 1975, and included in the Newburyport Historic District in 1984.