The configurable menus in WordPress have a checkbox labelled “automatically add new top-level pages to this menu” where if checked every time you make a new page it gets added to the menu. That might be good for drop-down menus but otherwise very rapidly clutters up the page. By default themes come with a ‘Primary menu’ and you can configure that as you would like, HOWEVER the ‘automatically add new top-level pages to this menu’ button, although you can check and uncheck it, doesn’t do anything… new pages are always added to the primary menu regardless. You can go into the ‘customize’ option for the theme and manually remove all these pages, but that gets to be a hassle.
To fix the problem, you have to actually create a new menu yourself, any menu at all, and use that. Confusingly, ‘Primary menu’ is both the name of a menu, and the location of a menu. After you’ve made your own menu (I called mine ‘main nav bar’) you can set that to be in the ‘Primary menu’ location and you’re then free to delete the ‘Primary menu’ menu and things work as expected.
It’s not a bug, it’s a feature!
(the checkbox not working on the default setup is definitely a bug)