Short bio: Computer Scientist, FOSS supporter (read more)
Tux Machines (TM)-specific
Q: Is it possible to set up chapters or sections in an OpenOffice Writer document so that I can jump to them easily, like maybe with tabs or a tree like some PDFs have?
A: Definitely. Press F5 and you'll see the Navigator window.
You can expand the Headings option and just double-click the heading name to jump to that part of the document.
You must either use the Heading1 through Heading10 styles on the text that is used as your document headings.