Why is the font size on Apple Books different on every chapter?

It's been happening for months now on each book whether I've had it for years or new. Nearly every chapter is a different font size and I'm constantly having to stop and change it because it's either huge or tiny. Is anyone else having this issue? Or know how to fix it? I've tried updating and redownloading but nothing works.