Compression is a tradeoff, not magic
A PDF can contain text, fonts, images, scanned pages, embedded thumbnails, forms, and metadata. Compression works differently depending on what is inside the file.
For scanned PDFs, image compression matters most. For text-heavy PDFs, file size may already be efficient, so the safest option is often moderate compression.