'Filter by category' resets when scrolling/loading more products

Hi there,
I’ve got a store gallery which I’ve set to display a certain product category. All items in said category are also sorted into multiple other categories.~ On the gallery I’ve allowed filtering by product category. The page is set to load 20 products.

When using my website, I can filter by a category, but as soon as I click the ‘load more’ button, it resets the filter and just shows all my products. I’ve tried switching the setting to ‘pagination’ and ‘infinite scroll’ and the same issue arises: whenever the page has to load beyond the 20th product the filter resets.

Does anyone have any advice on how to fix this, beyond setting the gallery to initially display a high number of products? Thanks so much.

~ If this is confusing: I sell a few types of products: pattern files, kits, accessories etc. Each type of product can be further categorised into various themes (e.g. Rainbow, Floral, Sarcastic etc). I have a separate gallery page for each ‘type’ (e.g. a page for ‘patterns’) which has the gallery set to display the ‘pattern’ category. I would like that page’s gallery to be able to filter patterns by their theme category (e.g. on the ‘patterns’ page, only display products that fit both ‘pattern’ and ‘floral’). The example given in the settings (The category ”Fruit” can be filtered by the categories ”Apples” and ”Bananas.”) shows this should be possible but I can’t seem to get it to work for me.