How can you do a queryReferenced on the backend? I'm stuck with a Promise, and need help

This question/discussion arises from this problem:
https://www.wix.com/velo/forum/velo-pro-discussion/can-a-dynamic-list-page-select-multiref-columns

Which I asked several days ago, and has gotten views, but no answers.

TLDR; There is no built-in way to SELECT a datastore item’s multiref columns, so I have to roll my own.

Minimal details:
I have a dataset with multirefs to several other datasets.
masterTable → multiref to referenceTable1, referenceTable2, referenceTable3
referenceTable1 → multiref to masterTable
referenceTable2 → multiref to masterTable
referenceTable3 → multiref to masterTable

The masterTable is involved in a repeater on a dynamic page.

Inside the repeater, I want to show each item’s reference field values.

There is no way that I have found to get the multiref data directly. I have tried about a thousand different approaches, and nothing works.

My next approach involves building a cache of the reference data.

TLDR; #2 - After posting this, I found the warmupData API, and I’m decently sure it’s what I should be using.

https://www.wix.com/velo/reference/wix-window/warmupdata-obj/get

However…either the API is broken, or the docs are wrong. The code examples they give don’t even compile. I copy/pasted them directly into the editor, so I know I didn’t just mistype something.

In the code snippet below, they get a result set from a datastore query and pass it to warmupData.set. But the API for set requires 2 strings (key, value). I’m not sure if the API is not WAI, or if the docs are wrong. If someone from Wix could figure out the correct usage of this API, I think it will solve my problem. For now, I’m getting around it by serializing my object (JSON.stringify) before calling set, and deserializing it on get. But I’d really like to know if the API has a bug. The docs say “New and Improved” and they’re from April:

https://www.wix.com/velo/forum/news-announcements/new-and-improved-warmup-data-api

So I wonder if this stuff is not yet really vetted.

async function getData() {
5  const results = await wixData.query("MyCollection")
6    .find();
7  if (wixWindow.rendering.env == "backend") {
8    wixWindow.warmupData.set("myWarmupData", results.items);  <--- This line is broken
9    wixWindow.warmupData.set("myWarmupMessage", "Rendering in the backend.");
10  }
11  return results;
12}

TLDR; #3 - After posting TLDR; #2, I woke up the following day and went back to work. My editor still contained the copy/pasted example code, but now it compiled. I changed nothing.

Great! I thought. Maybe someone read my feedback and fixed the API. However, it would not run. There was something completely broken about the imports, and all I got were various forms of TypeError: Cannot read property ‘warmupData’ of undefined. The imports in the docs are totally inconsistent. Some of them claim to import { warmupData }, others claim to import wixWindow. There are several other permutations of inconsistent docs leading to a really frustrating experience. Anyway, after about 30 minutes of scratching my head in baffled incredulity, my site went blank…I was logged out…made to re-log in…and then the site and editor came back…I thought this was totally bizarre…Then, without changing one single line of code, I ran the example code again, and it worked. It makes me think that someone is fixing this broken stuff as I’m trying to use it.

All the docs for warmupData do say that this is the “New and Improved” way, and those docs are from April. I honestly wonder if I am the first person to use these APIs.

TLDR; #4 - Ugh…I now know what was causing my Promise nightmares, and it’s a JS thing that I assumed would work, but actually doesn’t. I’m really not the most nuanced of JS coders…so, just to satisfy anyone’s curiosity, and so you don’t smash your own head against this brick wall wondering why your awaits won’t await…

array.forEach doesn’t work. Normal for loops do.

// This works:
for (i = 0; i < myarr.length; i++) {
  var foo = await doSomethingAsync(myarr[i]);
}
// This doesn't:
await myarr.forEach(async (item) => {
  var foo = await doSomethingAsync(item);
});

Anyway…just in case the warmupData API isn’t usable…back to my issue:
Once the cache is built, I can store it in memory/session/whatever.

Then in the repeater, I can show or hide the repeater element based on whether the cache contains the id if THIS item.

This data is slow changing…so slow changing that it’s basically NON changing. Once a row in the masterTable is created, it is rarely ever updated.

Now the problem becomes how to do this efficiently, and on the BACKEND, which is faster.

Approach 1:
use a wixData query on the referenceTable, use an include:

wixData.query(‘referenceTable1’).include(‘masterTable’)

THIS DOES NOT WORK. You can only ever get the first 50 entries. I have not found a way to get them all.

Approach 2:
use a wixData query on the referenceTable, and use queryReferenced:

async function doQueryReferenced(accumulator, referenceTable_id) {
 var page = 0;

 var result = await wixData.queryReferenced('referenceTable1', referenceTable_id, 'masterTable');
 var hasNext = result.hasNext();
    result.items.forEach((item) => {
        accumulator.add(item._id);
 });
 while (hasNext) {
        result = await result.next();
        page++;
        hasNext = result.hasNext();
        result.items.forEach((item) => {
            accumulator.add(item._id);
 });
 }
}

This works…sort of…
queryReferenced can only be run for a single row. So I have to run this on every row in my reference table. Thankfully, there are only 2.

async function buildCache() {
  // use a set to filter out duplicates. masterTable rows can have 0, 1, or 2 items from referenceTable1.
  // I don't actually care WHICH value it has...just that it has ANY value.
 var accumulator = new Set();
 
 await doQueryReferenced(accumulator, 'referenceTable1_id_1');
 await doQueryReferenced(accumulator, 'referenceTable1_id_2');
}

I can run this on the client, but it’s really expensive, and I’d rather do it on the back end.

I am STRUGGLING with how to get this to run on the backend. I need to page through all the data, build up the accumulator, and then return the final result to the client. I am struggling with how to get this to work, and I am lost in stupid Promises, especially because I have to loop through the pages, and each loop involves another dive down into Promise-land. I’ve only managed to return an empty accumulator to the client, and then sometime later, everything resolves on the back end. I cannot figure out how to set up the promise so that it resolves ONLY when everything on the back end is done and the accumulator is filled. This can’t be that hard, but I am so frustrated that I am not thinking clearly.

If anyone can possibly help here, I would really appreciate it. I have been beating my head into this brick wall for the better part of a week, and I am SUPREMELY frustrated.

On another note:
It really should not be this difficult to SELECT multiref values. I get it that this could lead to references referencing other references referencing other references and other scary things (like DOGS AND CATS LIVING TOGETHER! - Bonus if you’ve read this far, and know where that comes from), but couldn’t a compromise be reached by just returning the IDs of the reference values and not the entire reference value?

If it is so impossible to SELECT multiref values, then WHY DO MULTIREFS EVEN EXIST? It seems like…you can set them…but then you can never use them again. Pardon my bluntness/rudeness, but this seems DUMB.

IF IT IS POSSIBLE TO SELECT MULTIREF VALUES, THEN SOMEONE PLEASE TELL ME HOW.