How to limit the size of files users upload to my website?

Well, you didn’t really pick a file since File is undefined in your onCustomValidation() function.

What you want to do is get the array of files from the button value, and then check each one for validity. Something like this:

$w("#uploadButton1").onCustomValidation((value,reject)=>{
        let files = $w('#uploadButton1').value;
        files.forEach((file) => {
            console.log(file.size);
	    // check file.size here
        });
})