Wix PAY API/START Pay Not as a Pop-Up, is this possible?

Generally, a pop-up window that gets blocked IS a separate window/page. It’s up to the browser to decide whether it will be a pop-up or a new tab, and both will be blocked if the User for some reason decided to block any new windows/tabs opened from a particular site.

Can you clarify what exact Wix Pay API you are using? Is it startPayment ?
Also regarding the Square - did you connect Square as a credit card Payment Provider in the “Business Manager - Accept Payments”?