Excel is trying to open non .xlsx files

I think ku4hx is on to something. Assuming your Excel installation is in fact attempting to open everything in your Documents folder, try this: create a new, temporary folder on your Desktop (for example), and drag the entire contents of Documents to it. Emptying the Documents folder will presumably allow you to open Excel and examine that "At startup, open all files in:" field, since that folder will be empty. Presumably the Clear button adjacent to it should clear that field.

This should be harmless enough but I would be remiss if I did not point out the necessity to create a Time Machine backup first, if you have not done so already: Back up your files with Time Machine on Mac - Apple Support.

Determine if that works, then revert the modification by dragging the entire contents of that temporary folder on your Desktop back to Documents, where they began.

You would be much better off asking your question in a Microsoft product support forum, such as https://answers.microsoft.com/.

Apple Support Communities might not be the worst place to find help using Microsoft products, but it's close.

What non .xlsx files exactly?

Excel can open many different files noted by extension, but it can only open Excel acceptable files. If you open any qualifying Excel file and then do a Save As you'll see all the possibilities .

My guess is you've managed to set an auto open file that is not an Excel file, by putting that file into the XLStart folder. Intentionally or accidentally makes no difference..

Automatically open a specific workbook when you start Excel

To automatically open a specific workbook when you start Excel, you can place that workbook in the XLStart folder, or you can use an alternate startup folder in addition to the XLStart folder.

However, John Galt is correct, ask in an Excel forum.

You're welcome but if not for ku4hx's screenshot I would have had no idea Excel had that function. I also walked away from MSFT's increasingly inconvenient bloatware almost two decades ago in favor of better alternatives.

Who knew ASC would be better than Microsoft's support forum.

Look here, "At startup open all files in ....":

I suggest you take the simple way out and drop the use of MS Office, al least Excel, in favor of free LibreOffice or FreeOffice.