For some unknown reason macOS software update seems to have a bug where it will only see the newest version of macOS available and realize it is not compatible with your Mac and disregard that there may be an intermediate OS available as compatible & upgradeable.
Before you attempt to upgrade to macOS 10.13, I recommend you run DriveDx to check the health of the hard drive since an OS upgrade install can easily put the drive over the edge it is probably already teetering. If the upgrade fails for some reason, then you won't be able to check the health of the hard drive as easily to know if that was the ultimate source of the problem.
I also highly recommend you have a good backup before attempting the upgrade. If the hard drive is beginning to fail, then the upgrade process will likely make the failure worse where you may not be able to easily access the data on the drive.
I would also recommend creating a bootable macOS 10.13 USB installer while you can still boot this laptop normally. This way if the drive is bad you will have a method of reinstalling macOS onto another drive if you want to continue using this laptop.
Since the Apple Diagnostics won't boot, If you want to try checking the memory, then you can use a bootable Memtest86 USB stick. Use the downloaded Memtest86 .img file as a source for Etcher (Mac/Windows/Linux) which will create a bootable Memtest86 USB stick for you. Option Boot the Memtest86 USB stick and select the orange icon labeled "EFI".
The current version of Memtest86 only allows testing for four loops. If no errors are reported and the test does not freeze up, then try running modifying the Memtest86 CPU settings and change it to "Sequential" and "Round Robin" to test again. Recently I have discovered Memtest86 seems more likely to detect hardware issues with these CPU settings. In fact it will usually cause Memtest86 to freeze fairly quickly & regularly which would indicate some sort of hardware issue (most likely bad memory, but it could also be the Logic Board)...it rarely shows an actual memory error.
Edit: Forgot to include a link to an Apple article with instructions for creating a bootable macOS USB installer:
Create a bootable installer for macOS - Apple Support