Based on the BareFeats test results kindly linked to by Lance it would appear that High Sierra already has drivers for the Vega 64 at least built-in.
Therefore the only issue is the fact that no modern AMD video cards including the Vega 64 come with Mac EFI firmware. As a result you cannot utilise any of the pre-boot options such as holding down the Option key to select a different boot drive, holding down Command-R to access the recovery partition, or using FileVault encryption. If this is not a problem then what will happen is that the screen will stay completely black until High Sierra has loaded enough of itself to load the driver for the video card, you will then see the normal login screen.
Note: AMD do not provide their own drivers unlike Nvidia, so you are completely reliant on Apple drivers. Sadly Apple drivers have historically delivered poor performance and often failed to support certain features of the video card - even such a basic thing as audio via the HDMI port. 😟 As indicated by the BareFeats test at least in one category i.e. OpenGL the Apple driver seems to be causing very poor performance but this hopefully will improve over time.