Apple TV 4K won't connect via Ethernet, only WiFi

I recently bought the new 4K Apple TV and technically there's nothing wrong with it, works fine via WiFi, but I have the option to connect it to the router through a network cable and wanted to do it to avoid having too many devices connected to the same router wirelessly.

Here's the problem, though - it doesn't work which is very surprising because if anything, a wired connection should always be the most reliable solution IMO, yet in case of my Apple TV, it's actually the one that causes the most problems.

To give you more details here's what happens:

• At first Apple TV doesn't even notice that it's connected via Ethernet, it can take minutes before it realizes I plugged it in and sometimes this won't happen at all and I need to unplug and plug it in again.
• When it connects, it correctly sees the network, gets an IP address, sees the right DNSs so it seems that it should be fine, but while it can connect to the router, it won't connect to the internet so Netflix, YouTube, App Store, anything that requires internet, won't work/connect.
• The LED on my router is orange for the port where the Apple TV is connected while it's green for every other device. Switching ports won't help, I went through all 4, two of which are always used by my PC and a console so they definitely do work... but not for the Apple TV.
• My modem/router is a Cisco EPC3928S which I've got from my ISP, but here's the weirdest part - I also have internet from a second ISP (my phone company), for which I'm using my own router which is a TP-Link TD-W8951ND modem/router and the same thing happens with it - it will connect to the router, but not to the internet.

That said, my question is - does anyone have any ideas what could cause this? I've tried everything I could think of:

• I disabled MAC address filtering I normally use so that every device could connect to the internet with no restrictions.
• I assigned the Apple TV an IP address so it won't get a different one dynamically and then used DMZ mode for that IP which basically exposes it to the internet (it's like forwarding for all ports AFAIK).
• As mentioned, I used another router - didn't help.
• The cable has to be fine as it was tested by the guy who made it in front of me and as I've said - the Apple TV can successfully connect to the router, just not to the internet.
• Both routers are in DHCP mode and can successfully assign the Apple TV an IP address.

Did anyone have an issue like this? Again, technically the device can connect to both routers via WiFi and works fine like this, it just won't connect to the internet via Ethernet and I have no clue why 😟