LG OLED TV

On the iMac - Playing videos on the TV from my Mac by using the Apple TV app on the mac - I have a message : "This video format cannot be played on this TV".

The format of the video files must be exactly right: compatible with your Mac’s TV app, compatible with AirPlay, and compatible with the LG media player. Your best best is 1080p (or lower) H.264/H.265 video with AAC audio in an MP4/M4V/MOV file. There are some further constraints like MPEG-4 profiles and levels settings. Invisor Lite media inspector tool - Invisorapp.com

There are two different flavors of AirPlay, which are distinctly different for the sending device. (1) Basic AirPlay send the video as-is through in-app controls, and (2) AirPlay Screen Mirroring captures the screen at 30 fps and re-encodes as a 1080p/stereo streaming movie. Prefer basic AirPlay when possible (TV app, QuickTime Player, Safari web video) for best quality.

On the TV: from the Apple TV app I cannot find my iMac Music and Home video library. Despite that I shared everything from the settings of the my Mac.

In the Apple TV app on your smart TV, you can get online streaming video content as linked to your account, not content from local libraries. For that, you would need the Apple TV box and Home Sharing, or else keep using AirPlay. It works as intended.

Apple TV app vs. Apple TV box - Apple Community

You may be experiencing an hdcp error

Since you are not using an Apple TV, the hdcp error may occur because you are using a paid subscription, like the Apple TV app.