iPhoneX iPhone iCloud Photo Library almost cannot be used after upgrade to 11.3.X

iPhone: iPhoneX 256G

IOS: 11.3.0 or 11.3.1 (both doesn't work after upgrade)

I purchased 2TB iCloud Plan as I want to put all my photo into iCloud for backup and easier to manage and share with my Mac Ecosystem. I have around 160K and 1.2TB photos put in the iCloud and my iPhoneX enabled iCloud Photo Library, of course, using the optimized mode. It works perfectly before 11.3.X. However, after upgraded to 11.3.0 few months ago, my iPhoneX's photo function is almost unusable. It can still take photo/videos as normal. However, when I take a photo, it can only appear in the iPhone's photo library after a minute. When I try to open any photos store in the photo library, that even take a few minutes ago, it takes 10-15sec at least to get the full version one. The live photo is almost ignorable as it simply waits 15-30sec. My iPhone has just used ~80G of 256G. There is so much space for it to cache at least for a few months of photos. If the behavior cannot fix, I think I have to give up the idea in iCloud Photo Library.

It seems iCloud Photo Library cannot support a large volume of photos at all after 11.3.0. Any experience or solution you can share?

BTW, I have tried whatever I can do. including erased everything and restore the iPhone completely twice, but the same behavior as above happens without fixing.