Xbox Series X controller not working properly in Big Sur 11.3

The controller connects but is unusable (behaviour is erratic). Updated to Big Sur 11.3 ((https://support.apple.com/en-in/HT211896#macos113)) since the update mentions Support for "Xbox Series X|S Wireless Controller" but still unable to use it properly.

My old Xbox "one S" controller however works fine (controller is broken so upgraded to "series X controller").

System Report :

  Xbox Wireless Controller 441622CA3E61:

  Address: 44-16-22-CA-3E-61

  Random Address: No

  Services:

  Paired: Yes

  Configured: Yes

  Connected: Yes

  Manufacturer: Unknown (0x8, 0x2209)

  Bluetooth Core Spec: 4.2

  Firmware Version: 5​.​5.2641.0

  Vendor ID: 0x045E

  Product ID: 0x0B13

  Class of Device: Low Energy

  AFH: On

  AFH Map: c0ffff1f

  RSSI: -52

  Role: Central

  Connection Mode: Active Mode

  Interval: 0 ms