Apple's Admin Tools Compatibility Support Article:
suggests you can use 10.6 Server Admin Tools to administer most aspects of a 10.5 Server. However I would not recommend you do so. Simply because it can cause problems along the lines you're seeing. If it was me I would always use the version of the Server Tools that match as closely as possible (as the support article suggests) the version of the Server you're administering. Remote into the Server and try the option that way. If the problem persists you may have come across a deeper underlying problem? If the problem goes away you've found a solution that works.