Without having Duplicati, it's hard to tell, but there are some common gotchas/rules when running shell scripts within AppleScript or Automator.
The main one (and my first guess as to what's going on here) is that AppleScript/Automator will wait for the shell command to terminate before continuing. That's because it's designed to capture the output of the command so you can use it in later script/workflow actions.
In this case, if you're launching a server/background process, the shell command never terminates (until the server process shuts down), so the script is left hanging, and will eventually time out (it assume something went wrong).
If that's the case here, the solution is simple - you need to append some construct to the end of the shell command so that it returns control to the calling app/script. That looks like:
/Applications/Duplicati.app/Contents/MacOS/duplicati-server >/dev/null 2>&1 &
That cryptic thing at the end does three things.
First, the >/dev/null redirects/suppresses the command's stdout, which is the text you'd normally see the command output in terminal.app, and is what AppleScript/Automator are waiting for.
The 2>&1 adds the same redirection to stderr, which is the app's error reporting mechanism (where error messages would get output). AppleScript also waits for this, so that it can report on any error message the shell command returns.
Finally, & pushes the command to run in the background, returning control to the user. Typically in Terminal.app this gives you your prompt back, while the app continues to run in the background, but in Automator's case, it hands control back to Automator for subsequent Actions.
The three things together have the effect of launching the process in the background, returning control to Automator.
The downside of this is that you'll lose sight of any output from the shell command, but you likely weren't relying on that anyway, since server processes typically have other output logging mechanisms.
