Automator has no control logic. Data flows into and out of each action. If you use a Run AppleScript or Run Shell Script action, you can use language-specific control logic to determine the flow of processing within that particular action, and what data flows out of it.
Button response is the purview of AppleScript where you can get the name of the button clicked in the dialog and form your logic around that result. Here is some sample code for capturing the name of the button and testing for it in a control block:
set target_kind to button returned of (display dialog "Select icon target: " buttons {"Cancel", "Folder", "File"} default button "File" cancel button "Cancel")
try
if target_kind contains "File" then
set targetFile to POSIX path of (choose file with prompt target_msg1 default location target_dl without invisibles)
else if target_kind contains "Folder" then
set targetFile to POSIX path of (choose folder with prompt target_msg2 default location target_dl)
end if
on error errmsg number errnbr
-- because the cancel button was selected
my error_handler(errnbr, errmsg)
return
end try
set istatus to my set_icon(theImage, targetFile) as boolean
It may be useful to have access to the online AppleScript Language Guide. Display dialogs are discussed in the Command References section.