Unless the piece that broke off is hollow, I don't think a drywall screw will work. It won't dig into the metal of the jack. You'd need a self-tapping screw to be able to do that. I don't think they make self-tapping screws that small, but you could check at your local Home Depot. Either way, if the broken piece is loose enough to turn inside the port, it won't work anyway.
I think a really small tweezers might be the best tool to try. You could also try this, if all else fails and you feel you have nothing to lose:
Take a large paperclip, straighten it, and put a small glob of Superglue or some other STRONG adhesive on one end. Carefully insert it into the port until you make contact with the broken piece of the jack. Hold it, or support it somehow, until the glue/adhesive sets. Then, gently and as straight as possible, pull the paperclip out of the port. The broken piece should come out. I've done this, successfully, with broken pieces of metal in very small holes.
The key is not to use too much glue that the paperclip gets glued to the inner sleeve of the port. But you need enough glue to allow it to adhere to the jack. The worst case scenario: You ruin the port. But, if you can't get the broken piece out any other way, the port is essentially ruined anyway. So, like I said, if you have nothing to lose ...
Good luck!