What we would like to do is use an Asus wireless RT-N66R wireless router to extend the wireless signal to a small shop about 100 feet from where the main computer and Apple Airport Express is located in the house. Is this possible?
Apple uses proprietary settings to "extend a wireless network" that are designed to only allow another Apple wireless router to wirelessly extend the network......so, it is extremely unlikely that any third party product would be compatible for this type of service.
There are no settings or adjustments on the AirPort Extreme to allow another device to extend the network, so it is all up to the Asus device to try to connect and extend the signal. If you spent time with an Asus support person and have been unable to make this work, I think you already have your answer here.
How good of a range would another airport express have at extending the range of the wifi signal?
The problem here is that an extending device is normally located about half way between the main router....an AirPort Extreme, in your case, and the area that needs more wireless coverage.....the shop.
The Express can only extend the quality of signal that it receives, so it would make little sense to install the Express in the shop where the signal is already poor....it will extend the same poor signal.
If you install the Express in your home on an outside wall in the direction of the shop, it may......or may not....work. This is one of those situations where you cannot know if something will work until you try it out.
If wireless won't work, you have little choice but to run an Ethernet cable from the home out to the shop, as Kappy suggests.