is Studio Display 17" LCD monitor screen compatible with 1998 MAC G3 266mhz?

It depends on the Sudio display model. Later Studio displays requred Apple's failed-concept Apple Display Connection (ADC), something a 1998 G3 will not have unless you have instaled a PCI video card that supports ADC. Those are hard to find and still may not work given the limited power budget of the Beige G3 PCI banks.

As far as I can tell, the following early Studio Displays can work but may require an adaptor (those models indicated with the symbol †):

M6204†

M6221†

M4551 Rev A

M4551 Rev B†

You also need an inexpensive VGA-to-Mac adaptor like this:

http://www.amazon.com/C2G-Cables-02902-Female-Adapter/dp/B0002J1JAE/ref=sr_1_1?s =electronics&ie=UTF8&qid=1438440149&sr=1…

an example of a PCI videocard that will add new options for monitors with DVI and VGA connections, these older discontinued cards are getting harder to find

http://www.ebay.com/itm/Apple-Mac-G4-G5-ATI-Radeon-7000-PCI-64MB-Video-Card-DVI- VGA-S-Video-TV-Out-OS9-/200710223413?has…