how can I open a playlist file (m3u8) on itunes on an iphone

You need to import the .m3u8 file into iTunes which, providing the tracks are at the specified locations, will add them to the iTunes library and create a playlist of them in iTunes in the specified order. That playlist can then be synced to your device.

tt2

No, you cannot manage this directly on the iPhone. You need to import the playlist file into iTunes on a computer. You can drag & drop or use File > Library > Import Playlist... That makes a playlist object in the iTunes library. iTunes is no longer interested in the .m3u/.m3u8 file after the import. It is the playlist object that can be synced or dragged to the device in iTunes. The import will only be successful if the files referred to in the list are at the paths given in the list. In other words if you only have the .m3u8 file then it is like having the index to a book without the book. If you have the files, but they aren't where they're indicated as being in the index then the imported list in iTunes will be empty. All you get is the name of the playlist. M3U playlists can employ both absolute and relative naming schemes. iTunes generates absolute paths, but relative paths may be more useful if you're trying to copy a playlist and its media files from one place to another.

tt2