I am studying the level set method to solve Rayleigh-Taylor or Kelvin-Helmholtz instability. where can I find useful materials? TVD scheme is the most common numerical scheme in compressible gas dynamics, but in this problem, it cannot give satisfactory solution, which scheme is better? ENO or something else?

The recent textbook by Sethian (1999) Level set methods and Fast Marching Methods, Cambridge University Press, is a good starting point.

Many level set researchers utilise ENO schemes to simulate interfacial flow problems like the ones mentioned above, e.g. Sussman et al (1994), 114, J. Comput. Phys. Another alternative is unsplit Godunov type methods which were adopted in a recent publication by Sussman et al. (1999), 148, J. Comput. Phys. These references should get you going.

Note that mass conservation is still an outstanding problem for the level set method (even with re-initialization schemes and global constraints specifically developed to minimise mass conservation errors). Make sure you use fine grids to minimise these errors.

Good luck