I have an equation ∂^2U/∂X^2 +∂^2U/∂Y^2 = K^2U with the boundary conditions U = 0 on the boundary of the equilateral triangle. How to solve the equation

If your LHS should be -k^2U, this is the Helmholtz equation. It has analytic solution by separation of variables for certain geometries and BC, and maybe yours falls into these. See http://mathworld.wolfram.com/Helmhol...lEquation.html.

If this is not the case, please clarify - are you looking for analytic solution, or are you after a numerical one?

I thought the solution is simply U=0. U=0 satisfies the governing equation and also the boundary conditions at all points on the sides if the triangle. Could not be this simple?

I would assume this is an eigenvalue problem (modulo sign confusion) so you must find K and corresponding U, |U|=1 which solve this equation. That is, find the first few eigenmodes of the Laplacian on this geometry. For this special case, you can compute the modes analytically. For very accurate numerical methods for more general planar regions, see the references in the paper below. Of course, you can always just discretize the problem and hand it to a solver package.

http://eprints.ma.man.ac.uk/594/01/c...ep2006_367.pdf