What kind of camera/housing do you have now? Does it allow strobes?
Getting a strobe on your camera can make a bigger diff than the camera itself. Here's a cheaper camera which allows strobes (I don't know much about that cam though): viewtopic.php?f=45&t=26530
Another big element is the lens. For nudibranchs, getting a good macro lens, such as a Nikon 60mm is ideal.
You may want to look for an old used Nikon d70 or d80 with 60mm lens in ikelite housing with a single strobe, you can do quite well with that for fairly cheap (though it may slightly exceed 500 due to the lens cost). It would take you much much further than a newer 64 megapixel camera with poor lighting and lens
Another factor, probably the most important, is how you use your camera and lights. Have you gotten some good tips for usage? Come down to ThThTh or maybe the club dive and people can give tips!