How would you respond to a candidate who you rejected for a position, but then follows up with an email asking you what they could have done better in the interview?

I think it depends on the interviewee. For instance, if it was for a staff accountant's position for a fairly new professional, your advice could help them in a future interview. If it is for a senior accountant's position, the individual should have a better idea of what was done incorrectly. It could be a great opportunity to help build on a person's skills, helping them land their dream job later. Some people don't interview well, regardless of how well they prepare for it.

I say go for it! Be honest and professional with your response, even if you have to give advice they may not want to hear.

Please be careful with this type of advice. It runs the same risk as giving out references for past employees. Seemingly innoculous comments could be interpreted as some violation of EEOC rules and complaints being lodged. I understand the desire to want to help the interviewee for future interviews, but I personally would either ignore the request. If you feel compelled to give some response, I would do it via phone and not in email so there is no written record of the repsonse.

Verbal confidential feedback based on objective criteria (e.g. experience qualifications) can be given without risk. However, if the reason for not hiring is "chemistry" or cultural "fit", it is difficult to provide feedback in those terms. Talk to HR, confirm with them the "procedure" for handling such requests and proceed according.

It is very generous to give this type of advice; it is great to get, and good to model being honest. Per Ted, it is also risky.
1) Keep it factual; did they not have Hyperion experience? If that (or similar) is the problem), perhaps you can guide them to training.
2) Keep it verbal. Never ever give written feedback.