The mail you get is typically from the stylist, reporters, or 961. (Fanmail also comes, but you don't need to worry about them.) If you reply to the stylist or reporter, you're agreeing to meet them for an appointment, and the appointments have different times. They could be for the same week, next week, or 2 weeks later, in the morning or in the evening. If you reply to a 961 e-mail, you're agreeing to compete at a rival audition. Your image goes down if you fail to meet up with the stylist/reporter at the time agreed or if you skip a rival audition.
Your image can also be lowered if you lose an audition, haven't done an audition in a long time, or if you encounter a bad reporter (a gossip) and they run a bad article on your idol. Gossips usually appear after you lose an audition or sometimes if you keep skipping out on appointments with a reporter.
It takes some time, but eventually you'll be able to differentiate between reporter, stylist, and 961 e-mails, and choose the ones you want to reply to based on your plans for that week. Going to an appointment with a reporter can help raise your image level temporarily, so it's best to do one of those before an audition.
Hope this is of some help...I haven't played SP in awhile. I should get back to playing it again...I think I was in the middle of producing Iori in Wandering Star and Makoto in Perfect Sun. And I still need to get Missing Moon! XD