Who & Who?

There's a lot of time when my friends ask me this type of question; "Do you remember her/him?", "Do you know him/her?". This type of question is what I hate about. I do remember people, but not all of them. It's not like that I do that on purpose, it just become like that. When the time pass by, and I rarely sees that person, I am gonna start to forget their face or their name. But for a person that I always sees, I'll maybe remember their face, their name or either one. My friends always ask me; "How can you forget about them?", "They were our friends at school, how could you." This is what they always says when I respond to them "I don't remember." or "Maybe I remember." From my view, I don't want to remember peoples because they will eventually forget about me. And that why I always easy to forget about people. Is this seems like cruel? I don't know, I always think that maybe even if i remember them, they maybe did not know me or even know my name.