How do you loose someone that was not truly yours in the first place?

I met this really nice, sweet, kind, caring, funny, and cute guy a while back. I knew him for six months and then he moved to another city. It wasn't love at first sight, in fact i didn't know what to make of him when i first laid my eyes on him. For a few months I just watched him and every time he looked my way i smiled at him and shyly looked away. As time went on I began to like him more and more. At first I denied my feelings for him, as I shamelessly flirted with him. The thing is this guy was and is stilll off limits, I can't say why, but let's just say that all i could do was flirt with him and hope that he thought i was cute too. When a guy is interested in me I can usually tell and it was no different with this guy. I knew that as I opened up to him, he liked me and grew very fond of me. I caught him start to watch me first and i would look back and we would smile at eachother. In the span of time that we got to know and love eachother, we made lots and lots of sweet memories together (nothing sexual). Memories that I will always cherish and no one can take from me. When he moved, his face, his laugh, his smile, his love stayed behind and comforted me during the times I feel lonely and miss him the most. He was never truly mine because we never vocalled our feelings for eachother, but his actions said otherwise, so maybe that's why i feel like i got a part of him if not the whole thing. I feel like I lost someone that I was, am, in love with, but the question is how can it hurt so bad and how can i possibly loose someone that that was not really mine to begin with? I miss him every second, hour, and day!