My Brother Wants To Have Sex With Me What Do I Do?

Ever since me and my brother were little we have always been close. We learned almost everything in our younger lives together. We were always together and were inseperable. We used to share the bed sometimes at night.
Im not sure when it started, but he began to get curious about certain areas. We both were very little and curious so we exposed ourselves to each other when we couldnt sleep. At that time we could never figure out why girls and boys were so different. When we became preteens who had just learned about sex, we understood. He learned to do what boys do and he tried to get me to help him but i was afraid of some other things happening so I didnt. He would also ask me strange questions only girls should know and I didnt tell him because I didnt think he needed to know.
Now that we are older (still teens) we went our seperate ways for a while. He never done anything during that time, but I did. I went with an 18 yearold boy during my freshman year in highschool. When my brother discovered this he began to ask more questions about sex. I was no longer afraid since I had done it already so I told him. Then he wanted more.... which is sex.
I have tried to tell my g-ma since im adopted by her because my parents hate my guts and wish I was never born. My birth mom is a slut so my brother is only half. DSS has been called so many times because of abuse from family members its not even funny anymore. I just now got a permanent home. But anyways... I dont have real parents to talk to so I feel vonerable.
Just this morning I woke up and he had his fingers you-know-where and was looking at me smiling. Then he licked his fingers. Is that considered rape? I dont know, its just so strange to me and I dont know what to do.

Yesterday I was going swimming. I was walking down to the pool and noticed him looking at me funny. I got in the pool and we started to just play laugh and stuff like we usually do. Then he was laying on a float and he started talking about it again. I tried to change subject but he wouldnt let me. Then he got off the float and cornered me in the pool. He kept insisting that we talk about it. I got too scared to move. I just froze. He started touching me again. I felt him move my bathingsuit over to the side and I felt something else... Later I tried to tell my aunt but she didnt believe me. She was one of the members I was taken away from and she also hates my guts.

Need to know others opinion on this (Im 15 and hes 14)
(Is it right or wrong) Im creeped out