I was always scared of the water after someone held me under during swimming lessons at school. I got to the age of 38, and realised that I had a lot more reason to be scared of the water if I couldn't swim than if I could. I went for adult swimming lessons, and within 3 weeks, I could swim 25 metres unaided. Now I swim at every opportunity, and I can swim for up to an hour at a time. I could cry when I think of all those wasted years when I could have been enjoying myself swimming.
I'm telling you all this just to underline that it's never too late to learn. Swimming teachers who teach adults to swim understand the psychological barriers you face, and they will help you to overcome them, gently but firmly.