The sound represented by `r' in rōmaji is not the same sound as English `r' or English `l'. It is between the two. Try saying `r': the tip of your tongue does not touch the top of your mouth. When you say an English [r], your tongue approaches the alveolar ridge but doesn't touch. The tip of your tongue is slightly grooved. The edges of your tongue seal off the sides of your mouth. The air from your lungs passes over the top of your tongue on its way out. That is not the Japanese "r".
English L tongue position
Try saying `l': the tip of your tongue touches the top of your mouth. Feel behind the back of your upper teeth with the tip of your tongue. There's a ridge of hard gum there. That's called the alveolar ridge. When you say an English [l], the tip of your tongue is actually touching the alveolar ridge, and the air passes around the sides of your tongue on its way out.
Japanese R tongue position
The Japanese sound represented by rōmaji `r' is somewhere between these two sounds. It is called an alveolar tap. The consonant of ra ri ru re ro is sort of like an and an [l at the same time. The tip of your tongue TAPS against the alveolar ridge (hence the name "alveolar tap"). In the fraction of a second just before contact, it's like an In the fraction of a second during contact it's like an [l. That's why it's so hard for Japanese speakers to distinguish between those two English sounds.