Starter clicks once and doesnt start..Sometimes!

So I've been having this problem for a while but it just comes and goes. I thought my starter was bad so I replaced that but it still happens.

Cliff notes:
1. Morning cold starts start 100% of the time
2. Usually happens when Im driving and then I shut off the car and I try starting it again (ex. pumping gas at the gas station)
3. Sometimes it takes 5 attempts, sometimes it takes 15 attempts.
4. Sometimes if i hold the key to start it clicks once, (still holding) it will start after like 5ish seconds
5. When the car decides to crank it starts 100% of the time without hesitation.
6. Battery is about 3 years old? i think.
7. I bypassed the clutch switch so it always thinks the clutch is pressed in to start (did this because I thought that was the problem)
8. I can hear the starter click only once when it doesn't start. No weird sounds
9. Does not happen all the time
10. Weather doesn't seem to have a correlation with the problem

Any input will be great. This problem has been happening for about a year now but it never prevents me from getting from point A to B.

Starter clicks once and doesnt start..Sometimes!