I think you will feel better and better as times goes. We have no excuse for not spending time on English. Just tell you a cruel fact: One of my former classmates could not speak clear Mandarin. He learnt Russian at college. However, he is now speaking fluent English. He took English learning is an issue to be or not to be.

Take myself as an example. I could not remember how many times I have looked up the dictionary for the meaning of words such as "commit", "contigeous" ...

If we understand that mastering English is part of our life, it won't be so painful for us to spend time on it.

Good luck