It can be hard to know what is happening with your lime tree without seeing it. While lemon trees grow pretty fast, lime trees are not easy to fruit – especially if you live in a colder area.
I would recommend checking the root stock of your tree. When you buy a lime tree it will often be grafted to another tree. Just look down the bottom and if the leaves at the bottom are totally different from the leaves on the tree it means your grafted tree has taken over from your main tree. In this case I would recommend planting a new tree.
However, it is most likely your lime tree just hasn’t had time to fruit yet. I would give it some time and in the mean time feed it some pot ash which you can buy from your local garden centre. This will encourage fruiting. Or you could try feeding it nitrophoska blue, a balanced fertiliser with everything it needs to encourage growth.
Another trick to remember with planting citrus trees is not to plant them too close to a lemon tree. Bees have a tendency to cross pollinate from a lemon tree and this will mean you end up with a lot of seeds in the fruit of your lime or other citrus tree fruit.