Hi Darsh,
The difference between the first click and the others is, almost for sure, due the browser having the resources/data it's looking for in cache now.
This means that after the first time you click, images and so on are already in the client machine, not being necessary to download it again, reducing the time to render the page. Also, if you are caching aggregate and sql, you may be receiving data from server cache, without having to execute them again.
The only way to speed up the rendering in the first access to the page is to reduce the amount of data you are asking from the server, like images size and so on.
Depending on the cause of the slowness, techniques like laze load (for data) can help improve user experience, if there's no possibility of reducing the size of the request answer.
Cheers
Eduardo Jauch