If it's a Production environment, try to identify which process is consuming all your memory. Normally it's a heavy Timer / batch process, or a BPT transaction, with a huge volume of data being loaded into a variable or list (using memory). Talk to your infra-structure team and see if they can help monitoring the resources (CPU and memory) consumption of your server.
Now, if it's a development environment with Java stack, you can face issues related to memory consumption. Especially if you have a big team. Everytime someone publishes a new version of an espace through Service Studio, JBoss keeps a bit of memory busy and doesn't free it up. And it keeps growing. Best workaround is restarting the dev server every now and then.
Take a look at the Analytics log in Service Center to see the heaviest processes, pic time consuming, etc. They all can help you with insights in order to identify what is the root cause of your problem.