Hi Praveen,
You can monitor GC via wily introscope if you have one setup in your landscape. Else, you have to manually analyze your system logs. You can start at dev_server0. Also, try to capture thread dumps when your system is getting high on memory usage.
Going back from your 1st screenshot, your SAP memory usage is "green" while the red one is under OS. If your system is running on Linux, then this could explain the high consumption:
Hope this helps.