Just to clarify, Modals do not open screens inside screens. A Modal is just a kind of container that acts like if it was a Popup, but its content is in the same page where it opens.
Modal (only for traditional web applications): https://success.outsystems.com/Documentation/11/Developing_an_Application/Design_UI/Patterns/Using_Web_Patterns/Content/Modal
A Popup is able to open a screen inside a screen (React Web and Mobile + Traditional Web): https://success.outsystems.com/Documentation/11/Developing_an_Application/Design_UI/Inputs/Create_and_Use_a_Popup
Hope this helps.
Cheers.