A login loop and non-working mail/webmail is very often caused by the system running out of disk space. When the filesystem is full, mail and auth services can no longer write required files and will fail.
Please first check:
df -h
Also note: HestiaCP is intended for admins or power users.
The main risk is not the instance size itself, but running a publicly exposed server without sufficient system administration knowledge. Installing HestiaCP does not remove the responsibility to understand Linux, mail, DNS and security basics:
https://forum.hestiacp.com/t/read-this-first-important-notes-for-running-hestiacp-safely/383
If disk space is not the issue, please provide logs and system details.