Slow backup recovery. Migrating from VestaCP

[trogvar]

Migrating from VestaCP and got some strange behavior of recovery.
I have several servers with different sets of data from 1GB to 1 TB.
So successfully migrated small servers but stuck migrating big servers.
It goes very slow! Extremely slow.
I’ve started recovery 3 days ago and it recovered only web (1GB) and couple domains in DNS and it goes like 1 domain in 2 hours:
2021-08-18 20:26:01 domain
2021-08-18 22:20:45 domain
2021-08-19 00:15:53 domain
2021-08-19 02:11:02 domain
2021-08-19 04:05:51 domain
2021-08-19 06:00:21 domain
2021-08-19 07:55:44 domain
2021-08-19 09:50:10 domain
… and so on.

And there about 1TB data of mail ahead…

Server is the same as source - 4CPU+16GB RAM with SSD.
New server disk system is RAID5 with SSD. dd disk write test gives 650MB/s.
While recovering with v-restore-user gzip using 70-90% of one core from 4.
When stopped and manually ungzipped backup it took about 2 hours but not days.

What can be done?
What should I check?

[schiwe]

Are you recovering over the network? You better copy the backup files to the destination server and then do the recovery.

[trogvar]

I’m doing recovery from local backup downloaded from source server to new server.

[schiwe]

What you see if you run top command?

[trogvar]

gzip using 90% of CPU

[schiwe]

There you go. 4 CPUs is probably 4 vCores and you may have a large number of files in the backup. It’s definitely not HestiaCP issue…

[trogvar]

Tasks: 137 total, 3 running, 134 sleeping, 0 stopped, 0 zombie
%Cpu0 : 72.5 us, 23.5 sy, 0.0 ni, 3.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.0 us, 38.3 sy, 0.0 ni, 61.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 2.0 us, 4.0 sy, 0.0 ni, 94.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.0 us, 2.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 16978.5 total, 150.4 free, 1351.8 used, 10476.3 buff/cache
MiB Swap: 17364.0 total, 17093.6 free, 270.3 used. 10293.6 avail Mem

It uses only 1 core.

[trogvar]

Anyway I ungziped backup on a new server manually and it took about 2 hours approx. But not 3 days.

[trogvar]

Had to modify v-restore-user script to take data for input from manually untared archive.
Script from the box is too slow to manage big files.

p.s. Also v-extract-fs-archive should be modified. There is a path check that doesn’t allow to recover mail account from users path.

[jlguerrero]

@trogvar this could be an interesting enhancement.
@eris do you want trogvar to share the changes to the script with you?

[trogvar]

I can upload it, maybe it will help someone else.

[eris]

Please submit a pull request:
https://github.com/hestiacp/hestiacp/pulls