Cloudflare R2 for Backups

sams2526 · October 16, 2022, 5:06am

i guess so… tried running the rclone script again and get this:

The latest version of rclone rclone v1.59.2 is already installed.

im assuming i need to downgrade rclone?!

eris · October 16, 2022, 5:38am

No you need to download a beta / “main” branch build

sams2526 · October 16, 2022, 7:29am

okay got it now thanks

also anything worth placing in the .config/rclone/rclone.conf besides upload_cutoff = 5Gi ?

eris · October 16, 2022, 8:12am

I didn’t make any changes except entered my aws key details

sams2526 · October 17, 2022, 1:57pm

i did the beta rclone update earlier today, went into hestiacp and disabled, saved then enabled r2 backup.

First batch of backups im still getting ‘Can’t login to sftp host …’

eris · October 17, 2022, 2:03pm

Make sure that you have the current main branch working it should be merged right now

sams2526 · October 17, 2022, 2:06pm

okay installed main will try a backup soon

sams2526 · October 18, 2022, 2:13am

backups went fine, all good. whats the stance with the larger than 5gb tarballs? is there any chance that when the backup script is done it weighs in what protocol is used (in this case rclone) and chops up the archives into 5gb pieces to accommodate the backup limits?

sams2526 · November 5, 2022, 4:12pm

i noticed that files larger then 5gb are being uploaded to r2, not sure who to thank but its all good now. had one that was 9.2gb too

eris · November 5, 2022, 4:45pm

There seems to be a bug with older versions of Rclone it has been patched …

sams2526 · November 6, 2022, 6:57am

what about auto deleting anything older than the last 15 backups on r2? is that something we can do or would i need to chase it up with cf because i am manually deleting them atm?

eris · November 6, 2022, 7:52am

Doesn’t it delete when it passes the number backups in a package?

sams2526 · November 6, 2022, 10:37pm

i just checked and theres no max limit for the r2 side of it, i did a reinstall of main the other day too

eris · November 7, 2022, 7:00am

Screenshot 2022-11-07 at 07.58.13

This is the limit that should be taken in account

github.com

hestiacp/hestiacp/blob/4cf42a6288d198def7be40d6d089e368449faacd/func/backup.sh#L591-L598


      
          backup_list=$(rclone lsf $HOST:$BPATH | cut -d' ' -f1 |grep "^$user\.");
          backups_count=$(echo "$backup_list" |wc -l)
          backups_rm_number=$((backups_count - BACKUPS))
          if [ "$backups_count" -ge "$BACKUPS" ]; then
          for backup in $(echo "$backup_list" |head -n $backups_rm_number); do
              echo "Delete file: $backup"
              rclone deletefile $HOST:$BPATH/$backup
          done

And this takes care of it…

Or this code:

github.com

hestiacp/hestiacp/blob/4cf42a6288d198def7be40d6d089e368449faacd/func/backup.sh#L570-L584


      
          if [ -z "$BPATH" ]; then
              rclone copy -v $user.$backup_new_date.tar $HOST
              if [ "$?" -ne 0 ]; then
                  check_result "$E_CONNECT" "Unable to upload backup"
              fi
              
              backup_list=$(rclone lsf $HOST | cut -d' ' -f1 |grep "^$user\.");
              backups_count=$(echo "$backup_list" |wc -l)
              backups_rm_number=$((backups_count - BACKUPS))
              if [ "$backups_count" -ge "$BACKUPS" ]; then
              for backup in $(echo "$backup_list" |head -n $backups_rm_number); do
                  echo "Delete file: $backup"
                  rclone deletefile $HOST:/$backup
              done    
              fi

sams2526 · November 8, 2022, 12:51am

thanks for that but i am running main from a few days ago and do not see the option to choose it, that is sftp has the option to specify the limit in the admin panel but not rclone / r2 that is, setting the backup count.

eris · November 8, 2022, 5:22am

It can be changed at the package level…

sams2526 · November 8, 2022, 7:13am

oh yeah i forgot that lol mine was set to 300 when i was doing local cloud. set it correctly now. thanks, as this is a godsend being on cgnat now. kudos to you guys that dev’ed this and your help in this thread is appreciated!!

system · December 8, 2022, 7:14am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.