DNS Cluster Sync incomplete

xeruf · November 6, 2023, 10:01pm

I have a main server with about 200 domains, most of them imported from a previous instance. There is a slave with everything setup according to the docs (DNS clusters and DNSSEC | Hestia Control Panel), checked thrice and rebooted multiple times, but the slave only receives about 20 domains, and querying it for other domains it refuses.
Both servers have a public IPv4.

Example:

❯ drill ftt.gmbh @ns1.iridion.it
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 23232
;; flags: qr aa rd ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 
;; QUESTION SECTION:
;; ftt.gmbh.	IN	A

;; ANSWER SECTION:
ftt.gmbh.	300	IN	A	89.58.52.124

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 28 msec
;; SERVER: 89.58.52.124
;; WHEN: Mon Nov  6 22:59:31 2023
;; MSG SIZE  rcvd: 42

❯ drill ftt.gmbh @ns2.iridion.it
;; ->>HEADER<<- opcode: QUERY, rcode: REFUSED, id: 37833
;; flags: qr rd ; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 
;; QUESTION SECTION:
;; ftt.gmbh.	IN	A

;; ANSWER SECTION:

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 31 msec
;; SERVER: 152.89.105.120
;; WHEN: Mon Nov  6 22:59:33 2023
;; MSG SIZE  rcvd: 26

xeruf · November 6, 2023, 10:45pm

Found a workaround by executing v-add-remote-dns-domain for each domain of each user, but that does not feel right

eris · November 6, 2023, 11:00pm

v-sync-dns-cluster should do the trick…

But it does more or less the same …

xeruf · November 6, 2023, 11:02pm

no it did not, that is the odd thing, as said I executed it multiple times on the master node

eris · November 6, 2023, 11:06pm

github.com

hestiacp/hestiacp/blob/e0008086578a92e99942201ea3058dbd0314879c/bin/v-sync-dns-cluster#L62


      
          	clear_dns_cluster_settings
          
          	# Parsing host values
          	parse_object_kv_list "$cluster"
          
          	# Wiping remote domains
          	cluster_cmd v-delete-dns-domains-src $DNS_USER $HOSTNAME no
          	check_result $? "$HOST connection failed" $E_CONNECT
          
          	# Syncing user domains
          	user_list=$(ls -d $HESTIA/data/users/*/ | sed "s#$HESTIA/data/users/##" | sed s"/.$//" | grep -v "dns-cluster")
          	for user in $user_list; do
          		USER_DATA="$HESTIA/data/users/$user"
          		ROLE=$(get_user_value '$ROLE')
          		if [ "$ROLE" != "dns-cluster" ]; then
          			for str in $(cat $HESTIA/data/users/$user/dns.conf); do
          				unset $SLAVE
          				parse_object_kv_list "$str"
          				if [ "$SLAVE" != "yes" ]; then
          					if [ "$DNS_CLUSTER_SYSTEM" != "hestia-zone" ]; then
          						# Syncing domain index

It is designed to do that and loop trough all the domains…

And then do send the records to the new server:

github.com

hestiacp/hestiacp/blob/e0008086578a92e99942201ea3058dbd0314879c/bin/v-sync-dns-cluster#L85C1-L102


      
          					if [ "$DNS_CLUSTER_SYSTEM" = "hestia-zone" ]; then
          						str=$(echo "$str" | sed "s/SLAVE='no'/SLAVE='yes'/g")
          						str=$(echo "$str" | sed "s/SLAVE=''/SLAVE='yes'/g")
          
          						ip=$(ip addr | grep 'inet ' | grep global | head -n1 | awk '{print $2}' | cut -f1 -d/)
          						source_conf $HESTIA/data/ips/$ip
          						if [ -z $NAT ]; then
          							str=$(echo "$str" | sed "s/MASTER=''/MASTER='$ip'/g")
          						else
          							str=$(echo "$str" | sed "s/MASTER=''/MASTER='$NAT'/g")
          						fi
          
          						# Syncing domain data
          						cluster_cmd v-insert-dns-domain $DNS_USER "$str" $HOSTNAME $flush 'no'
          						check_result $? "$HOST connection failed" "$E_CONNECT"
          
          						cluster_cmd v-rebuild-dns-domain "$DNS_USER" "$DOMAIN"
          						rndc notify $DOMAIN > /dev/null 2>&1

eris · November 6, 2023, 11:07pm

And

github.com

hestiacp/hestiacp/blob/main/bin/v-add-remote-dns-domain

#!/bin/bash
# info: add remote dns domain
# options: USER DOMAIN [FLUSH]
#
# example: v-add-remote-dns-domain admin mydomain.tld yes
#
# This function synchronise dns domain with the remote server.

#----------------------------------------------------------#
#                Variables & Functions                     #
#----------------------------------------------------------#

# Argument definition
user=$1
domain=$2
flush=$3

# Includes
# shellcheck source=/etc/hestiacp/hestia.conf
source /etc/hestiacp/hestia.conf

This file has been truncated. show original

Does the same …

xeruf · December 14, 2023, 6:40pm

This problem is unfortunately persisting.
v-sync-dns-cluster runs for a long time, but domains are missing.

If I call v-add-remote-dns-domain, I get a connection error even though the connection works just fine:

$ v-add-remote-dns-domain USER DOMAIN
Error: 152.89.105.XXX connection failed

xeruf · August 25, 2025, 10:10am

@dpin I don’t know what happened but it seems fine again maybe an update?

dpin · August 25, 2025, 6:05pm

I have had the same issue, at least 2 times a week. Everything works again with a manual sync

v-sync-dns-cluster

xeruf · August 28, 2025, 4:24pm

now it was out of sync again, I ran v-sync-dns-cluster and it said Error: connection failed but now things are back in sync, curious

xeruf · August 29, 2025, 10:52am

now I am getting servfails for all new domains - one thing to keep in mind is to regularly update your ns2 as well

Raphael · August 29, 2025, 11:00am

probaly have a look at bind entries in syslog, sync works properly over here, there should be no need to run v-sync-dns-cluster.

xeruf · August 29, 2025, 1:23pm

now I updated my ns2 after a while and ended up in a very curious situation where ns2 refuses any query while ns1 still returns a servfail for new domains, even after dns-domain-rebuilds and restarts

EDIT: the ns2 issue was a temporary thing, the servfail for the new domain persists and there are no relevant logs I can find

xeruf · August 29, 2025, 4:54pm

SERVFAIL fixed, see [Feature] SRV records in the DNS zone editor · Issue #2645 · hestiacp/hestiacp · GitHub