Lets Encrypt validation fails - Invalid response

xeruf · January 4, 2023, 2:50am

I have the same error as discussed many times, e.g. in forum.hestiacp com/t/unable-to-get-ssl-for-hestia-cp/8248/10

Situation

I tried everything I found to no avail, the issue persisted throughout the last weeks, through restarts of the VM and nginx, with no apparent errors.
It happens for all new domains and some current ones, but even within a single user there are domains that error out and some that work.
There is no Cloudflare proxying.
The issue occurs both from CLI and Web UI, e.g.:

v-add-letsencrypt-domain owner DOMAIN
Error: Let's Encrypt validation status 400 (DOMAIN). Details: 403:"HESTIA_IP: Invalid response from http://DOMAIN/.well-known/acme-challenge/QsqBVN6pyV9QeFohNmJmzKRyI4z3dYW_XbPhRivb1UI: 404"

Debugging

I plowed through:

github com/hestiacp/hestiacp/issues/1509
forum.hestiacp com/t/lets-encrypt-ssl-error-error-lets-encrypt-validation-status-400-domain-com-details-403-my-ip-invalid-response/7651>

According to https://docs.hestiacp.com/admin_docs/web/ssl_certificates.html#error-let-s-encrypt-validation-status-400 I checked /var/log/hestia/LE-owner-DOMAIN.log which contained

details: Unable to update challenge :: authorization must be pending

letsdebug reports no problems

According to https://docs.hestiacp.com/admin_docs/settings.html I enabled DEBUG_MODE and checked the logs, but did not get anything out of theat, see below.

Logs

This is an extract from lnav /var/log/hestia, maybe something unusual here:

I am a little bit confused about all the failed added records and domains, because we did not add any except the one with letsencrypt.

eris · January 4, 2023, 9:22am

Check /var/log/hestia/LE-user-domain.log

It should contain more info

xeruf · January 10, 2023, 10:42am

I did check it, as mentioned above. Here is the full one with private info redacted:

=============================
Date Time: 2022-12-20 11:48:49
WEB_SYSTEM: apache2
PROXY_SYSTEM: nginx
user: owner
domain: DOMAIN


- aliases: 
- proto: http-01
- wildcard: 


==[Step 1]==
- status: 200
- nonce: F977DKtq9LE411CKahC9RbBzC9yPOfODmzc_ZQiivLpS21o
- answer: HTTP/2 200 
server: nginx
date: Tue, 20 Dec 2022 10:48:50 GMT
content-type: application/json
content-length: 659
cache-control: public, max-age=0, no-cache
replay-nonce: F977DKtq9LE411CKahC9RbBzC9yPOfODmzc_ZQiivLpS21o
x-frame-options: DENY
strict-transport-security: max-age=604800



==[API call]==
exit status: 0


==[Step 2]==
- status: 201
- nonce: A5FEsTHUVC3yKB2fYuFmZAlJYnGawazYEuJbPtbn2ocsmv4
- authz: https://acme-v02.api.letsencrypt.org/acme/authz-v3/188507753607
- finalize: https://acme-v02.api.letsencrypt.org/acme/finalize/515162167/153847387397
- payload: {"identifiers":[{"type":"dns","value":"DOMAIN"}]}
- answer: HTTP/2 201 
server: nginx
date: Tue, 20 Dec 2022 10:48:51 GMT
content-type: application/json
content-length: 334
boulder-requester: 515162167
cache-control: public, max-age=0, no-cache
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
location: https://acme-v02.api.letsencrypt.org/acme/order/515162167/153847387397
replay-nonce: A5FEsTHUVC3yKB2fYuFmZAlJYnGawazYEuJbPtbn2ocsmv4
x-frame-options: DENY
strict-transport-security: max-age=604800

{
  "status": "pending",
  "expires": "2022-12-27T10:48:51Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "DOMAIN"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz-v3/188507753607"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/515162167/153847387
397"
}


==[API call]==
exit status: 0


==[Step 3]==
- status: 200
- nonce: 371CmL9LQe1OR5E2m1mOCaJPygED74-6xYCOQ_KETgV78mE
- url: https://acme-v02.api.letsencrypt.org/acme/chall-v3/188507753607/upM-qg
- token: REDACTED1NxYMOnsOclTOS3ecLe5Uhh4KTErVePnW3o
- answer: HTTP/2 200 
server: nginx
date: Tue, 20 Dec 2022 10:48:51 GMT
content-type: application/json
content-length: 793
boulder-requester: 515162167
cache-control: public, max-age=0, no-cache
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
replay-nonce: 371CmL9LQe1OR5E2m1mOCaJPygED74-6xYCOQ_KETgV78mE
x-frame-options: DENY
strict-transport-security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "DOMAIN"
  },
  "status": "pending",
  "expires": "2022-12-27T10:48:51Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/188507753607/upM-qg"
,
      "token": "REDACTED1NxYMOnsOclTOS3ecLe5Uhh4KTErVePnW3o"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/188507753607/TGH3gA"
,
      "token": "REDACTED1NxYMOnsOclTOS3ecLe5Uhh4KTErVePnW3o"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/188507753607/Towuwg"
,
      "token": "REDACTED1NxYMOnsOclTOS3ecLe5Uhh4KTErVePnW3o"
    }
  ]
}


==[API call]==
exit status: 0


==[Step 5]==
- status: 200
- url: https://acme-v02.api.letsencrypt.org/acme/chall-v3/188507753607/upM-qg
- nonce: F977Df_98vqDY6D4OUMy3At4rwbzcBHfHf7amkTzBTfQXAw
- validation: pending
- details: 
- answer: HTTP/2 200 
server: nginx
date: Tue, 20 Dec 2022 10:49:01 GMT
content-type: application/json
content-length: 187
boulder-requester: 515162167
cache-control: public, max-age=0, no-cache
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
link: <https://acme-v02.api.letsencrypt.org/acme/authz-v3/188507753607>;rel="up"
location: https://acme-v02.api.letsencrypt.org/acme/chall-v3/188507753607/upM-qg
replay-nonce: F977Df_98vqDY6D4OUMy3At4rwbzcBHfHf7amkTzBTfQXAw
x-frame-options: DENY
strict-transport-security: max-age=604800

{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/188507753607/upM-qg",
  "token": "REDACTED1NxYMOnsOclTOS3ecLe5Uhh4KTErVePnW3o"
}


==[API call]==
exit status: 0


==[Step 5]==
- status: 400
- url: https://acme-v02.api.letsencrypt.org/acme/chall-v3/188507753607/upM-qg
- nonce: 2712Jw91S5_d1mfpPQwhvdc92QleimES2g8Tk6ozTJyZfbY
- validation: 
- details: Unable to update challenge :: authorization must be pending
- answer: HTTP/2 400 
server: nginx
date: Tue, 20 Dec 2022 10:49:05 GMT
content-type: application/problem+json
content-length: 144
boulder-requester: 515162167
cache-control: public, max-age=0, no-cache
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
replay-nonce: 2712Jw91S5_d1mfpPQwhvdc92QleimES2g8Tk6ozTJyZfbY

{
  "type": "urn:ietf:params:acme:error:malformed",
  "detail": "Unable to update challenge :: authorization must be pending",
  "status": 400
}


==[Debug information Step 5]==
{
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:unauthorized",
    "detail": "IP: Invalid response from http://DOMAIN/.well-known/ac
me-challenge/REDACTED1NxYMOnsOclTOS3ecLe5Uhh4KTErVePnW3o: 404",
    "status": 403
  },
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/188507753607/upM-qg",
  "token": "REDACTED1NxYMOnsOclTOS3ecLe5Uhh4KTErVePnW3o",
  "validationRecord": [
    {
      "url": "http://DOMAIN/.well-known/acme-challenge/REDACTED1NxYMOnsOclTOS3ecLe
5Uhh4KTErVePnW3o",
      "hostname": "DOMAIN",
      "port": "80",
      "addressesResolved": [
        "IP"
      ],
      "addressUsed": "IP"
    }
  ],
  "validated": "2022-12-20T10:49:01Z"
}


==[Abort Step 5]==
=> Wrong status

eris · January 10, 2023, 5:08pm

If you visits this page you should see a string but a 403 page got served without a domain name I can’t do a lot on debugging… You can try to restart nginx…

xeruf · January 10, 2023, 7:39pm

I did restart nginx and the whole machine, no luck.
Visiting that URL does produce a 403.

Raphael · January 10, 2023, 7:46pm

is your website redirecting? probaly custom htaccess?

xeruf · January 11, 2023, 8:18am

no, nothing customised, this happens for fresh domains as well!

linkp · January 11, 2023, 1:41pm

Have you tested with Let’s Debug?

xeruf · January 11, 2023, 4:58pm

yes I did, as written above, reported no issues either

eris · January 11, 2023, 5:27pm

With a domain I really can’t do any thing.

xeruf · January 13, 2023, 2:28pm

We checked it in private chat, this is the temporary fix provided by @eris

Replace:

github.com

hestiacp/hestiacp/blob/31d5f892e848e867338ec1d84e565f3683a8dbf3/bin/v-add-letsencrypt-domain#L344


      
          				sconf="$HOMEDIR/$user/conf/mail/$root_domain/nginx.ssl.conf_letsencrypt"
          				echo 'location ~ "^/\.well-known/acme-challenge/([-_A-Za-z0-9]+)$" {' \
          					> $conf
          				echo '    default_type text/plain;' >> $conf
          				echo '    return 200 "$1.'$THUMB'";' >> $conf
          				echo '}' >> $conf
          				if [ ! -e "$sconf" ]; then
          					ln -s "$conf" "$sconf"
          				fi
          				if [ -n "$PROXY_SYSTEM" ]; then
          					$BIN/v-restart-proxy
          					check_result $? "Proxy restart failed" > /dev/null
          				fi
          			else
          				get_object_value 'mail' 'DOMAIN' "$root_domain" "WEBMAIL"
          				if [ -n "$WEBMAIL" ]; then
          					well_known="/var/lib/$WEBMAIL/.well-known"
          					acme_challenge="$well_known/acme-challenge"
          					mkdir -p $acme_challenge
          					echo "$token.$THUMB" > $acme_challenge/$token
          					chown -R $user:$user $well_known

With

$BIN/v-restart-proxy yes

And

github.com

hestiacp/hestiacp/blob/31d5f892e848e867338ec1d84e565f3683a8dbf3/bin/v-add-letsencrypt-domain#L359


      
          				if [ -n "$WEBMAIL" ]; then
          					well_known="/var/lib/$WEBMAIL/.well-known"
          					acme_challenge="$well_known/acme-challenge"
          					mkdir -p $acme_challenge
          					echo "$token.$THUMB" > $acme_challenge/$token
          					chown -R $user:$user $well_known
          				fi
          			fi
          		fi
          		if [ "$WEB_SYSTEM" = 'nginx' ]; then
          			$BIN/v-restart-web
          			check_result $? "Web restart failed" > /dev/null
          		fi
          	fi
          
          
	if [ "$DNS_CLUSTER" = "yes" ]; then
          		$BIN/v-update-sys-queue dns-cluster
          	fi
          
          
	# Requesting ACME validation / STEP 5
          	validation_check=$(echo "$answer" | grep '"valid"')

With

$BIN/v-restart-web yes

And then run v-add-letsencrypt-domain user domain.com