Hello. I have many domains on the server and one of them is refusing to renew Lets Encrypt.
I get this error both from the console and the web interface.
Error: Let’s Encrypt finalize bad status 403.
Here is LE domain log:
1445 =============================
1446 Date Time: 2024-10-31 15:27:45
1447 WEB_SYSTEM: apache2
1448 PROXY_SYSTEM: nginx
1449 user: mousespecs
1450 domain: mousespecs.org
1451
1452
1453 - aliases: www.mousespecs.org
1454 - proto: http-01
1455 - wildcard:
1456
1457
1458 ==[Step 1]==
1459 - status: 200
1460 - nonce: ymL2jeiesa_NMFAT5ASfe2g4W-iCZ2n4FPVwYLQvWEGVwP8O8VM
1461 - answer: HTTP/2 200
1462 server: nginx
1463 date: Thu, 31 Oct 2024 14:27:46 GMT
1464 content-type: application/json
1465 content-length: 746
1466 cache-control: public, max-age=0, no-cache
1467 replay-nonce: ymL2jeiesa_NMFAT5ASfe2g4W-iCZ2n4FPVwYLQvWEGVwP8O8VM
1468 x-frame-options: DENY
1469 strict-transport-security: max-age=604800
1470
1471
1472
1473 ==[API call]==
1474 exit status: 0
1475
1476
1477 ==[Step 2]==
1478 - status: 201
1479 - nonce: l4CEvUUqY6lC9H9D5tShLkgDjPxxQswK3UU_gSDdhF9uZjvy2_M
1480 - authz: https://acme-v02.api.letsencrypt.org/acme/authz-v3/423680238627
1481 https://acme-v02.api.letsencrypt.org/acme/authz-v3/423719217607
1482 - finalize: https://acme-v02.api.letsencrypt.org/acme/finalize/839677907/318740509407
1483 - payload: {"identifiers":[{"type":"dns","value":"mousespecs.org"},{"type":"dns","value":"www.mousespecs.org"}]}
1484 - answer: HTTP/2 201
1485 server: nginx
1486 date: Thu, 31 Oct 2024 14:27:47 GMT
1487 content-type: application/json
1488 content-length: 480
1489 boulder-requester: 839677907
1490 cache-control: public, max-age=0, no-cache
1491 link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
1492 location: https://acme-v02.api.letsencrypt.org/acme/order/839677907/318740509407
1493 replay-nonce: l4CEvUUqY6lC9H9D5tShLkgDjPxxQswK3UU_gSDdhF9uZjvy2_M
1494 x-frame-options: DENY
1495 strict-transport-security: max-age=604800
1496
1497 {
1498 "status": "pending",
1499 "expires": "2024-11-07T14:27:47Z",
1500 "identifiers": [
1501 {
1502 "type": "dns",
1503 "value": "mousespecs.org"
1504 },
1505 {
1506 "type": "dns",
1507 "value": "www.mousespecs.org"
1508 }
1509 ],
1510 "authorizations": [
1511 "https://acme-v02.api.letsencrypt.org/acme/authz-v3/423680238627",
1512 "https://acme-v02.api.letsencrypt.org/acme/authz-v3/423719217607"
1513 ],
1514 "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/839677907/318740509407"
1515 }
1516 order: https://acme-v02.api.letsencrypt.org/acme/order/839677907/318740509407
1517
1518
1519 ==[API call]==
1520 exit status: 0
1521
1522
1523 ==[Step 3]==
1524 - status: 200
1525 - nonce: t7QKW8KeDVN-QiEhH1_wkiiyDLCndl3DTbJz1HZ2WW_i-0qFDOQ
1526 - url: https://acme-v02.api.letsencrypt.org/acme/chall-v3/423680238627/FR9UJg
1527 - token:
1528 - answer: HTTP/2 200
1529 server: nginx
1530 date: Thu, 31 Oct 2024 14:27:47 GMT
1531 content-type: application/json
1532 content-length: 757
1533 boulder-requester: 839677907
1534 cache-control: public, max-age=0, no-cache
1535 link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
1536 replay-nonce: t7QKW8KeDVN-QiEhH1_wkiiyDLCndl3DTbJz1HZ2WW_i-0qFDOQ
1537 x-frame-options: DENY
1538 strict-transport-security: max-age=604800
1539
1540 {
1541 "identifier": {
1542 "type": "dns",
1543 "value": "mousespecs.org"
1544 },
1545 "status": "valid",
1546 "expires": "2024-11-30T12:16:56Z",
1547 "challenges": [
1548 {
1549 "type": "http-01",
1550 "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/423680238627/FR9UJg",
1551 "status": "valid",
1552 "validated": "2024-10-31T12:16:35Z",
1553 "token": "KHVWXiDMBZqdqRQpuZtM_-1pfIehS7BhgP2pfnQHYBs",
1554 "validationRecord": [
1555 {
1556 "url": "http://mousespecs.org/.well-known/acme-challenge/KHVWXiDMBZqdqRQpuZtM_-1pfIehS7BhgP2pfnQHYBs",
1557 "hostname": "mousespecs.org",
1558 "port": "80",
1559 "addressesResolved": [
1560 "188.40.133.44"
1561 ],
1562 "addressUsed": "188.40.133.44"
1563 }
1564 ]
1565 }
1566 ]
1567 }
1568
1569
1570 ==[API call]==
1571 exit status: 0
1572
1573
1574 ==[Step 3]==
1575 - status: 200
1576 - nonce: XM4ukI6H80UEOa4IcgOLJtKGX0ZnWpmstft7S-Uo8AAAOaR3HFQ
1577 - url: https://acme-v02.api.letsencrypt.org/acme/chall-v3/423719217607/zPXqAA
1578 - token: MA_t_D53CBJ8ZX2aw02fUsnEzhKBOAqtL_dPdA22aHM
1579 - answer: HTTP/2 200
1580 server: nginx
1581 date: Thu, 31 Oct 2024 14:27:48 GMT
1582 content-type: application/json
1583 content-length: 802
1584 boulder-requester: 839677907
1585 cache-control: public, max-age=0, no-cache
1586 link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
1587 replay-nonce: XM4ukI6H80UEOa4IcgOLJtKGX0ZnWpmstft7S-Uo8AAAOaR3HFQ
1588 x-frame-options: DENY
1589 strict-transport-security: max-age=604800
1590
1591 {
1592 "identifier": {
1593 "type": "dns",
1594 "value": "www.mousespecs.org"
1595 },
1596 "status": "pending",
1597 "expires": "2024-11-07T14:27:47Z",
1598 "challenges": [
1599 {
1600 "type": "tls-alpn-01",
1601 "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/423719217607/s1a-Pg",
1602 "status": "pending",
1603 "token": "MA_t_D53CBJ8ZX2aw02fUsnEzhKBOAqtL_dPdA22aHM"
1604 },
1605 {
1606 "type": "dns-01",
1607 "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/423719217607/fdNyTQ",
1608 "status": "pending",
1609 "token": "MA_t_D53CBJ8ZX2aw02fUsnEzhKBOAqtL_dPdA22aHM"
1610 },
1611 {
1612 "type": "http-01",
1613 "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/423719217607/zPXqAA",
1614 "status": "pending",
1615 "token": "MA_t_D53CBJ8ZX2aw02fUsnEzhKBOAqtL_dPdA22aHM"
1616 }
1617 ]
1618 }
1619
1620
1621 ==[API call]==
1622 exit status: 0
1623
1624
1625 ==[Step 5]==
1626 - status: 200
1627 - url: https://acme-v02.api.letsencrypt.org/acme/chall-v3/423719217607/zPXqAA
1628 - nonce: l4CEvUUqTmsuN28K2xBe4ZmcpHUkIETcp4M1p908xAktuynpUbU
1629 - validation: https://acme-v02.api.letsencrypt.org/acme/chall-v3/423719217607/zPXqAA
1630 - details:
1631 - answer: HTTP/2 200
1632 server: nginx
1633 date: Thu, 31 Oct 2024 14:27:55 GMT
1634 content-type: application/json
1635 content-length: 187
1636 boulder-requester: 839677907
1637 cache-control: public, max-age=0, no-cache
1638 link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
1639 link: <https://acme-v02.api.letsencrypt.org/acme/authz-v3/423719217607>;rel="up"
1640 location: https://acme-v02.api.letsencrypt.org/acme/chall-v3/423719217607/zPXqAA
1641 replay-nonce: l4CEvUUqTmsuN28K2xBe4ZmcpHUkIETcp4M1p908xAktuynpUbU
1642 x-frame-options: DENY
1643 strict-transport-security: max-age=604800
1644
1645 {
1646 "type": "http-01",
1647 "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/423719217607/zPXqAA",
1648 "status": "pending",
1649 "token": "MA_t_D53CBJ8ZX2aw02fUsnEzhKBOAqtL_dPdA22aHM"
1650 }
1651
1652
1653 ==[API call]==
1654 exit status: 0
1655
1656
1657 ==[Step 6]==
1658 - status: 403
1659 - nonce: l4CEvUUqWKIIEcuUzclHu45mCfXmb74V86pXIiX0ZPq6QGnBoeg
1660 - payload: {"csr":"MIIFGzCCAwMCAQAwgZUxIjAgBgkqhkiG9w0BCQEWE2luZm9AbW91c2VzcGVjcy5vcmcxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNpc2NvMQ8wDQYDVQQKDAZIZXN0aWExCzAJBgNVBAsMAklUMRcwFQYDVQQDDA5tb3VzZXNwZWNzLm9yZzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMBdFgIj4zJ6TV0M8XnuPLYWDWCesmU_Ig7Dk3ThfeHyT8bij28hBzA2iA_RS9PhCIHcs3XhkMxlhxSB-4ho6MJluHNAtkQCI4cMXl9kr1mrKzQ5j4xD0za3WQkUsOrdJooKCxFEkWkFKy0fj6dy52JB7NX6GYLSOTVfqG8w2yEO_vVmMdZNATVjuLpi7lhBeJt115Qov6Y7braNXRpyiEaRGUSBxlq4ClArwZe6Iel4yw_lj-LFbV3uJfcPgaMSdVBuuoXNuGDUp5iIOvsXxyoIHSLkGf4p9nbzfaTp5Ur3ivkDBWz5DwY2ujOxlc1itIv-G78_JJ94HyqFd2pj0s_emWm-yRWIPDwjCs4drSxLcRYBloIFe5MdIBZ-lI-igK63SosA_kq5WA3Hm1SmchnssEfh6ia_8sp8vLjD5AsWNy4Tj_nAQ0mx7VlDLVLdctPiPK7Y0Khfvs3yqKuyNv0YZfQofeLXEt2xQs1MflVUQ6B5XaJk8g0n5w-wrqSsr4vD4M42Xh_iwJrmptf3A-0Xu5nQz9G8XiD0HbjnrnohC6YupmAA1ulD49IW3HXx4X78oHkc_HT3vsilcgbhDME7bTg7-ZfYLIshM3Dc0DdcDnQW2G7d6MoWl82sdPxTBLtfW8H2GPWG1hoegOhZ8Ka0jPuFykbdrtcwDPI1lg97AgMBAAGgQDA-BgkqhkiG9w0BCQ4xMTAvMC0GA1UdEQQmMCSCDm1vdXNlc3BlY3Mub3JnghJ3d3cubW91c2VzcGVjcy5vcmcwDQYJKoZIhvcNAQELBQADggIBAKj0-rxh497epXMomsHpCewmXHvQ3xntKoyPLC_gkqyLiJBZHAImOfh5dVi3qjmvA71cHndA4GzdAOvdmUR5hYQKMYGF9UalIEc8QHhydE4NX2eF94lnKnZ3CxO3A8bVOBQKRbOJ552jeXlm6heEda8zYhMqp54QX9e9BbC3YjJ_oWNQIpiB9nZeIFuAko88XKOiGU-AqcJlsfgnjN2x_wsfyrTaKr-Q7rnMHDtzKysLVYac3Z0KsYoxaXjg4cMIdnXimjOFhZL9c2-Ia7Lfc0nkJvxUYe4B2qL-9Hx6BEJ8yxjHqBrdi6wBCXsH3pwXQ286PgBuDCm5P3hHMBy7YH87eaX7L56lHITb5Qq-C6TewGw3C_xvKgxLwEAUGr_teVW52qr1jwv9BAQ6yZvO1h3XFb-P-D9rwAb3xz_GcGHwcCMXUBXgDuvPl8Vw0KnKE4XCqi6AGEfN9kNjp01yZ1EVewvD2jfl42OAehdHYE_kjM0tWpGGDnanyGRCs_4qsmKGGJG1Xkj5_KoGRDVPG21CbZ3A-DSgyeYBinWPuZGSSf4BDnXBXFNcjDPkpp-mh2w246gDlyrfH1xZNTXb_um6TN_RzM3jGz_JzlqYcqhzmxN5mDlDYqK6y6TATPcXxcu6At4lHbfMU_VQRV1jXfWvqQWmuE5_XWo8nGyMtQdL"}
1661 - certificate:
1662 - answer: HTTP/2 403
1663 server: nginx
1664 date: Thu, 31 Oct 2024 14:28:00 GMT
1665 content-type: application/problem+json
1666 content-length: 152
1667 boulder-requester: 839677907
1668 cache-control: public, max-age=0, no-cache
1669 link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
1670 replay-nonce: l4CEvUUqWKIIEcuUzclHu45mCfXmb74V86pXIiX0ZPq6QGnBoeg
1671
1672 {
1673 "type": "urn:ietf:params:acme:error:orderNotReady",
1674 "detail": "Order's status (\"pending\") is not acceptable for finalization",
1675 "status": 403
1676 }
1677
Nothing was changed to the account as I am aware, although it’s a client site. DNS settings look good.
I had https redirect in place causing 301 for http acme-challenge but it worked for 2 years with it. I now removed it and acme challenge return status 200.
root@server1 /var/log/hestia # curl -I http://mousespecs.org/.well-known/acme-challenge/0mNU9-woYSSawwNoFa_rrpS_EQ4yNxtI5C-xMMwSW6w
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 31 Oct 2024 14:49:46 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 87
Connection: keep-alive
Config files looks similar to other domains and it looks good. Lets debug return status as good.
In the access log, there is no IP trying to open acme challenge.
NGINX reloads fine.
Where should I look next?