problem adding support for OVH

[m-martin-78]

Hi,

I tried to allow support of OVH API as a DNS provider, so I created the OVH plugin based on the Infoblox class:

@PluginRegistry.register_plugin("ovh")
class OvhClient(ChallengeSolver):
    """OVH DNS-01 challenge solver.

    This challenge solver connects to the OVH API to provision
    DNS TXT records in order to complete the ACME DNS-01 challenge type.
    """

    SUPPORTED_CHALLENGES = frozenset([ChallengeType.DNS_01])
    """The types of challenges that the solver supports."""

    POLLING_DELAY = 1.0
    """Time in seconds between consecutive DNS requests."""

    POLLING_TIMEOUT = 60.0 * 5
    """Time in seconds after which placing the TXT record is considered a failure."""

    DEFAULT_DNS_SERVERS = ["1.1.1.1", "8.8.8.8"]
    """The DNS servers to use if none are specified during initialization."""


    def __init__(self, *, ovh_endpoint, ovh_application_key, ovh_application_secret, ovh_consumer_key, dns_servers=None):
        self._client = ovh.Client(
        	endpoint = ovh_endpoint,
        	application_key = ovh_application_key,
        	application_secret = ovh_application_secret,
        	consumer_key = ovh_consumer_key,
        )
        self._psl = PSL()
        self._loop = asyncio.get_event_loop()

        self._resolvers = []

        for nameserver in dns_servers or self.DEFAULT_DNS_SERVERS:
            resolver = dns.asyncresolver.Resolver()
            resolver.nameservers = [nameserver]
            self._resolvers.append(resolver)


    async def connect(self):
        """Connects to the InfoBlox API.

        This method must be called before attempting to complete challenges.
        """
        return

    async def set_txt_record(self, name: str, text: str, views=None, ttl: int = 60):
        """Sets a DNS TXT record.

        :param name: The name of the TXT record.
        :param text: The text of the TXT record.
        :param views: List of views to set the TXT record in. Defaults to *Intern* and *Extern*.
        :param ttl: Time to live of the TXT record in seconds.
        """
        logger.debug("Setting TXT record %s = %s, TTL %d", name, text, ttl)
        apex = self._psl.private_suffix(name)
        privatepart = '.'.join(self._psl.privateparts(name)[:-1])
        # create record
        res = self._client.post(f'/domain/zone/{apex}/record',
            fieldtype="TXT",
            subDomain=privatepart,
            target=text,
            TTL=60)
        # apply modifications to zone (publish record)
        res = self._client.post(f'/domain/zone/{apex}/refresh')
        

    async def delete_txt_record(self, name: str, text: str):
        """Deletes a DNS TXT record.

        :param name: The name of the TXT record to delete.
        :param text: The text of the TXT record to delete.
        """
        logger.debug("Deleting TXT record %s = %s", name, text)
        apex = self._psl.private_suffix(name)
        privatepart = '.'.join(self._psl.privateparts(name)[:-1])
        # find record
        res = self._client.get(f'/domain/zone/{apex}/record',
            fieldtype="TXT",
            subDomain=privatepart)
        candidates_ids = [i for i in res]
        corresponding_id = None
        for i in candidates_ids:
            res = self._client.get(f'/domain/zone/{apex}/record/{i}')
            if res['target'] == text:
                corresponding_id = i
        if corresponding_id is None:
            raise ValueError('Could not find matching record to delete.')
        # delete record
        res = self._client.delete(f'/domain/zone/{apex}/record/{corresponding_id}')
        # apply modifications to zone (publish record)
        res = self._client.post(f'/domain/zone/{apex}/refresh')

    async def query_txt_record(
        self, resolver: dns.asyncresolver.Resolver, name: str
    ) -> typing.Set[str]:
        """Queries a DNS TXT record.

        :param name: Name of the TXT record to query.
        :return: Set of strings stored in the TXT record.
        """
        txt_records = []

        with contextlib.suppress(
            dns.asyncresolver.NXDOMAIN, dns.asyncresolver.NoAnswer
        ):
            resp = await resolver.resolve(name, "TXT")

            for records in resp.rrset.items.keys():
                txt_records.extend([record.decode() for record in records.strings])

        return set(txt_records)

    async def _query_until_completed(self, name, text):
        while True:
            record_sets = await asyncio.gather(
                *[self.query_txt_record(resolver, name) for resolver in self._resolvers]
            )

            # Determine set of records that has been seen by all name servers
            seen_by_all = set.intersection(*record_sets)

            if text in seen_by_all:
                return

            logger.debug(
                f"{name} does not have TXT {text} yet. Retrying (Records seen by all name servers: {seen_by_all}"
            )
            logger.debug(f"Records seen: {record_sets}")
            await asyncio.sleep(1.0)

    async def complete_challenge(
        self,
        key: josepy.jwk.JWK,
        identifier: acme.messages.Identifier,
        challenge: acme.messages.ChallengeBody,
    ):
        """Completes the given DNS-01 challenge.

        This method provisions the TXT record needed to complete the given challenge.
        Then it polls the DNS for up to :attr:`POLLING_TIMEOUT` seconds to ensure that the record is visible
        to the remote CA's DNS.

        :param key: The client's account key.
        :param identifier: The identifier that is associated with the challenge.
        :param challenge: The challenge to be completed.
        :raises: :class:`~acmetk.client.exceptions.CouldNotCompleteChallenge`
            If the challenge completion attempt failed.
        """
        name = challenge.chall.validation_domain_name(identifier.value)
        text = challenge.chall.validation(key)

        try:
            await self.set_txt_record(name, text)
        except Exception as e:
            logger.exception(
                "Could not set TXT record to solve challenge: %s = %s", name, text
            )
            raise CouldNotCompleteChallenge(
                challenge,
                acme.messages.Error(typ="infoblox", title="error", detail=str(e)),
            )

        # Poll the DNS until the correct record is available
        try:
            await asyncio.wait_for(
                self._query_until_completed(name, text), self.POLLING_TIMEOUT
            )
        except asyncio.TimeoutError:
            raise CouldNotCompleteChallenge(
                challenge,
                acme.messages.Error(
                    typ="infoblox",
                    title="error",
                    detail="Could not complete challenge due to a DNS polling timeout",
                ),
            )

    async def cleanup_challenge(
        self,
        key: josepy.jwk.JWK,
        identifier: acme.messages.Identifier,
        challenge: acme.messages.ChallengeBody,
    ):
        """Performs cleanup for the given challenge.

        This method de-provisions the TXT record that was created to complete the given challenge.

        :param key: The client's account key.
        :param identifier: The identifier that is associated with the challenge.
        :param challenge: The challenge to clean up after.
        """
        name = challenge.chall.validation_domain_name(identifier.value)
        text = challenge.chall.validation(key)

        await self.delete_txt_record(name, text)

My problem is that when my Caddy webserver is pointed to the acme server (I removed the reverse-proxy as it cannot use acmetk itself to get a certificate), I keep getting a problem about deserializing "status" in JWTs:

2023-08-18 16:42:47,954 - aiohttp.access - INFO - 10.0.2.100 [18/Aug/2023:14:42:47 +0000] "POST /new-order HTTP/1.1" 400 389 "-" "Caddy/ CertMagic acmez (linux; amd64)"
2023-08-18 16:42:48,222 - aiohttp.server - ERROR - Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/josepy/json_util.py", line 321, in fields_from_json
    fields[slot] = field.decode(value)
  File "/usr/local/lib/python3.9/site-packages/josepy/json_util.py", line 110, in decode
    return self.fdec(value)
  File "/usr/local/lib/python3.9/site-packages/acme/messages.py", line 87, in from_json
    raise jose.DeserializationError(f'{cls.__name__} not recognized')
josepy.errors.DeserializationError: Deserialization error: Status not recognized

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/app/acmetk/server/server.py", line 1253, in error_middleware
    response = await handler(request)
  File "/app/acmetk/server/server.py", line 1216, in host_ip_middleware
    return await handler(request)
  File "/app/acmetk/server/server.py", line 1243, in aiohttp_jinja2_middleware
    return await handler(request)
  File "/app/acmetk/server/server.py", line 1567, in new_order
    obj = acme.messages.NewOrder.json_loads(jws.payload)
  File "/usr/local/lib/python3.9/site-packages/josepy/interfaces.py", line 177, in json_loads
    return cls.from_json(loads)
  File "/usr/local/lib/python3.9/site-packages/josepy/json_util.py", line 331, in from_json
    return cls(**cls.fields_from_json(jobj))
  File "/usr/local/lib/python3.9/site-packages/josepy/json_util.py", line 323, in fields_from_json
    raise errors.DeserializationError('Could not decode {0!r} ({1!r}): {2}'.format(
josepy.errors.DeserializationError: Deserialization error: Could not decode 'status' (''): Deserialization error: Status not recognized

I also tried updating josepy and all the other dependences of the project, but I still have the same problem.

[commonism]

The error is unrelated to your ovh dns01 validation.

acmetk/acmetk/server/server.py

Lines 744 to 754 in 4bf6202

	@routes.post("/new-order", name="new-order")
	async def new_order(self, request):
	"""Handler that creates a new order.
	`7.4. Applying for Certificate Issuance <https://tools.ietf.org/html/rfc8555#section-7.4>`_
	:return: The order object.
	"""
	async with self._session(request) as session:
	jws, account = await self._verify_request(request, session)
	obj = acme.messages.NewOrder.json_loads(jws.payload)

The message sent by caddy is not valid.
Deserialization of the status field fails, "" is not recognized as valid value.
https://datatracker.ietf.org/doc/html/rfc8555#section-7.1.3
https://github.com/certbot/certbot/blob/b1978ff18837e40d16eedf2090330af53d8ceaa5/acme/acme/messages.py#L207-L219
https://github.com/certbot/certbot/blob/b1978ff18837e40d16eedf2090330af53d8ceaa5/acme/acme/messages.py#L613-L681

I'd look into using lexicon to get ovh dns01 validation, maybe pick up my PR?
https://github.com/noahkw/acmetk/pull/63/files

That said - currently I'd consider this code base unmaintained.

[m-martin-78]

@commonism: Thanks for your help! considering the error message, I indeed think that the problem comes from the Caddy ACME client (or the parser if a null value is valid according to the protocol) and not from the OVH challenge solver as it is not even called at this point. I'll try your PR this week I hope.

I understand this project is unmaintained, the latest commit is two years old, yet the need is still there: allowing internal resources to obtain LE certificates without giving each of them the keys to the public DNS zone. Do you know of an alternative project that does that?

[commonism]

Your uscase is exactly what this project was meant to cover.
I'd go with it.

For your debug - have caddy access the acme service via http, capture stream or dump the data in the service
Share the capture, I'll have a look.
But I think I 'll have to refer to caddy, and I I'd expect them to refer you to acmez.

[commonism]

In case you are still interested … it's in progress.

• protocol compatibility issue with caddy/acmez has been worked out with the help of @mholt in Set omitempty on Order.Status field mholt/acmez#21

• code is getting updated to match current versions of all used libraries used in update dependencies & use ci for unit testing #65

• github CI/codecov.io will be used for codecoverage/unit testing update dependencies & use ci for unit testing #65 as well

• picking up the lexicon PR add solver plugin using the lexicon library to solve dns challenges #63

• container images

  • update broker
  • update db
  • replace reverse_proxy openresty with apache2/mod_md or caddy as acmez got part of the unit tests

• increase code coverage

  • tests for challenge solving/validation most likely be part of add solver plugin using the lexicon library to solve dns challenges #63
  • management ui