Add ASN1_F_ASN1_CHECK_TLEN

[palaviv]

needed for #3369.

[mention-bot]

@palaviv, thanks for your PR! By analyzing the history of the files in this pull request, we identified @reaperhulk, @public and @alex to be potential reviewers.

[reaperhulk]

@palaviv unrelated to this PR, but do you think the current DH work needs to land before we do a 1.8 release? Or could roll it into a 1.9 (March release)? Is there anything in current master that is broken or unusable re: DH right now?

[palaviv]

As was discussed in #3364 @mtury and @markrwilliams couldn't use some valid parameters becuase of the DH_check issue. #3369 will give them more flexibility but won't solve all their problems (@mtury, @markrwilliams please correct me if I am wrong). We will need to improve DH_check as discussed in #3364.
When would you like to release 1.8? I have time now if you want to close #3369 I can come to the IRC channel if it will help.

[reaperhulk]

I'd like to get 1.8 out tomorrow if possible actually. Otherwise it'll have to wait until early next week. If you want to hop in channel that would definitely help with quicker review turnaround on that other PR.

[ria4]

Indeed #3369 might help (and you've put impressive work in it), yet it won't solve the whole problem on my side. TLS does not provide a way to transmit the subgroup order, thus when receiving DH group parameters with a long generator from a server, I'll still be blocked by the DH_check.

At this point I'm thinking of overloading the dh_parameters_supported backend test from my program. That's unfortunate, but hey, I won't complain because of openssl providing security checks which TLS specifications do not allow to use...

(As a side note, TLS 1.3 forbids the use of custom groups in favor of the five ones defined in RFC 7919 which all have a generator g = 2, so that's good.)

In any case, I'll still try to provide the test vectors as discussed in #3364!