174,940 certificates that fail to parse: Left/Exception

[tomfitzhenry]

Further to #27 , I have a bunch more certificates that fail to parse on hs-certificate.

The 174,940 certificates and code to reproduce the failures is available at https://gist.github.com/tomfitzhenry/9124641 .

In addition to Left error cases, I've included the few certificates that cause exceptions to be thrown too.

The certificates fail to parse for a variety of reasons. Here they are with the number of failures per group:

174504 Returned left: "signed object error: \"fromASN1: X509.SignatureALG: unknown format\""
 371 Returned left: "ParsingPartial"
  38 Threw exception: TypeDecodingFailed "integer: not shortest encoding"
  23 Returned left: "signed object error: \"fromASN1: X509.ExtensionRaw: OID=[2,5,29,17]: cannot decode data: ParsingPartial\""
   1 Returned left: "signed object error: \"expecting [OID,String] got [OID [2,5,4,45],BitString (BitArray 96 \\\"rvEoBcr+xUA=\\\")]\""
   1 Returned left: "signed object error: \"expecting [OID,String] got [OID [2,5,4,45],BitString (BitArray 96 \\\"7oo9lCJ5/us=\\\")]\""
   1 Returned left: "signed object error: \"expecting [OID,String] got [OID [2,5,4,45],BitString (BitArray 112 \\"example.com")]\""
   1 Returned left: "signed object error sigalg: fromASN1: X509.SignatureALG: unknown format"

[cl04]

Many certificates in failed-certificates.txt.bz2 are not signed, it just a raw TBSCertificate (https://tools.ietf.org/html/rfc5280, section 4.1), without any signature Algorithm Identifier, nor Signature Value (BitArray of signaure_bits, 2048 when RSA-2048 is used).

I.E.:

( ex2 is one of the failing certificate in failed-certificates.txt.bz2. )

> let ex2 = Data.ByteString.Char8.pack "0\130\STX\171\160\ETX\STX\SOH\STX\STX\a\ENQ\r\164\247\246\137\185\&0\r\ACK\t*\134H\134\247\r\SOH\SOH\ENQ\ENQ\NUL0\DEL1\v0\t\ACK\ETXU\EOT\ACK\DC3\STXGB1\SI0\r\ACK\ETXU\EOT\b\f\ACKLondon1\ETB0\NAK\ACK\ETXU\EOT\n\f\SOGoogle UK Ltd.1!0\US\ACK\ETXU\EOT\v\f\CANCertificate Transparency1#0!\ACK\ETXU\EOT\ETX\f\SUBMerge Delay Intermediate 10\RS\ETB\r150127164429Z\ETB\r150128164429Z071\v0\t\ACK\ETXU\EOT\ACK\DC3\STXGB1(0&\ACK\ETXU\EOT\n\f\USGoogle Certificate Transparency0\130\SOH\"0\r\ACK\t*\134H\134\247\r\SOH\SOH\SOH\ENQ\NUL\ETX\130\SOH\SI\NUL0\130\SOH\n\STX\130\SOH\SOH\NUL\213#\184\194\204\181\SYN3\248\231\206f\167-\163N#W6\145d[zME\130\226+7\228t\198]\243\SYN\132s\SO{\EM\214p\ENQ\210\128\&6\142q!J\171\SI\223\132\b\242=@\129\245@\RSeqt\245\178\130N\SO\236\226(H6\220\252\232\145S\ACK*\239\129W)\143z<\254\136%\199{Y\195\231W\187`F\231x\DC1\185U\174'\ETX\231\EOT\235\t\230\224\243Le8\180Q\205\139?\143\154\216-\140y\199\243\203\186\v\a\203 \143\165\179\163;r\149\181\193\DC1\251\182wV\223{}\147U\232\164\132\180:c\189jO\FS\246+4iS\182\215j\159\234\ENQ\140\250d\161\171\204G-0<\214\209\DLE\233\r@0\254\177\167n\187@\166\164\140\DC3\195\154\149\160)\SI\252'\235G\206\194\172\240\128#\247\203y\195\131\178\190\a\157*~?\a\232\158\244\aQ\150\ETBnv\224\&14\246\226\135\186\218\133\201\174\174'\STX\ETX\SOH\NUL\SOH\163\129\139\&0\129\136\&0#\ACK\ETXU\GS\DC1\EOT\FS0\SUB\130\CANflowers-to-the-world.com0\f\ACK\ETXU\GS\DC3\SOH\SOH\255\EOT\STX0\NUL0\DC3\ACK\ETXU\GS%\EOT\f0\n\ACK\b+\ACK\SOH\ENQ\ENQ\a\ETX\SOH0\US\ACK\ETXU\GS#\EOT\CAN0\SYN\128\DC4\233<\EOT\225\128/\194\132\DC3-&p\158\242\253\SUB\207\170\254\198\&0\GS\ACK\ETXU\GS\SO\EOT\SYN\EOT\DC4\188\184\204: I^\211%\136l\"\203\NULd54\226\212S" :: ByteString

> let Right x1 = Data.ASN1.Encoding.decodeASN1Repr' Data.ASN1.BinaryEncoding.BER ex2
> fromASN1 (init . tail . map fst $ x1) :: Either String (Certificate, [ASN1])
> Right (Certificate {certVersion = 2, certSerial = 1422377069480377, certSignatureAlg = SignatureALG HashSHA1 PubKeyALG_RSA, certIssuerDN = DistinguishedName {getDistinguishedElements = [([2,5,4,6],ASN1CharacterString {characterEncoding = Printable, getCharacterStringRawData = "GB"}),([2,5,4,8],ASN1CharacterString {characterEncoding = UTF8, getCharacterStringRawData = "London"}),([2,5,4,10],ASN1CharacterString {characterEncoding = UTF8, getCharacterStringRawData = "Google UK Ltd."}),([2,5,4,11],ASN1CharacterString {characterEncoding = UTF8, getCharacterStringRawData = "Certificate Transparency"}),([2,5,4,3],ASN1CharacterString {characterEncoding = UTF8, getCharacterStringRawData = "Merge Delay Intermediate 1"})]}, certValidity = (DateTime {dtDate = Date {dateYear = 2015, dateMonth = January, dateDay = 27}, dtTime = TimeOfDay {todHour = 16h, todMin = 44m, todSec = 29s, todNSec = 0ns}},DateTime {dtDate = Date {dateYear = 2015, dateMonth = January, dateDay = 28}, dtTime = TimeOfDay {todHour = 16h, todMin = 44m, todSec = 29s, todNSec = 0ns}}), certSubjectDN = DistinguishedName {getDistinguishedElements = [([2,5,4,6],ASN1CharacterString {characterEncoding = Printable, getCharacterStringRawData = "GB"}),([2,5,4,10],ASN1CharacterString {characterEncoding = UTF8, getCharacterStringRawData = "Google Certificate Transparency"})]}, certPubKey = PubKeyRSA (PublicKey {public_size = 256, public_n = 26906373995610626272186541780221586693272555406421871263485864782915726711992473808553907282757790253414394899523785467773274550622044428067743688498147306099333882115613053284212571749735626348182382838734983771502543586987870724643710895843735429376942245368156854244393363911917994157998231963440649730411557027557971202572782068873106994704870785631554959034358457745284316951779699487173332799888730425244581238875033401563974856342850988505695522495614792899531496847219710613087387843202424581167471888722436482591229320612577008316563402045366074721019469971069680729395903462448504797991455680190446589357607, public_e = 65537}), certExtensions = Extensions (Just [ExtensionRaw {extRawOID = [2,5,29,17], extRawCritical = False, extRawASN1 = [Start Sequence,Other Context 2 "flowers-to-the-world.com",End Sequence]},ExtensionRaw {extRawOID = [2,5,29,19], extRawCritical = True, extRawASN1 = [Start Sequence,End Sequence]},ExtensionRaw {extRawOID = [2,5,29,37], extRawCritical = False, extRawASN1 = [Start Sequence,OID [1,3,6,1,5,5,7,3,1],End Sequence]},ExtensionRaw {extRawOID = [2,5,29,35], extRawCritical = False, extRawASN1 = [Start Sequence,Other Context 0 "\233<\EOT\225\128/\194\132\DC3-&p\158\242\253\SUB\207\170\254\198",End Sequence]},ExtensionRaw {extRawOID = [2,5,29,14], extRawCritical = False, extRawASN1 = [OctetString "\188\184\204: I^\211%\136l\"\203\NULd54\226\212S"]}])},[])
> fmap snd (fromASN1 (init . tail . map fst $ x1) :: Either String (Certificate, [ASN1]))
> Right []

As you can see, there's nothing left for parsing. While with X509 signed certificate, there're should be

    SEQUENCE(signatureAlgorithm   AlgorithmIdentifier);
    SEQUENCE(signatureValue       BIT STRING);

If you try fromASN1 and parse it as a Certificate only, it should return success.

Thanks