Several Docs Missing in Repo, but Referenced and Published on Live AWS User Guide?

[ScriptAutomate]

I've been auto-generating code from CFN Resource specification files, and part of the generation runs through and automatically builds help documentation. Though, many of the source files that I would be pulling from only exist on the live user guide and not in the GitHub source documentation. Is there some kind of release strategy where documentation becomes available in the official AWS documentation before hitting the source repo here?

Existing Public Page	Missing from GitHub
AWS::CodeBuild::SourceCredential	aws-resource-codebuild-sourcecredential.md
AWS::CodeStar::GitHubRepository	aws-resource-codestar-githubrepository.md
AWS::Glue::MLTransform	aws-resource-glue-mltransform.md
AWS::LakeFormation::DataLakeSettings	aws-resource-lakeformation-datalakesettings.md
AWS::LakeFormation::Permissions	aws-resource-lakeformation-permissions.md
AWS::LakeFormation::Resource	aws-resource-lakeformation-resource.md
AWS::Batch::JobDefinition Device	aws-properties-batch-jobdefinition-device.md
AWS::Batch::JobDefinition LinuxParameters	aws-properties-batch-jobdefinition-containerproperties-linuxparameters.md
AWS::CodeStar::GitHubRepository Code	aws-properties-codestar-githubrepository-code.md
AWS::CodeStar::GitHubRepository S3	aws-properties-codestar-githubrepository-s3.md
AWS::Cognito::UserPool UserPoolAddOns	aws-properties-cognito-userpool-userpooladdons.md
AWS::Cognito::UserPool VerificationMessageTemplate	aws-properties-cognito-userpool-verificationmessagetemplate.md
AWS::Cognito::UserPoolClient AnalyticsConfiguration	aws-properties-cognito-userpoolclient-analyticsconfiguration.md
AWS::DLM::LifecyclePolicy Parameters	aws-properties-dlm-lifecyclepolicy-parameters.md
AWS::EFS::FileSystem LifecyclePolicy	aws-properties-elasticfilesystem-filesystem-lifecyclepolicy.md
AWS::Glue::MLTransform FindMatchesParameters	aws-properties-glue-mltransform-transformparameters-findmatchesparameters.md
AWS::Glue::MLTransform GlueTables	aws-properties-glue-mltransform-inputrecordtables-gluetables.md
AWS::Glue::MLTransform InputRecordTables	aws-properties-glue-mltransform-inputrecordtables.md
AWS::Glue::MLTransform TransformParameters	aws-properties-glue-mltransform-transformparameters.md
AWS::LakeFormation::DataLakeSettings Admins	aws-properties-lakeformation-datalakesettings-admins.md
AWS::LakeFormation::DataLakeSettings DataLakePrincipal	aws-properties-lakeformation-datalakesettings-datalakeprincipal.md
AWS::LakeFormation::Permissions DatabaseResource	aws-properties-lakeformation-permissions-databaseresource.md
AWS::LakeFormation::Permissions DataLakePrincipal	aws-properties-lakeformation-permissions-datalakeprincipal.md
AWS::LakeFormation::Permissions Resource	aws-properties-lakeformation-permissions-resource.md
AWS::LakeFormation::Permissions TableResource	aws-properties-lakeformation-permissions-tableresource.md

[AWSRandall]

Hey, thanks for bringing up the issue! Your script sounds cool. I'd be interested to know whether you also generate the content portions (outside of the structures and parameters).

We also auto-generate our documentation from the CFN resource specification. As such there is a slight delay before we update our GitHub docs repo with the latest supported resources and properties (see the Release History here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html). We try to keep the delay as short as possible so there's very little drift.

Having said all that, we'll double check your diff to see if these all fall into that category of recently added resources/properties. Thanks for pointing this out!

[ScriptAutomate]

Hey, thanks for bringing up the issue! Your script sounds cool. I'd be interested to know whether you also generate the content portions (outside of the structures and parameters).

We also auto-generate our documentation from the CFN resource specification. As such there is a slight delay before we update our GitHub docs repo with the latest supported resources and properties (see the Release History here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html). We try to keep the delay as short as possible so there's very little drift.

Ah, cool, that helps explain. I completely overlooked the Release History page! I'll definitely have some comparison lookups in the future to see if missing stuff is likeley new stuff on delay. I should have also said this is using the latest us-west-2 cfn resource spec file, as I haven't included eval of the other regions.

I do actually pull other content to add into my code, for documentation purposes, such as resource and parameter descriptions. The current example is where I am using CFN resource documentation to generate resource-specific PowerShell Core functions. Here is an example output of one:

function New-AWSCFNApiGatewayAccount {
<#
.Synopsis
The AWS::ApiGateway::Account resource specifies the IAM role that Amazon API Gateway uses to write API logs to Amazon CloudWatch Logs.
.DESCRIPTION
The AWS::ApiGateway::Account resource specifies the IAM role that Amazon API Gateway uses to write API logs to Amazon CloudWatch Logs.
**Important**
If an API Gateway resource has never been created in your AWS account, you must add a dependency on another API Gateway resource, such as an AWS::ApiGateway::RestApi: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-restapi.html or AWS::ApiGateway::ApiKey: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-apikey.html resource.
If an API Gateway resource has been created in your AWS account, no dependency is required (even if the resource was deleted.
 .PARAMETER CloudWatchRoleArn
The Amazon Resource Name (ARN of an IAM role that has write access to CloudWatch Logs in your account.
 .LINK
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-account.html
 #>
  [CmdletBinding()]
  param(
    [Parameter(Mandatory=$False)]
    [String]$CloudWatchRoleArn
  )
  [PSCustomObject]@{
    'Type' = 'AWS::ApiGateway::Account'
    'Properties' = @{
      'CloudWatchRoleArn' = "$CloudWatchRoleArn"
    }
  }
}

PowerShell Core comment-based help allows for help documentation in-terminal output to look like this:

PS /home/scriptautomate> Get-Help New-AWSCFNApiGatewayAccount -Full  

NAME
    New-AWSCFNApiGatewayAccount
    
SYNOPSIS
    The AWS::ApiGateway::Account resource specifies the IAM role that Amazon API Gateway uses to write API logs to Amazon CloudWatch Logs.
    
    
SYNTAX
    New-AWSCFNApiGatewayAccount [[-CloudWatchRoleArn] <String>] [<CommonParameters>]
    
    
DESCRIPTION
    The AWS::ApiGateway::Account resource specifies the IAM role that Amazon API Gateway uses to write API logs to Amazon CloudWatch Logs.
    **Important**
    If an API Gateway resource has never been created in your AWS account, you must add a dependency on another API Gateway resource, such 
    as an AWS::ApiGateway::RestApi: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-restapi.html or 
    AWS::ApiGateway::ApiKey: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-apikey.html resource.
    If an API Gateway resource has been created in your AWS account, no dependency is required (even if the resource was deleted.
    

PARAMETERS
    -CloudWatchRoleArn <String>
        The Amazon Resource Name (ARN of an IAM role that has write access to CloudWatch Logs in your account.
        
        Required?                    false
        Position?                    1
        Default value                
        Accept pipeline input?       false
        Accept wildcard characters?  false
        
    <CommonParameters>
        This cmdlet supports the common parameters: Verbose, Debug,
        ErrorAction, ErrorVariable, WarningAction, WarningVariable,
        OutBuffer, PipelineVariable, and OutVariable. For more information, see
        about_CommonParameters (https://go.microsoft.com/fwlink/?LinkID=113216). 
    
INPUTS
    
OUTPUTS
    
    
RELATED LINKS
    http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-account.html

With the ability to convert into JSON:

PS /home/scriptautomate> New-AWSCFNApiGatewayAccount -CloudWatchRoleArn 'arn.xxxxx.xxx' | ConvertTo-Json
{
  "Type": "AWS::ApiGateway::Account",
  "Properties": {
    "CloudWatchRoleArn": "arn.xxxxx.xxx"
  }
}

It's a work in progress, but being able to simply pull this git repo, and parse the appropriate Markdown docs allows for easy built-in documentation/help generation. I'm going to be extending it to include region-specific rules/checks, so I had to build a tool that will create and manage my own reference repository as I couldn't find the spec files in any AWS github repo: ScriptAutomate/aws-cfn-resource-specs (I point this out in #4 )

Broken Documentation Links (using us-west-2 spec file as reference)

Does the lag in source repo documentation replication explain these Broken Documentation Links in CloudFormation Resource Specification Files? These links include for base of resource, and all sub-instances of link referenced by sub properties:

Resource	BrokenLink	WorkingLink
AWS::AutoScaling::AutoScalingGroup.InstancesDistribution	Broke	ReplaceWith
AWS::AutoScaling::AutoScalingGroup.MixedInstancesPolicy	Broke	ReplaceWith

If not, I'll open a second issue in relation to the broken links.

[PatMyron]

@ScriptAutomate Resource specification for us-east-1 will have a bit more than us-west-2:

"region-specific rules/checks" is interesting, would cfn-lint be a good fit?

[ScriptAutomate]

@PatMyron thanks for the info on most-populated resource specs, and I'll also be taking a look at cfn-lint!

@AWSRandall said:
We also auto-generate our documentation from the CFN resource specification. As such there is a slight delay before we update our GitHub docs repo with the latest supported resources and properties (see the Release History here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html). We try to keep the delay as short as possible so there's very little drift.

Looking at the release history over the past week, there haven't been any new resource specs that have been published reflecting the updates from the Aug. 29th update. Is there a similar lag time that occurs with spec json files reflecting changes referenced in release version history? For example, AWS::Config::OrganizationConfigRule was recently added as a new Config Resource Type, but doesn't appear in any region's cfn resource spec json (v5.3.0 is latest as of Aug. 31).

Having said all that, we'll double check your diff to see if these all fall into that category of recently added resources/properties. Thanks for pointing this out!

I haven't done a new Documentation link scan since opening this issue, so I don't know if any new doc links may have been added that are to documentation not with the source repo. With that being said, and reviewing the table in this issue, I believe the GitHub source documentation has been fully caught up with what I found missing or broken except for:

Existing Public Page	Missing from GitHub	Release History Date Ref
AWS::EFS::FileSystem LifecyclePolicy	aws-properties-elasticfilesystem-filesystem-lifecyclepolicy.md	June 13, 2019

Remaining stuff:

• I'll leave this issue open in the meantime until the documentation is added here, since this doesn't seem to be a natural time gap related to lag time. I'd like to include a day count range in CFN spec resource file updates where, if the documentation has yet to appear in the GitHub source doc repo, it will wait x amount of days before an issue should be raised in this repository. I'll probably set a 1 - 2 week expectation in my tooling, so that I raise an issue only if any documentation is still lagging behind. Would an expectation of that length be reasonable?
• Should I open a new issue for the broken Documentation links referenced in the CFN resource spec files? As in, the links that are present, but have since moved or were otherwise incorrect URL links? I am only asking because I do not know where else to raise this issue, since the CFN resource files themselves are not versioned within this AWS repo and this documentation repo seemed the best bet (I created Missing CloudFormationResourceSpecification.json Files in This Repository #4 a while back for this reason). The latest CFN spec version (v5.3.0) still has the following broken Documentation links for these resources (total of 10 broken links in the spec):

Resource	BrokenLink	WorkingLink
AWS::AutoScaling::AutoScalingGroup.InstancesDistribution	Broke	ReplaceWith
AWS::AutoScaling::AutoScalingGroup.MixedInstancesPolicy	Broke	ReplaceWith

NOTE: I've created an issue in my own repository for these broken links where I am tracking these files at ScriptAutomate/aws-cfn-resource-specs: [v5.1.0] Broken Documentation Links

[ScriptAutomate]

UPDATE: I have finished some automated tooling around broken Documentation field lookups in CFN resource spec files, as issue tracked here: [Documentation Tracker JSON] Verification of Documentation Links Working, and Existing in AWS CFN Doc Source Repo. Here are most-current results:

• ScriptAutomate/aws-cfn-resource-specs/documentation-lookup-errors.json
• ScriptAutomate/aws-cfn-resource-specs/documentation-broken-links-detailed.json <-- Contains manual work done to find location of MissingDocLinks (which are Documentation key values that don't exist in the CFN resource spec files even though they should) and BrokenDocLinks solutions, if I could figure out a resolution.

I found a total of 17 issues. They seem to be a mix of issues around:

• GitHub AWS CFN UserGuide Doc source
• AWS CFN UserGuide website
• The CloudFormationResourceSpecification.json files
• The process in which the GitHub source is updated to reflect the UserGuide website content

Here are the types of issues I've come across:

BrokenDocLinks

• AWS Doc html page is broken, with this page unreachable and redirecting to UserGuide home. Seems to be an issue with the AWS CloudFormation User Guide website.
• URL in Documentation key of CloudFormationResourceSpecification.json file is pointing to an incorrect html page, and needs to be updated. Seems to be an issue with the CloudFormationResourceSpecification.json file.

MissingDocLinks

• Documentation key is not present in the CloudFormationResourceSpecification.json file, and should be added. Seems to be an issue with the CloudFormationResourceSpecification.json file.

MissingFromAWSGitHubSourceRepo

• Source repo pages are missing, even though they are present in AWS CFN UserGuide docs. Both causes seem to point to the possibility of the GitHub repo being updated via a manual instead of automated process, allowing for unclear periods of documentation lag and the potential of missing content to go unsynced. The causes look to be either:
  • This source repository has yet to catch up to the latest release(s), as @AWSRandall has mentioned as possibility and has been the case for most links referenced in this open issue. Source repo expected to lag behind in content by maybe 1 - 2 weeks.
  • This source repository never had the documentation pushed to it, not merely because it is lagging behind, but was missed at some point. This seems to currently be the case for AWS::DLM::LifecyclePolicy.Parameters and AWS::EFS::FileSystem.LifecyclePolicy, both part of a collection of updates referenced in release history from a June 13, 2019 release.

[ScriptAutomate]

11/11/2019 EDIT: Increased days of time in below table for latest 8.0.0 cfn spec file release.

10/31/2019 EDIT: Increased days of time where items are still broken by another 30 days.

Based on the history of this issue, and what is actually considered missing docs that aren't merely a result of the slight lag between source documentation syncs (as in, has the error existed for greater than 2 weeks, as mentioned by @AWSRandall ), these are the 14 missing or otherwise broken documentation errors spread across the three categories mentioned in the previous comment:

Resource	DaysBrokenAtLeast	Error
AWS::Transfer::User.SshPublicKey	94	MissingDocLinks
AWS::EC2::LaunchTemplate.CapacityReservationPreference	94	MissingDocLinks
AWS::AutoScaling::AutoScalingGroup.InstancesDistribution	94	BrokenDocLinks
AWS::AutoScaling::AutoScalingGroup.LaunchTemplate	94	BrokenDocLinks
AWS::AutoScaling::AutoScalingGroup.LaunchTemplateOverrides	94	BrokenDocLinks
AWS::AutoScaling::AutoScalingGroup.MixedInstancesPolicy	94	BrokenDocLinks
AWS::CodePipeline::CustomActionType.ArtifactDetails	80	BrokenDocLinks
AWS::CodePipeline::CustomActionType.ConfigurationProperties	80	BrokenDocLinks
AWS::CodePipeline::CustomActionType.Settings	80	BrokenDocLinks
AWS::OpsWorks::Instance.EbsBlockDevice	80	BrokenDocLinks
AWS::OpsWorks::Layer.VolumeConfiguration	80	BrokenDocLinks
AWS::DLM::LifecyclePolicy.Parameters	151	MissingFromAWSGitHubSourceRepo
AWS::EFS::FileSystem.LifecyclePolicy	151	MissingFromAWSGitHubSourceRepo
AWS::Glue::Trigger.NotificationProperty	45	MissingFromAWSGitHubSourceRepo

This table was generated from documentation-broken-links-detailed.json, located within my CFN auditing repository. That JSON includes potential fixes for most of the issues, and should be referenced when resolving problems along with the previous comment defining BrokenDocLinks, MissingDocLinks, and MissingFromAWSGitHubSourceRepo

[ScriptAutomate]

I will no longer be updating the table in the last comment. The most up-to-date documentation lookup errors can be found here, which will continue keeping results of AWS documentation audits for changes/errors/etc:

• ScriptAutomate/aws-cfn-resource-specs/documentation-lookup-errors.json

It includes a list of errors that will be related to this issue. As of today, at least 10 or 13 of the documentation errors have existed for anywhere between 100 and 210 days (broken documentation links, completely missing documentation, absence of documentation link in spec file, missing in GitHub source mirror, etc.).

Some additional, helpful information (but not fully updated/accurate because it's rather tedious and no longer managed), can be found here:

• ScriptAutomate/aws-cfn-resource-specs/documentation-broken-links-detailed.json

If there are any questions, feel free to ask.