Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incremental static regeneration #1028

Merged
merged 61 commits into from
May 19, 2021
Merged
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
31f313c
wip: initial proof-of-concept incremental static regeneration
kirkness Apr 28, 2021
35cf5ff
fix: dont ts build the regeneration handler
kirkness Apr 28, 2021
65ba637
fix: dynamic paths should regenerate
kirkness Apr 28, 2021
0ee2f55
add Expires header to the regenerated S3 objects
kirkness Apr 29, 2021
6f1ce93
Merge branch 'master' into feature/incremental-static-regeneration
kirkness Apr 29, 2021
fc8bd38
use expires header rather than last-modified and a small code tidy
kirkness Apr 29, 2021
8a0d5ff
revalidation relies on the expires header
kirkness Apr 29, 2021
67cda68
fix fallback isr fallback behaviour
kirkness Apr 30, 2021
16fe62b
Merge branch 'master' into feature/incremental-static-regeneration
kirkness Apr 30, 2021
26ed898
update cdk snapshot
kirkness Apr 30, 2021
ef7d03b
remove sqs and use lambda async invocation
kirkness May 2, 2021
f4548b3
adds e2e test and serverless infrastructure
kirkness May 2, 2021
b52d762
remove uneeded isNaN check
kirkness May 2, 2021
3dc480f
remove prerenderManifest variable from handler
kirkness May 2, 2021
127698b
add e2e test for isr page with getStaticPaths
kirkness May 2, 2021
7eb5bb4
revert serverless change
kirkness May 2, 2021
a3eb635
add tests to regeneration handler
kirkness May 4, 2021
30d0e51
add tests for getStaticRegenerationResponse
kirkness May 4, 2021
c03105f
create the regeneration lambd ain the same region as the bucket
kirkness May 4, 2021
6adf890
revery async lambda changes and use SQS
kirkness May 4, 2021
9dbfb27
add sqs creation to serverless components
kirkness May 4, 2021
f0650f2
fix conflicts
kirkness May 4, 2021
57b403b
update snapshot andd mocks
kirkness May 4, 2021
43f064a
fix regeneration tests with sqs event
kirkness May 5, 2021
db31e75
Merge branch 'master' into feature/incremental-static-regeneration
kirkness May 5, 2021
6f0f45c
minor test fixes
kirkness May 5, 2021
26c8449
Merge branch 'feature/incremental-static-regeneration' of https://git…
kirkness May 5, 2021
f397f50
fix all tests
kirkness May 5, 2021
631d8c0
add cache-control header tests to default-handler
kirkness May 5, 2021
5fc9807
add tests for triggerStaticRegeneration
kirkness May 5, 2021
b8cb485
remove removeHeader call
kirkness May 5, 2021
99ec48c
remove only test
kirkness May 5, 2021
93a0826
add tests for sqs component
kirkness May 5, 2021
10f9f67
add tests to unhappy-paths in triggerStaticRegeneration
kirkness May 5, 2021
cb5ad0b
add extra test cases to sqs component
kirkness May 5, 2021
f9806d2
add case for sqs delete
kirkness May 5, 2021
3552751
check assertions against sdk api calls
kirkness May 5, 2021
09f739f
Merge branch 'master' into feature/incremental-static-regeneration
kirkness May 5, 2021
41202f3
regenerate yarn lock files
kirkness May 8, 2021
d9417b8
Merge branch 'feature/incremental-static-regeneration' of https://git…
kirkness May 8, 2021
1f757d1
chore: resolve conflicts
kirkness May 11, 2021
fbe017e
docs: add ISR related deployment permissions
kirkness May 12, 2021
027e9c1
fix: update all yarn lock files in all e2e packages
kirkness May 12, 2021
855fc88
fix: include ready checks in e2e tests for isr
kirkness May 13, 2021
436b584
fix: redirect test in trailing slash e2e tests
kirkness May 13, 2021
90ce486
fix: include isr e2e path checks in test-utils
kirkness May 13, 2021
d546e02
Merge branch 'master' into feature/incremental-static-regeneration
dphang May 13, 2021
a3c3e36
fix: dont deploy a queue if we dont need one
kirkness May 14, 2021
738335e
fix: merge conflicts
kirkness May 14, 2021
bf8038a
fix: revert normalise uri call
kirkness May 16, 2021
e49eac4
fix: update snapshot
kirkness May 16, 2021
b13cb05
fix: update regeneration fallback test
kirkness May 16, 2021
aaa0916
fix: convert secs to millis
kirkness May 16, 2021
f104d19
fix: revert 404 case
kirkness May 16, 2021
bf6ceed
fix: revert 308 case
kirkness May 16, 2021
200f195
fix: revert 404 case
kirkness May 16, 2021
e2cae31
fix: set cache folder for more reliability(??)
kirkness May 17, 2021
f4e5218
fix: 404 redirect test, update post cypress bump
kirkness May 17, 2021
2cde26b
fix: resolve conflits
kirkness May 18, 2021
3c37c3f
fix: merge conflict
kirkness May 18, 2021
7586260
Merge branch 'master' into feature/incremental-static-regeneration
dphang May 19, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -774,7 +774,7 @@ Please see the [contributing](./CONTRIBUTING.md) guide.

This project exists thanks to all the people who contribute. [[Contribute](CONTRIBUTING.md)].
<a href="https://github.com/serverless-nextjs/serverless-next.js/graphs/contributors">
<img src="https://contrib.rocks/image?repo=serverless-nextjs/serverless-next.js" />
<img src="https://contrib.rocks/image?repo=serverless-nextjs/serverless-next.js" />
</a>

Made with [contributors-img](https://contrib.rocks).
Expand Down
41 changes: 39 additions & 2 deletions packages/deprecated/serverless-plugin/__mocks__/aws-sdk.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
const promisify = (mockFunction) => {
const mockPromise = jest.fn(() => Promise.resolve());
const promisify = (mockFunction, mockResolvedValue) => {
const mockPromise = jest.fn(() => Promise.resolve(mockResolvedValue));
mockFunction.mockReturnValue({
promise: mockPromise
});
Expand Down Expand Up @@ -71,6 +71,23 @@ const {
} = promisify(jest.fn());
MockSTS.prototype.getCallerIdentity = mockGetCallerIdentity;

const MockSQS = jest.fn();
const {
mockFunction: mockGetQueueAttributes,
mockPromise: mockGetQueueAttributesPromise
} = promisify(jest.fn());
const {
mockFunction: mockCreateQueue,
mockPromise: mockCreateQueuePromise
} = promisify(jest.fn());
const {
mockFunction: mockDeleteQueue,
mockPromise: mockDeleteQueuePromise
} = promisify(jest.fn());
MockSQS.prototype.createQueue = mockCreateQueue;
MockSQS.prototype.deleteQueue = mockDeleteQueue;
MockSQS.prototype.getQueueAttributes = mockGetQueueAttributes;

const MockAPIGateway = function () {};
const {
mockFunction: mockGetRestApis,
Expand All @@ -86,6 +103,9 @@ const mockMetadataRequest = jest
.mockImplementation((path, cb) => cb(null, {}));
MockMetadataService.prototype.request = mockMetadataRequest;

const mockListEventSourceMappingsPromise = jest.fn();
const mockCreateEventSourceMappingPromise = jest.fn();

module.exports = {
EnvironmentCredentials: MockEnvironmentCredentials,
S3: jest.fn(() => {
Expand All @@ -97,10 +117,27 @@ module.exports = {
CloudFormation: MockCloudFormation,
CloudWatchLogs: MockCloudWatchLogs,
STS: MockSTS,
SQS: MockSQS,
Lambda: jest.fn().mockImplementation(() => ({
listEventSourceMappings: jest.fn().mockReturnValue({
promise: mockListEventSourceMappingsPromise
}),
createEventSourceMapping: jest.fn().mockReturnValue({
promise: mockCreateEventSourceMappingPromise
})
})),
APIGateway: MockAPIGateway,
SharedIniFileCredentials: MockSharedIniFileCredentials,
MetadataService: MockMetadataService,

mockDeleteQueue,
mockDeleteQueuePromise,
mockCreateQueue,
mockCreateQueuePromise,
mockGetQueueAttributes,
mockGetQueueAttributesPromise,
mockListEventSourceMappingsPromise,
mockCreateEventSourceMappingPromise,
mockDescribeStacks,
mockDescribeStacksPromise,
mockCreateStack,
Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/next-app-dynamic-routes/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
"aws-sdk": "2.758.0",
"babel-plugin-istanbul": "6.0.0",
"babel-plugin-transform-class-properties": "6.24.1",
"cypress": "5.1.0",
"cypress": "7.2.0",
kirkness marked this conversation as resolved.
Show resolved Hide resolved
"istanbul-lib-coverage": "3.0.0",
"nyc": "15.1.0",
"ts-node": "9.0.0",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
"aws-sdk": "2.758.0",
"babel-plugin-istanbul": "6.0.0",
"babel-plugin-transform-class-properties": "6.24.1",
"cypress": "5.1.0",
"cypress": "7.2.0",
"istanbul-lib-coverage": "3.0.0",
"nyc": "15.1.0",
"ts-node": "9.0.0",
Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/next-app-with-base-path/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
"aws-sdk": "2.758.0",
"babel-plugin-istanbul": "6.0.0",
"babel-plugin-transform-class-properties": "6.24.1",
"cypress": "5.1.0",
"cypress": "7.2.0",
"istanbul-lib-coverage": "3.0.0",
"nyc": "15.1.0",
"ts-node": "9.0.0",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
"aws-sdk": "2.758.0",
"babel-plugin-istanbul": "6.0.0",
"babel-plugin-transform-class-properties": "6.24.1",
"cypress": "5.1.0",
"cypress": "7.2.0",
"istanbul-lib-coverage": "3.0.0",
"nyc": "15.1.0",
"ts-node": "9.0.0",
Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/next-app-with-locales/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
"aws-sdk": "2.758.0",
"babel-plugin-istanbul": "6.0.0",
"babel-plugin-transform-class-properties": "6.24.1",
"cypress": "5.1.0",
"cypress": "7.2.0",
"istanbul-lib-coverage": "3.0.0",
"nyc": "15.1.0",
"ts-node": "9.0.0",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
"aws-sdk": "2.758.0",
"babel-plugin-istanbul": "6.0.0",
"babel-plugin-transform-class-properties": "6.24.1",
"cypress": "5.1.0",
"cypress": "7.2.0",
"istanbul-lib-coverage": "3.0.0",
"nyc": "15.1.0",
"ts-node": "9.0.0",
Expand Down
1 change: 0 additions & 1 deletion packages/e2e-tests/next-app/cypress.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
"supportFile": "cypress/support/index.ts",
"responseTimeout": 15000,
"requestTimeout": 15000,
"experimentalNetworkStubbing": true,
"experimentalFetchPolyfill": true,
"retries": 4
}
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ describe("Redirects Tests", () => {
{
path: "//example.com/",
expectedPath: "/example.com",
expectedStatus: 404
expectedStatus: 308
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was not passing after the update to cypress - the 308 is returned initially and the 404 is the status code of the subsequent request.

}
].forEach(({ path, expectedPath, expectedStatus }) => {
it(`does not redirect page ${path}`, () => {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
describe("ISR Tests", () => {
before(() => {
cy.ensureAllRoutesNotErrored();
});

// We don't want this test to retry as the cache will not be expired between
// runs
describe("SSG page", { retries: 0 }, () => {
[
"/revalidated-ssg-page",
// Pre-rendered ISR page
"/revalidated-ssg-pages/101",
// Blocking dynamic generated page
"/revalidated-ssg-pages/105"
].forEach((path) => {
it(`serves the cached re-rendered page "${path}" after 2 reloads`, () => {
// The initial load will have expired in the cache
cy.ensureRouteNotCached(path);
cy.visit(path);
cy.location("pathname").should("eq", path);

cy.get("[data-cy=date-text]")
.invoke("text")
.then((text1) => {
// When we reload again the page still should not be cached as this
// should be the first time its being served from the origin
cy.ensureRouteNotCached(path);
cy.reload();
cy.get("[data-cy=date-text]")
.invoke("text")
.then((text2) => {
// Check that the date text has changed since the initial page
// load
expect(text1).not.to.be.eq(text2);
// The new date should be greater than the original
expect(new Date(text2).getTime()).to.be.greaterThan(
new Date(text1).getTime()
);
// Make sure the next load is cached
cy.ensureRouteCached(path);
// Be sure that the regeneration has run and uploaded the file
cy.wait(2000);
cy.reload();
});
});

// Wait for the cache to expire after the 10s
cy.wait(8000);
cy.ensureRouteNotCached(path);
cy.reload();
});
});
});
});
2 changes: 1 addition & 1 deletion packages/e2e-tests/next-app/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
"aws-sdk": "2.758.0",
"babel-plugin-istanbul": "6.0.0",
"babel-plugin-transform-class-properties": "6.24.1",
"cypress": "5.1.0",
"cypress": "7.2.0",
"istanbul-lib-coverage": "3.0.0",
"nyc": "15.1.0",
"ts-node": "9.0.0",
Expand Down
27 changes: 27 additions & 0 deletions packages/e2e-tests/next-app/pages/revalidated-ssg-page.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
import React from "react";
import { GetStaticPropsResult } from "next";

type SSGPageProps = {
date: string;
};

export default function RevalidatedSSGPage(props: SSGPageProps): JSX.Element {
return (
<React.Fragment>
<div>
<p data-cy="date-text">{props.date}</p>
</div>
</React.Fragment>
);
}

export async function getStaticProps(): Promise<
GetStaticPropsResult<SSGPageProps>
> {
return {
revalidate: 10,
props: {
date: new Date().toJSON()
}
};
}
62 changes: 62 additions & 0 deletions packages/e2e-tests/next-app/pages/revalidated-ssg-pages/[id].tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
import React from "react";
import {
GetStaticPaths,
GetStaticPropsContext,
GetStaticPropsResult
} from "next";

const sampleUserData = [
{ id: 101, name: "Alice" },
{ id: 102, name: "Bob" },
{ id: 103, name: "Caroline" },
{ id: 104, name: "Dave" }
];

const newUser = { id: 105, name: "Henry" };

type SSGPageProps = {
date: string;
user?: typeof sampleUserData[number];
};

export default function RevalidatedSSGPage(props: SSGPageProps): JSX.Element {
return (
<React.Fragment>
<div>
<p data-cy="date-text">{props.date}</p>
<p data-cy="user-id-text">{props.user?.id ?? "No user found"}</p>
</div>
</React.Fragment>
);
}

export const getStaticPaths: GetStaticPaths = async () => {
// To simulate new data becoming available in a data source we only make the
// `newUser` available when rendering on AWS, rather than at build time.
const runningOnAws = !!process.env.AWS_LAMBDA_FUNCTION_NAME;
const paths = sampleUserData.map((user) => ({
params: { id: user.id.toString() }
}));

if (runningOnAws) {
paths.push({ params: { id: newUser.id.toString() } });
}

return { paths, fallback: "blocking" };
};

export async function getStaticProps(
context: GetStaticPropsContext<{ id: string }>
): Promise<GetStaticPropsResult<SSGPageProps>> {
const users = [...sampleUserData, newUser];
const user = users.find(
({ id }) => context.params?.id.toString() === id.toString()
);
return {
revalidate: 10,
props: {
date: new Date().toJSON(),
user
}
};
}
Loading