Skip to content

feat: compress debug symbols #1760

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sirasistant merged 8 commits into from
Aug 23, 2023
Merged

feat: compress debug symbols #1760

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
7214882
feat: compress debug information for lighter abis
sirasistant Aug 23, 2023
ea2ce05
chore: update contracts
sirasistant Aug 23, 2023
119207c
Update yarn-project/foundation/src/abi/abi.ts
sirasistant Aug 23, 2023
ee469dc
Update yarn-project/foundation/src/abi/abi.ts
sirasistant Aug 23, 2023
c1384a3
chore: use aztec version for compiling
sirasistant Aug 23, 2023
7cc3ad7
Merge branch 'master' into arv/compress_debug
sirasistant Aug 23, 2023
c8c09c1
Merge branch 'master' into arv/compress_debug
sirasistant Aug 23, 2023
fea761b
Merge branch 'master' into arv/compress_debug
sirasistant Aug 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26,078 changes: 51 additions & 26,027 deletions yarn-project/aztec.js/src/abis/ecdsa_account_contract.json
View file
Open in desktop

Large diffs are not rendered by default.

20,742 changes: 50 additions & 20,692 deletions yarn-project/aztec.js/src/abis/schnorr_account_contract.json
View file
Open in desktop

Large diffs are not rendered by default.

19,835 changes: 18 additions & 19,817 deletions yarn-project/aztec.js/src/abis/schnorr_single_key_account_contract.json
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions yarn-project/foundation/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@
"levelup": "^5.1.1",
"lodash.clonedeepwith": "^4.5.0",
"memdown": "^6.1.1",
"pako": "^2.1.0",
"sha3": "^2.1.4"
},
"devDependencies": {
Expand All @@ -86,6 +87,7 @@
"@types/lodash.clonedeepwith": "^4.5.7",
"@types/memdown": "^3.0.1",
"@types/node": "^18.7.23",
"@types/pako": "^2.0.0",
"@types/supertest": "^2.0.12",
"@typescript-eslint/eslint-plugin": "^6.2.1",
"@typescript-eslint/parser": "^6.2.1",
Expand Down
10 changes: 7 additions & 3 deletions yarn-project/foundation/src/abi/abi.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import { inflate } from 'pako';

/**
* A named type.
*/
Expand Down Expand Up @@ -202,9 +204,9 @@ export type DebugFileMap = Record<
*/
export interface DebugMetadata {
/**
* The debug information for each function.
* The debug information, serialized as JSON.
Comment thread
sirasistant marked this conversation as resolved.
Outdated
*/
debugSymbols: DebugInfo[];
debugSymbols: string[];
/**
* The map of file ID to the source code and path of the file.
*/
Expand Down Expand Up @@ -254,7 +256,9 @@ export interface FunctionDebugMetadata {
export function getFunctionDebugMetadata(abi: ContractAbi, functionName: string): FunctionDebugMetadata | undefined {
const functionIndex = abi.functions.findIndex(f => f.name === functionName);
if (abi.debug && functionIndex !== -1) {
const debugSymbols = abi.debug.debugSymbols[functionIndex];

@ludamad ludamad Aug 23, 2023

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this only run if there's a need for debug data? (I ask as if so, the fact that json is somewhat inefficient doesn't matter much)

@sirasistant sirasistant Aug 23, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's run if you execute a function that you have stored debug metadata for (the idea is that only developers will use ABIs with debug metadata) and only for the function being executed

@sirasistant sirasistant Aug 23, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we see that it hurts performance we can change it to a structure that parses it lazily when accessed but I don't see it being an issue for now

const debugSymbols = JSON.parse(
inflate(Buffer.from(abi.debug.debugSymbols[functionIndex], 'base64'), { to: 'string' }),
);
const files = abi.debug.fileMap;
return {
debugSymbols,
Expand Down
2 changes: 2 additions & 0 deletions yarn-project/noir-compiler/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
"lodash.compact": "^3.0.1",
"lodash.times": "^4.3.2",
"lodash.upperfirst": "^4.3.1",
"pako": "^2.1.0",
"toml": "^3.0.0",
"tslib": "^2.4.0"
},
Expand All @@ -61,6 +62,7 @@
"@types/lodash.times": "^4.3.7",
"@types/lodash.upperfirst": "^4.3.7",
"@types/node": "^18.7.23",
"@types/pako": "^2.0.0",
"jest": "^29.5.0",
"ts-jest": "^29.1.0",
"ts-node": "^10.9.1",
Expand Down
4 changes: 3 additions & 1 deletion yarn-project/noir-compiler/src/contract-interface-gen/abi.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
import { ContractAbi, DebugMetadata, FunctionAbi, FunctionType } from '@aztec/foundation/abi';

import { deflate } from 'pako';

import { mockVerificationKey } from '../mocked_keys.js';
import { NoirCompilationArtifacts, NoirFunctionEntry } from '../noir_artifact.js';

Expand Down Expand Up @@ -45,7 +47,7 @@ export function generateAztecAbi({ contract, debug }: NoirCompilationArtifacts):
parsedDebug = {
debugSymbols: sortedFunctions.map(fn => {
const originalIndex = originalFunctions.indexOf(fn);
return debug.debug_symbols[originalIndex];
return Buffer.from(deflate(JSON.stringify(debug.debug_symbols[originalIndex]))).toString('base64');
}),
fileMap: debug.file_map,
};
Expand Down
18 changes: 18 additions & 0 deletions yarn-project/yarn.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,7 @@ __metadata:
"@types/lodash.clonedeepwith": ^4.5.7
"@types/memdown": ^3.0.1
"@types/node": ^18.7.23
"@types/pako": ^2.0.0
"@types/supertest": ^2.0.12
"@typescript-eslint/eslint-plugin": ^6.2.1
"@typescript-eslint/parser": ^6.2.1
Expand All @@ -458,6 +459,7 @@ __metadata:
levelup: ^5.1.1
lodash.clonedeepwith: ^4.5.0
memdown: ^6.1.1
pako: ^2.1.0
prettier: ^2.7.1
sha3: ^2.1.4
supertest: ^6.3.3
Expand Down Expand Up @@ -537,6 +539,7 @@ __metadata:
"@types/lodash.times": ^4.3.7
"@types/lodash.upperfirst": ^4.3.7
"@types/node": ^18.7.23
"@types/pako": ^2.0.0
base64-js: ^1.5.1
commander: ^9.0.0
fs-extra: ^11.1.1
Expand All @@ -546,6 +549,7 @@ __metadata:
lodash.compact: ^3.0.1
lodash.times: ^4.3.2
lodash.upperfirst: ^4.3.1
pako: ^2.1.0
toml: ^3.0.0
ts-jest: ^29.1.0
ts-node: ^10.9.1
Expand Down Expand Up @@ -3440,6 +3444,13 @@ __metadata:
languageName: node
linkType: hard

"@types/pako@npm:^2.0.0":
version: 2.0.0
resolution: "@types/pako@npm:2.0.0"
checksum: 50240a036b5e6acabbf36ac4dca93ec9e619241f0404da8d401cdb427bec3029833324b8a04c4b1ae2ecbc33422fdec31dbf9f43653d9d07cafb82ace78dfccd
languageName: node
linkType: hard

"@types/qs@npm:*":
version: 6.9.7
resolution: "@types/qs@npm:6.9.7"
Expand Down Expand Up @@ -9863,6 +9874,13 @@ __metadata:
languageName: node
linkType: hard

"pako@npm:^2.1.0":
version: 2.1.0
resolution: "pako@npm:2.1.0"
checksum: 71666548644c9a4d056bcaba849ca6fd7242c6cf1af0646d3346f3079a1c7f4a66ffec6f7369ee0dc88f61926c10d6ab05da3e1fca44b83551839e89edd75a3e
languageName: node
linkType: hard

"parent-module@npm:^1.0.0":
version: 1.0.1
resolution: "parent-module@npm:1.0.1"
Expand Down