track line and column #59

bakkot · 2020-04-22T00:22:05Z

cc @rbuckton; this changes location tracking to include line and column numbers, not just offsets.

lib/parser.js

lib/ecmarkdown.d.ts

lib/parser.js

bakkot · 2020-05-01T05:52:53Z

@rbuckton I addressed your comments; do you want to take another look?

rbuckton · 2020-05-01T21:19:27Z

Out of curiosity, what does ecmarkdown consider to be a "line terminator"? Not all languages/editors agree ([1], [2]), so we should make sure ecmarkup, ecmarkdown, and grammarkdown agree.

[1] https://twitter.com/SeaRyanC/status/1253037372263952387
[2] microsoft/TypeScript#38078 (comment)

bakkot · 2020-05-01T21:22:20Z

Out of curiosity, what does ecmarkdown consider to be a "line terminator"?

'\n'.

rbuckton

A few minor suggestions, but you're welcome to ignore them.

rbuckton · 2020-05-01T21:25:24Z

src/parser.ts

@@ -365,31 +370,32 @@ export class Parser {

  pushPos() {
    if (this._posStack) {
-      this._posStack.push(this.getPos());
+      this._posStack.push(this.getPos() as Position);


NOTE: TS has a postfix-! syntax as a simple cast to remove null/undefined, but just as with the as cast, both are type-level only with no runtime enforcement.

This is also possibly unsound if somehow you end up with a defined this._posStack but an undefined node.location. Perhaps a better approach would be ensure the value is defined:

Suggested change

this._posStack.push(this.getPos() as Position);

const pos = this.getPos();

if (pos) {

this._posStack.push(pos);

}

Alternatively, you could change the body in this way (if you are using TS 3.7 or later):

pushPos() { const pos = this.getPos(); if (pos) this._posStack?.push(pos); }

rbuckton · 2020-05-01T21:26:18Z

src/parser.ts

    }
  }

  popPos() {
-    return this._posStack ? this._posStack.pop() : -1;
+    return this._posStack ? this._posStack.pop() : undefined;


Suggested change

return this._posStack ? this._posStack.pop() : undefined;

return this._posStack?.pop();

rbuckton · 2020-05-01T21:29:19Z

src/parser.ts

-    return this._posStack && tok.location ? tok.location.pos : -1;
+  // TODO rename to getStart ?
+  getPos(node: Node | Token = this._t.peek()) {
+    return this._posStack && node.location ? node.location.start : undefined;


Suggested change

return this._posStack && node.location ? node.location.start : undefined;

return this._posStack && node.location?.start;

rbuckton · 2020-05-01T21:29:35Z

src/parser.ts

  }

  getEnd(node: Node | Token) {
-    return this._posStack && node.location ? node.location.end : -1;
+    return this._posStack && node.location ? node.location.end : undefined;


Suggested change

return this._posStack && node.location ? node.location.end : undefined;

return this._posStack && node.location?.end;

rbuckton · 2020-05-01T21:32:58Z

src/parser.ts

+      let actualStart: Position = start ?? (this.popPos() as Position);
+      let actualEnd: Position =
+        end ??
+        (this._t.previous === undefined


I think this could also be:

let actualEnd: Position = end ?? { line: 1, column: 1, offset: 0, ...this._t.previous?.location?.end };

rbuckton · 2020-05-01T21:34:14Z

src/tokenizer.ts

@@ -425,12 +429,20 @@ export class Tokenizer {
    }
  }

-  enqueueLookahead(tok: Token, pos: number) {
+  getLocation() {


Suggested change

getLocation() {

getLocation(): Position {

Useful if you decide to rename or add a new member.

rbuckton · 2020-05-01T21:36:12Z

src/parser.ts

@@ -306,7 +310,8 @@ export class Parser {
      this._t.next();
    }

-    return this.finish({ name: 'text', contents });
+    let endLoc = !this._posStack || lastRealTok === null ? undefined : lastRealTok.location!.end;


Suggested change

let endLoc = !this._posStack || lastRealTok === null ? undefined : lastRealTok.location!.end;

let endLoc = this._posStack && lastRealTok?.location?.end;

lastRealTok?.location?.end

The ! is fine after location because it really is always defined if it's reached - this._posStack being truthy implies every token has a location property.

... Well, that's what I thought, but using lastRealTok?.location!.end breaks tests. Which is surprising to me; I had thought the ! was just a type assertion, but it appears to be changing the semantics of the generated code. I guess TS is parsing it as the end of the "chain"? That's really surprising to me.

... Oh, I see this has already been discovered and fixed upstream. I guess I'll just // @ts-ignore this line until that's released.

rbuckton · 2020-05-01T21:46:38Z

Out of curiosity, what does ecmarkdown consider to be a "line terminator"?

'\n'.

For parsing or emit? Only supporting \n for parse isn't Windows friendly. It should at least support CR, LF, and CRLF, but should probably support the same line terminators HTML uses, or possibly any additional line terminators recognized by markdown implementations such as commonmark (given that ecmarkdown claims to be a markdown-like syntax). I think I need to go over what line terminators are supported by grammarkdown as well in the near future.

bakkot · 2020-05-01T21:55:28Z

For parsing or emit?

For both.

I mostly would like to maintain ecmarkdown as a tool specifically for processing parts of the ECMAScript specification. In that context, the goal is not to accept all reasonable inputs, but rather to accept only a very constrained set of inputs and reject all others, so that there are as few ways of writing the spec as possible. Concretely, if a contributor to ECMA-262 mixed CRLF line endings into an algorithm, it would be rejected by me in my capacitor as editor, which means it should ideally be rejected by the tooling so that the contributor can find that out for themselves.

(That said, we should certainly at least give better errors than the generic parse error or malformed output you'd get now.)

Is there a particular reason to support inputs which would be rejected as parts of ECMA-262?

given that ecmarkdown claims to be a markdown-like syntax

Well, I'm also planning on changing that, since it seems like overkill for what this tool is actually used for.

bakkot · 2020-05-03T00:15:54Z

@rbuckton I adopted most of your suggestions and filed #67 to track line endings; thank you!

bakkot force-pushed the more-position branch 3 times, most recently from 0149581 to 393cd89 Compare April 22, 2020 01:00

rbuckton suggested changes Apr 22, 2020

View reviewed changes

lib/parser.js Outdated Show resolved Hide resolved

lib/parser.js Outdated Show resolved Hide resolved

lib/ecmarkdown.d.ts Outdated Show resolved Hide resolved

lib/parser.js Outdated Show resolved Hide resolved

bakkot added 2 commits April 30, 2020 15:36

track line and column

55c1111

fix types

30db71f

bakkot force-pushed the more-position branch from 393cd89 to 30db71f Compare April 30, 2020 23:08

clone location

05c6955

bakkot mentioned this pull request Apr 30, 2020

Add visitor #66

Merged

bakkot added 2 commits April 30, 2020 22:33

lint

72e3379

use helper

3d99ccb

rbuckton approved these changes May 1, 2020

View reviewed changes

bakkot added 2 commits May 2, 2020 17:10

address comments

b192bbc

lint

ee0b99d

bakkot merged commit 747edc1 into master May 3, 2020

bakkot deleted the more-position branch May 3, 2020 00:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

track line and column #59

track line and column #59

bakkot commented Apr 22, 2020

bakkot commented May 1, 2020

rbuckton commented May 1, 2020

bakkot commented May 1, 2020

rbuckton left a comment

rbuckton May 1, 2020

rbuckton May 1, 2020

rbuckton May 1, 2020

rbuckton May 1, 2020

rbuckton May 1, 2020

rbuckton May 1, 2020

rbuckton May 1, 2020

rbuckton May 1, 2020

bakkot May 2, 2020

rbuckton commented May 1, 2020

bakkot commented May 1, 2020

bakkot commented May 3, 2020

-      this._posStack.push(this.getPos() as Position);
+      const pos = this.getPos();
+      if (pos) {
+          this._posStack.push(pos);
+      }

	return this._posStack ? this._posStack.pop() : undefined;
	return this._posStack?.pop();

	return this._posStack && node.location ? node.location.start : undefined;
	return this._posStack && node.location?.start;

	let endLoc = !this._posStack \|\| lastRealTok === null ? undefined : lastRealTok.location!.end;
	let endLoc = this._posStack && lastRealTok?.location?.end;

track line and column #59

track line and column #59

Conversation

bakkot commented Apr 22, 2020

bakkot commented May 1, 2020

rbuckton commented May 1, 2020

bakkot commented May 1, 2020

rbuckton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rbuckton commented May 1, 2020

bakkot commented May 1, 2020

bakkot commented May 3, 2020