Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump Traceur to 0.0.82 #80

Open
wants to merge 21 commits into
base: master
Choose a base branch
from
Open

Bump Traceur to 0.0.82 #80

wants to merge 21 commits into from

Conversation

jmm
Copy link
Collaborator

@jmm jmm commented Jan 30, 2015

This is a work in progress, but I wanted to get the discussion going. It appears to now be necessary to pass a sourceRoot param to traceur to get correct paths in source maps. So I think it's time to make the options to es6ify.configure() more general or turn this into a plugin. In this pull request I've implemented the former approach, turning the filePattern param into opts (which can currently take filePattern and sourceRoot properties). It falls back to the previous behavior if a regex is passed. (I duck typed it based on ignoreCase -- cool?) I rearranged some of the internal options passing based on that change and the addition of sourceRoot to the mix.

* Generalize es6ify() argument from filePattern => opts, with
  fallback to previous behavior if a regex is passed. Now filePattern
  should be specified as {filePattern: ...}.

* Support opts.sourceRoot param to pass to traceur. It now appears
  to be neccessary to specifify this to traceur to get correct paths
  in source map.
@jmm
Copy link
Collaborator Author

jmm commented Jan 30, 2015

I forgot to mention, it may be desirable to default sourceRoot to browserify's basedir, which would be facilitated by making this a plugin, although I don't think that's advertised as part of the API (which I think it ought to be, for plugins to utilize.)

@thlorenz
Copy link
Owner

Nice work @jmm!
Does a6aafc6 break backwards compat since you rearranged some arg order there?

WRT plugin, would totally love a PR that makes this work as plugin while keeping it working as transform as well. Something similar was done(among other fixes) to proxyquireify a while ago.

@jmm
Copy link
Collaborator Author

jmm commented Jan 30, 2015

Thanks @thlorenz!

Does a6aafc6 break backwards compat since you rearranged some arg order there?

It shouldn't. As long as compile.js's compileFile() is only called from index.js it should be all good. If it does, then some new tests must be needed to catch it. Is there any expectation that compile.js should be usable on its own outside of index.js depending on it?

WRT plugin, would totally love a PR that makes this work as plugin while keeping it working as transform as well. Something similar was done(among other fixes) to proxyquireify a while ago.

I might be able to work on that. Thanks for the link, I'll take a look. I've done some work along those lines (turning a transform into a plugin) on other projects. How do you feel about reading browserify's basedir if it's not a documented part of the API though?

@jmm
Copy link
Collaborator Author

jmm commented Jan 30, 2015

I pushed a couple more commits that are oriented toward getting the correct file name in the source map (e.g. not exposing absolute local paths). They are hasty and rough and definitely need more review. There'll probably need to be separate tests for with and without sourceRoot specified.

function es6ify(filePattern) {
filePattern = filePattern || /\.js$/;
function es6ify(opts) {
opts || (opts = {});
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use a proper if or ?:. Also best to follow ES6 and do if (opts === undefined) { opts = {}; } instead of using any falsy value.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about just opts = opts || {}?
Unless I"m missing something could opts ever be anything but an object?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not ES6ey enough, only undefined should trigger defaults :)

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, let's agree to disagree on this one then. I feel the verbosity isn't worth the es6iness.
However doesn't matter enough to me to fight you on that :)

In either case we should make this consistent across the repo, no matter which way we pick.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also best to follow ES6 and do if (opts === undefined) { opts = {}; } instead of using any falsy value.
Not ES6ey enough, only undefined should trigger defaults :)

I don't see what it accomplishes -- the new test will still allow a bunch of nonsensical values, and convert es6ify(null) (which admittedly doesn't make sense) to a TypeError -- but it's up to you guys. Also, if making this codebase ES6ey is an imperative, why not actually author it in ES6 and transpile it?

Please use a proper if or ?:

You're referring to readability preferences not functionality, right? A ?: in this situation seems like the worst option to me.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, TypeError sounds great, and yes, readability.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok.

@domenic
Copy link
Collaborator

domenic commented Feb 1, 2015

Left some comments. Overall looks good, although I am curious about taking this opportunity to convert to a plugin. (I don't see a need for it to be a transform if it is a plugin, personally.) In that case we should also automatically include the runtime, as it's basically nonsensical to use Traceur without it.

It sounds like the source map filename stuff is a mess so I'll trust you to be doing the right thing there. Wish Traceur just did the right thing automatically. Maybe they should try copying 6to5.

@jmm
Copy link
Collaborator Author

jmm commented Feb 2, 2015

@domenic Thanks for the feedback. I'll fix anything that needs fixing.

Overall looks good, although I am curious about taking this opportunity to convert to a plugin. (I don't see a need for it to be a transform if it is a plugin, personally.)

I think you're saying you're in favor and you'd prefer it to be only a plugin from now on (breaking backward compatibility by removing the option to use it as a transform)? The only reason I'd give for making it a plugin currently is to access browserify's basedir as a default for sourceRoot, though I must caution that it's not a documented part of the API. If I'm not mistaken both you and @thlorenz are maintainers on browserify though, right?

In that case we should also automatically include the runtime, as it's basically nonsensical to use Traceur without it.

I'm currently using it on a project purely to transpile ES6 module syntax, for which I don't need or want the runtime.

It sounds like the source map filename stuff is a mess so I'll trust you to be doing the right thing there. Wish Traceur just did the right thing automatically. Maybe they should try copying 6to5.

Well I think the addition of the sourceRoot param is probably an improvement. I'll be looking into the filename stuff more and report back here on what I find out.

@jmm
Copy link
Collaborator Author

jmm commented Feb 3, 2015

@domenic
Copy link
Collaborator

domenic commented Feb 3, 2015

For a test? The latter seems better.

@jmm
Copy link
Collaborator Author

jmm commented Feb 4, 2015

For a test? The latter seems better.

Yes, both snippets are in the test file I linked to.

@jmm
Copy link
Collaborator Author

jmm commented Feb 4, 2015

Here's the rundown on how the paths work out:

  • es6ify 1.6.0 / traceur 0.0.79:
    Absolute local file system paths are exposed in sources. (es6ify / traceur expose the absolute local path in file also, but that ends up being overwritten by browserify).
  • es6ify / traceur 0.0.82
    By default (and at best) fairly useless basenames are populated in sources. If there are multiple files (in different directories, of course) with the same basename, then they'll overwrite each other (e.g. if you have src/a.js that requires src/sub/a.js that requires src/sub/sub/a.js you'll wind up with sources: ["a.js"] and sourcesContent for one of the files. (Ditto about es6ify / traceur exposing the local path in file but browserify overwriting.)
  • es6ify after these changes are done / traceur 0.0.82
    I have to get more feedback about how traceur itself works before this can be properly resolved. There's a new sourceRoot argument to the compiler in the mix, but I'm not sure it's being applied properly. See Need relative paths in source map google/traceur-compiler#1676. Presumably some combination of sourceName and sourceRoot arguments to traceur will eventually allow generating the desired source map.

In all cases the resulting source map output by browserify has file: "generated.js", sourceRoot: "".

@jmm jmm mentioned this pull request Feb 4, 2015
@jmm
Copy link
Collaborator Author

jmm commented Feb 4, 2015

Created #81 for further discussion of conversion to a plugin.

@domenic
Copy link
Collaborator

domenic commented Feb 4, 2015

Regarding docme and jsdoc, that's all @thlorenz, I am not a jsdoc fan and just try to copy what I see around me :P.

The basedir scheme sounds good, although I wonder if a name involving source maps might be more transparent. 6to5ify uses "sourceMapRelative".

@jmm
Copy link
Collaborator Author

jmm commented Feb 4, 2015

Regarding docme and jsdoc, that's all @thlorenz, I am not a jsdoc fan and just try to copy what I see around me :P.

Ok, thanks. Hopefully @thlorenz can point me in the right direction.

The basedir scheme sounds good, although I wonder if a name involving source maps might be more transparent. 6to5ify uses "sourceMapRelative".

Like I said, until the last minute I had it as sourceRoot, but I no longer believe that makes sense. Then I thought of basedir, because browserify's basedir might be a sensible default (haven't thought that all the way through yet). I think another name could be ok, but I don't find sourceMapRelative very descriptive (does that come from anywhere else, a spec or anything?). I think a name more indicative of source maps but more descriptive would be sourcesBase or something like that.

@domenic
Copy link
Collaborator

domenic commented Feb 4, 2015

sourceMapBase or sourceMapBaseDir?

@jmm
Copy link
Collaborator Author

jmm commented Feb 4, 2015

I sort of feel like prefixing any source map-related opts with sourceMap is going to do more harm than good unless the names are made really verbose, like sourceMapSourcesBaseDir. I think that's what the documentation is for, to explain that an option like sourcesRoot relates to the source map. If it's really necessary to prefix them, perhaps sm would be sufficient as a hint? E.g. smBaseDir? I think if everything would need to be prefixed with sourceMap I'd be tempted to make sourceMap a hash that can contain properties like sourcesRoot, e.g. b.plugin(es6ify, {sourceMap: {sourcesRoot: '...'}})

@thlorenz
Copy link
Owner

thlorenz commented Feb 5, 2015

Possibly @guzart wants to join in WRT to API choices as he was working on a major API update himself in #25.
We should also revisit that PR to get some API ideas from there.

@thlorenz
Copy link
Owner

thlorenz commented Feb 5, 2015

@jmm added you as collab, so once we agree on everything you could squash and forward merge into master.
Also now you can close issues (I suppose some questions will come up related to the API changes once they are in). Hoping for you to triage some of those.
Thanks for the good work so far.

@jmm
Copy link
Collaborator Author

jmm commented Feb 5, 2015

I'm going to leave some notes here for future reference.

The current pull request brings traceur up to 0.0.82 with proper sources paths in the output (when basedir [or whatever it ends up being called] is supplied), but the way that's achieved is hackneyed because of traceur's misapplication of its sourceRoot input param to alter the values that end up in sources. It looks like traceur has come around to stop doing that. Once that happens the solution should probably be to pass the exact desired sources output to traceur's compile function (the sourceName arg). (Which might've been possible for es6ify to do before traceur added the misapplied sourceRoot param.) So probably the values passed as sourceName and outputName will be generated with something like path.resolve(opts.basedir, file). It may or may not be necessary to pass a sourceRoot argument to traceur (perhaps with value false). Unless someone can think of a use case for populating the source map generated by traceur with a sourceRoot value, I think the ideal thing is to have it omitted (either by passing nothing or false to traceur depending how that shakes out), because:

  • sourcesContent is populated.
  • Browserify overwrites it anyway.

So if that all works out there probably shouldn't be separate tests anymore for with / without the sourceRoot argument to traceur, but maybe there should be tests for with / without a basedir argument to es6ify.

@jmm
Copy link
Collaborator Author

jmm commented Feb 5, 2015

Possibly @guzart wants to join in WRT to API choices as he was working on a major API update himself in #25.
We should also revisit that PR to get some API ideas from there./

Ok. Sorry @guzart, didn't realize you'd already done so much work in this direction. I skimmed #25 (but haven't looked at any of the commits) and it sounds similar to what I've been working on in #81. I think I'll clean up what I have so far a little but and push it so everyone can take a look and if @guzart wants to take the reins again can cherry pick anything good from there. One thought I do have right off the bat is that I think addTraceurRuntime is unnecessarily verbose. I currently have includeRuntime, which is barely better (maybe addRuntime is better than both), but I don't think there'll be any confusion about which runtime it is.

@jmm
Copy link
Collaborator Author

jmm commented Feb 5, 2015

@jmm added you as collab, so once we agree on everything you could squash and forward merge into master.

Ok, thanks. I kind of think the history is useful, but it's up to you. If you want me to squash, all into one commit?

Also now you can close issues (I suppose some questions will come up related to the API changes once they are in). Hoping for you to triage some of those.

Ok, I'll see what I can do.

Thanks for the good work so far.

Thanks.

@thlorenz
Copy link
Owner

thlorenz commented Feb 5, 2015

@jmm squash as you see fit.
A lot of commits here were good to drive the discussion, but I believe there should be up to 5 things that were added/changed.
So squashing to <=5 commits seems sensible.

@domenic
Copy link
Collaborator

domenic commented Feb 5, 2015

I think addTraceurRuntime is unnecessarily verbose. I currently have includeRuntime, which is barely better (maybe addRuntime is better than both), but I don't think there'll be any confusion about which runtime it is.

Since runtimes will be on by default it should be false by default, so e.g. omitRuntime: true or noRuntime: true would be the way to get rid of the runtime.

@jmm
Copy link
Collaborator Author

jmm commented Feb 5, 2015

I think a double negative is more confusing (do not do it? yes) than a straightforward: do it? -- yes or no -- that defaults to yes. But, it's up to you.

@domenic
Copy link
Collaborator

domenic commented Feb 5, 2015

Yeah I feel pretty strongly that boolean options should have a default value of false. If you can think of a better phrasing that's cool (e.g. maybe "omit" is better than "no").

@jmm
Copy link
Collaborator Author

jmm commented Feb 5, 2015

I think no matter what you name it it's more confusing than the straightforward version. I'll let you know if I have a suggestion for a default: false name.

@jmm
Copy link
Collaborator Author

jmm commented Feb 5, 2015

Possibly @guzart wants to join in WRT to API choices as he was working on a major API update himself in #25.
We should also revisit that PR to get some API ideas from there.

I pushed more commits to the branch I linked in #81 to show what I have so far.

@jmm
Copy link
Collaborator Author

jmm commented Feb 5, 2015

FYI, traceur 0.0.83 is out now and partially fixes the problems with sourceRoot and sources. See also google/traceur-compiler/issues/1695 google/traceur-compiler/issues/1696

@domenic
Copy link
Collaborator

domenic commented Feb 9, 2015

So there's been a lot of back-and-forth. At some point we should merge and do a release :). What are the unresolved issues remaining? (Apologizes if you're waiting on me or @thlorenz to respond to something that got lost in the shuffle...)

@guzart
Copy link
Collaborator

guzart commented Feb 9, 2015

unfortunately :( there isn't much I can bring in to the conversation, main points have already been discussed:

  • include traceur runtime by default
  • forward traceur options via traceurOverrides
  • and simply allow to pass options directly via the main method instead of using es6ify.configure

@jmm
Copy link
Collaborator Author

jmm commented Feb 9, 2015

Ok, I think these are the unresolved issues:

For this pull request:

  • What to call the param currently known as basedir. See discussion in this thread. I favor something like basedir, sourcesBase, or smBasedir (or smBaseDir). If source map-related options need to be segregated, I could also see passing a sourceMap hash containing options like basedir or sourcesBase.
  • Finer points of the JSdoc comments and regeneration of the docs. Maybe @thlorenz can point me in the right direction.
  • Make sure all relevant documentation is updated.
  • Squashing the commits.

Here's a summary of issues that have been discussed in or are related to this thread that relate to conversion to a plugin and should be hashed out in #25 or #81:

  • Whether or not to default the basedir param (whatever it ends up being called) to browserify's basedir.
  • What to call the option for controlling inclusion of the runtime.
  • How to include the runtime.
  • Whether to eliminate compileFile export.

@jmm
Copy link
Collaborator Author

jmm commented Feb 9, 2015

@guzart Do you want to jump back in and make the actual commits that are going to get merged?

There's also the question of how to include the runtime. It looks like you were taking a different approach. I've posted more about that in #81.

@domenic
Copy link
Collaborator

domenic commented Feb 9, 2015

@jmm

What to call the param currently known as basedir

Can you explain why sourcesBase makes sense to you? What does "sources" even mean?

My reasoning for something more explicit like "sourceMapBaseDir" is that (a) I don't mind if people type extra characters; (b) it's very clear what it does; (c) it explicitly lets you know this is only related to source maps and not to all "sources" or to es6ify in general.

@jmm
Copy link
Collaborator Author

jmm commented Feb 9, 2015

Can you explain why sourcesBase makes sense to you? What does "sources" even mean?

It means the sources array in the source map. The parameter is to be used to do this to generate the value that will be populated in sources:

path.relative(basedir, file)

Where file is the absolute path browserify passes to es6ify (or any transform).

My reasoning for something more explicit like "sourceMapBaseDir" is that (a) I don't mind if people type extra characters; (b) it's very clear what it does; (c) it explicitly lets you know this is only related to source maps and not to all "sources" or to es6ify in general.

A) In my opinion, all else being equal (including descriptiveness), shorter is preferable.

B) I'm not so sure it is. It's not used in any other context (like the source map spec or mozilla/source-map), is it?, and there are a number of paths / URLs related to source maps: sources, sourceRoot, file, sourceMappingURL, sourceURL (, others?) that may or may not be influenced by this param. I wouldn't count on users knowing what it does without reading the documentation. I actually do want to apply it to the file property of the source map as well, although that'll be overwritten by browserify in normal usage. I'm looking into what's up with the //# sourceURL= parts -- it should probably be applied to those as well.

C) I guess. There's only a handful of parameters for es6ify and the user isn't going to know what they are or how to use them in the first place without looking at the documentation.

@adbl
Copy link

adbl commented Mar 30, 2015

We are experience these issues, is this PR going in or are you waiting for further traceur updates? including traceur 0.0.83 which includes other related changes?

@jmm is there any workaround for the time being to get a working (in chrome) source map?

@jmm
Copy link
Collaborator Author

jmm commented Mar 30, 2015

@adbl

We are experience these issues, is this PR going in or are you waiting for further traceur updates? including traceur 0.0.83 which includes other related changes?

I personally am waiting for some combination of traceur maturing its API with respect to source maps...
google/traceur-compiler#1695
google/traceur-compiler#1715
google/traceur-compiler#1748
...

and continuation of discussion here about the unresolved issues in this thread.

is there any workaround for the time being to get a working (in chrome) source map?

Here's the workaround I'm using to deal with google/traceur-compiler#1748 / chromium # 459682 (which was bogusly merged with another issue):

var rs = require('readable-stream');
var stream = new rs.Transform;

stream.data = '';

stream._transform = function (chunk, enc, cb) {
  this.data += chunk;
  cb();
};

stream._flush = function (cb) {
  this.data = this.data.replace(/\/\/# sourceURL=\S+/mg, '');
  this.push(this.data);
  cb();
};

return b
  .transform(es6ify)
  .bundle()
  .pipe(stream)

And here's the workaround I'm using to correct the paths in the source map:

gulp.task('build:fix-source-maps', ['build:bundle'], function (done) {
  if (! use_source_map) return;
  var map_file = './dist/rsrc/app.js.map';

  fs.readFile(map_file, function (err, src) {
    if (err) throw err;
    var map = JSON.parse(src);
    map.sources = map.sources.map(function (source) {
      var
        base_dir = [__dirname, 'src'],
        ret = source;

      [
        base_dir,
        base_dir.slice(0, -1),
        base_dir.slice(-1)
      ].every(function (base_dir) {
        base_dir = path.join.apply(path, base_dir);
        if (ret.indexOf(base_dir + '/') === 0) {
          ret = path.relative(base_dir, ret);
          return false;
        }
        return true;
      });

      return ret;
    });

    fs.writeFile(map_file, JSON.stringify(map, null, 2), done);
  });
});

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants