Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Properly decode this latin-encoding thing #9

Open
1 of 2 tasks
edgarogh opened this issue Jul 16, 2022 · 1 comment
Open
1 of 2 tasks

Properly decode this latin-encoding thing #9

edgarogh opened this issue Jul 16, 2022 · 1 comment

Comments

@edgarogh
Copy link
Collaborator

edgarogh commented Jul 16, 2022

2 solutions :

  1. We ditch strings altogether. That's radical but not necessarily impossible, it mostly depends on rowan's and logos' ability to work with byte slices. That's why Paths exist in Rust in the first place : they may not be UTF-8. Clearly the best solution if it's doable.
  2. We convert files before parsing them. That's unfortunately a bit slower and IMO a little more undesirable as we're clearly transforming user input just because we like it better. That also prevents us from pretending to be 100% compatible with lustres's spec. And this also requires asking the user which encoding their files are in, which is a new CLI parameter/function argument

Edit: byte slice support

  • logos: should be good in theory, but I don't know if regexes work well with non-string input
  • rowan: unfortunately, the GreenNode::new constructor takes an &str. We can try to open an issue, but if this doesn't work, it's always possible to cheat. We could for instance encode the byte slice as base64 and prepend a discriminator char. Dirty, but still better than option 2 IMO as we're not imposing our own rules.
@edgarogh
Copy link
Collaborator Author

Also this wouldn't require a full rewrite of anything once we're using nom (#8), as our parser won't deal with the source code anymore at all, it's all hidden in the rowan_nom module.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant