Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add tests for the C tokenizer and expose it as a private module #27924

Merged
merged 5 commits into from
Aug 24, 2021
Merged

Add tests for the C tokenizer and expose it as a private module #27924

merged 5 commits into from
Aug 24, 2021

Conversation

pablogsal
Copy link
Member

No description provided.

@pablogsal pablogsal requested a review from a team as a code owner August 24, 2021 00:09
@pablogsal pablogsal force-pushed the test_c_tokenizer branch 7 times, most recently from 9d34dc1 to b11828c Compare August 24, 2021 12:08
isidentical
isidentical approved these changes Aug 24, 2021
View reviewed changes
Copy link
Member

@isidentical isidentical left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (a couple PEP 7 nits, and one question about the condition)

Lib/tokenize.py Outdated Show resolved Hide resolved
Python/Python-tokenize.c Outdated Show resolved Hide resolved
Python/Python-tokenize.c Outdated Show resolved Hide resolved
Python/Python-tokenize.c Outdated Show resolved Hide resolved
@isidentical
Copy link
Member

Running the buildbots (for refleaks, if there are any)

@isidentical isidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 24, 2021
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @isidentical for commit deee43b 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 24, 2021
@pablogsal
Copy link
Member Author 

❯ ./python -m test test_tokenize -R :
0:00:00 load avg: 0.52 Run tests sequentially
0:00:00 load avg: 0.52 [1/1] test_tokenize
beginning 9 repetitions
123456789
.........

== Tests result: SUCCESS ==

1 test OK.

Total duration: 12.7 sec
Tests result: SUCCESS

@pablogsal pablogsal merged commit a24676b into python:main Aug 24, 2021
@pablogsal pablogsal deleted the test_c_tokenizer branch August 24, 2021 16:50
ambv
ambv reviewed Aug 24, 2021
View reviewed changes
Python/Python-tokenize.c
Comment on lines +24 to +25
[clinic start generated code]*/
/*[clinic end generated code: output=da39a3ee5e6b4b0d input=96d98ee2fef7a8bc]*/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, check other modules that have classes created with the argument clinic. Maybe I am missing what you mean though 😅

Python/Python-tokenize.c
if (self == NULL) {
return NULL;
}
PyObject* filename = PyUnicode_FromString("<string>");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This returns a new reference, don't you have to decref it in line 53?

Copy link
Member Author

@pablogsal pablogsal Aug 24, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the tokenizer free function does it

Ah, I misread the comment. Yeah, we are missing a decref in the error path

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #27935

Python/Python-tokenize.c
{Py_tp_getattro, PyObject_GenericGetAttr},
{Py_tp_iter, PyObject_SelfIter},
{Py_tp_iternext, tokenizeriter_next},
{0, NULL},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are some of the initializers 8-char indented while others are 4-char indented?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hummm, I think is because I messed up with the format. All should be 4 char. Will correct this in another pR unless you manage to do it before

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #27935

Python/Python-tokenize.c
};

static PyModuleDef_Slot tokenizemodule_slots[] = {
{Py_mod_exec, tokenizemodule_exec},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Example of a 4-indented initializer.

@sunmy2019 sunmy2019 mentioned this pull request Apr 12, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ambv ambv ambv left review comments

@isidentical isidentical isidentical approved these changes

Assignees
No one assigned
Labels
skip issue skip news
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@pablogsal @isidentical @bedevere-bot @ambv @the-knights-who-say-ni