Repository navigation
feat(spider-py): Add boilerplate for spider-py; Add build and lint tasks for spider-py. #179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 96 commits
Commits
Show all changes
105 commits
Select commit
Hold shift + click to select a range
d2ac198
Update yscope-dev-utils
sitaowang1998 cc8a097
Use boost install task
sitaowang1998 7f68bdc
Update install task variable names
sitaowang1998 2394649
Set CMP0074 to NEW to find boost
sitaowang1998 6ff3e33
Add uv to install script
sitaowang1998 7ae8a78
Fix cpp-lint root paths
sitaowang1998 56502f9
Fix clang-tidy file pattern
sitaowang1998 6f6becf
Limit build parallelism
sitaowang1998 a74523d
Bug fix
sitaowang1998 2345d22
Merge branch 'dep-concurrency' into yscope-dev-utils
sitaowang1998 77ee494
Bug fix
sitaowang1998 38b86c8
Merge branch 'dep-concurrency' into yscope-dev-utils
sitaowang1998 3698339
Bug fix
sitaowang1998 369e9f1
Rename variables to mirror CLP core
sitaowang1998 90aa5a2
Rename variables to mirror clp core
sitaowang1998 1769c95
Merge branch 'dep-concurrency' of github.com:sitaowang1998/spider int…
sitaowang1998 edaa834
Merge branch 'dep-concurrency' into yscope-dev-utils
sitaowang1998 7faac8f
Revert "Merge branch 'dep-concurrency' of github.com:sitaowang1998/sp…
sitaowang1998 f494a90
Add comment for deps parallelism default value
sitaowang1998 eb01bb2
Merge branch 'dep-concurrency' into yscope-dev-utils
sitaowang1998 ff2fe1c
Update yscope-dev-utils
sitaowang1998 d476e42
Merge branch 'main' into yscope-dev-utils
sitaowang1998 850126d
Merge branch 'yscope-dev-utils' into python_lint
sitaowang1998 65841a0
Add latest python lint config files
sitaowang1998 8564fc2
Update ruff lint tasks
sitaowang1998 a6e7d29
Fix ruff lint
sitaowang1998 573b448
Fix ruff lint
sitaowang1998 6ad72c3
Bug fix
sitaowang1998 ba7c6e5
Fix ruff
sitaowang1998 209acb1
Fix ruff
sitaowang1998 406b514
Reformat files
sitaowang1998 66892f5
Remove .inc from cpp linting
sitaowang1998 80f0a10
Merge branch 'yscope-dev-utils' into python_lint
sitaowang1998 3186a42
Add mypy and merge lint and test requirements.txt
sitaowang1998 0925938
Fix mysql connection type
sitaowang1998 6936a9b
Fix socket name type
sitaowang1998 7407090
Fix return type from db cursor
sitaowang1998 6a95b24
Fix db cursor return type
sitaowang1998 f29dcba
Fix mypy import untyped
sitaowang1998 453f209
Fix mypy and Popen
sitaowang1998 d1011c7
Fix generator type hint
sitaowang1998 a7dc642
Fix mypy
sitaowang1998 a7d92c2
Fix ruff
sitaowang1998 1c6213b
Simply socket return types.
sitaowang1998 0f13d07
Merge branch 'main' into python_lint
sitaowang1998 14d6326
Merge branch 'python_lint' into mypy_lint
sitaowang1998 55a727a
Add tombi lint tasks
sitaowang1998 c98fea1
Rename cpp build tasks
sitaowang1998 ef64454
Merge branch 'main' into mypy_lint
sitaowang1998 09ba8ad
Merge branch 'main' into tombi
sitaowang1998 4e7237e
Add basic python structure
sitaowang1998 1f944d0
Fix code structure
sitaowang1998 68992fc
Fix ruff lint
sitaowang1998 1127b22
Extend lint tasks to python directory
sitaowang1998 4c51c91
Add python build tasks
sitaowang1998 56878b2
Merge branch 'main' into mypy_lint
sitaowang1998 bb428c9
Merge branch 'mypy_lint' into tombi
sitaowang1998 944bd5e
Merge branch 'tombi' into build-task
sitaowang1998 d75bc15
Merge branch 'build-task' into python_structure
sitaowang1998 4b7eca2
Merge branch 'main' into tombi
sitaowang1998 8ddbda7
Merge branch 'tombi' into build-task
sitaowang1998 0d998ec
Merge branch 'build-task' into python_structure
sitaowang1998 6004d64
Fix yaml lint error
sitaowang1998 bcab5db
Fix tombi lint
sitaowang1998 a50d25a
Remove wrong mypy config
sitaowang1998 fbebb96
Fix typo and format file
sitaowang1998 0322bd5
Use typed msgpack and remove mypy config for msgpack from pyproject
sitaowang1998 2a08155
Update uv lock
sitaowang1998 0b62f73
Merge branch 'main' into tombi
sitaowang1998 7384158
Merge branch 'main' into build-task
sitaowang1998 70ef90a
Merge branch 'main' into python_structure
sitaowang1998 b09515c
Increase min version of tombi
sitaowang1998 bad9675
Merge branch 'tombi' into build-task
sitaowang1998 e009ff4
Merge branch 'tombi' into python_structure
sitaowang1998 a0407a7
Merge branch 'main' into tombi
sitaowang1998 aa9a8de
Merge branch 'tombi' into build-task
sitaowang1998 3c7e794
Merge branch 'tombi' into python_structure
sitaowang1998 b4d6576
Add uv in README
sitaowang1998 23f31cd
Merge branch 'main' into build-task
sitaowang1998 0167ac0
Merge branch 'build-task' into python_structure
sitaowang1998 f447614
Fix redenduncy caused by merge
sitaowang1998 a84de83
Merge branch 'build-task' into python_structure
sitaowang1998 2d1b6fc
Merge branch 'main' into python_structure
sitaowang1998 061d101
Merge branch 'main' into python_structure
sitaowang1998 de149ee
Add integral types
sitaowang1998 d332dbe
Revert "Add integral types"
sitaowang1998 8c27b16
Restructure under src/spider
sitaowang1998 95b98c8
Merge branch 'main' into python_structure
sitaowang1998 427dd6b
Merge branch 'main' into python_structure
sitaowang1998 854cd11
Fix missing link
sitaowang1998 5565c9e
Merge branch 'python_structure' of github.com:sitaowang1998/spider in…
sitaowang1998 01dd0c6
Fix task executor path
sitaowang1998 e022404
Merge branch 'main' into python_structure
sitaowang1998 7753153
Merge branch 'main' into python_structure
sitaowang1998 526482c
Rename project
sitaowang1998 539738f
Rename project source root to match the new project name
sitaowang1998 2a36054
Bug fix
sitaowang1998 05f9c57
Merge branch 'main' into python_structure
sitaowang1998 2c96e13
Merge branch 'main' into python_structure
sitaowang1998 bc0c3b1
Add another level spider-py under python dir
sitaowang1998 af6da1c
Update python src dir
sitaowang1998 f38feb3
Split python lint tasks
sitaowang1998 9e61e88
Rename build tasks
sitaowang1998 eb31097
Add readme and fix pyproject.toml.
LinZhihao-723 9a05967
Formatting..
LinZhihao-723 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| 3.13 | ||
|
sitaowang1998 marked this conversation as resolved.
Resolved
|
||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we draft a simple README to include how to build and lint this project? |
Empty file.
|
sitaowang1998 marked this conversation as resolved.
Resolved
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| [project] | ||
| name = "spider-py" | ||
| version = "0.0.1" | ||
| description = "Spider is a distributed task execution framework" | ||
| readme = "README.md" | ||
| requires-python = ">=3.10" | ||
| dependencies = [ | ||
| "mariadb>=1.1.13", | ||
| "msgpack-types>=0.5.0", | ||
| ] | ||
|
sitaowang1998 marked this conversation as resolved.
Resolved
|
||
|
|
||
| [project.scripts] | ||
| spider_task_executor = "spider-py.task_executor.task_executor:main" | ||
|
|
||
| [dependency-groups] | ||
| dev = [ | ||
| "mypy>=1.12.0", | ||
| "ruff>=0.4.4", | ||
| ] | ||
|
|
||
| [build-system] | ||
| requires = ["hatchling>=1.18.0"] | ||
| build-backend = "hatchling.build" | ||
|
|
||
| [tool.mypy] | ||
| strict = true | ||
| # Additional output | ||
| pretty = true | ||
| show_error_code_links = true | ||
| show_error_context = true | ||
|
|
||
| [tool.ruff] | ||
| line-length = 100 | ||
|
|
||
| [tool.ruff.format] | ||
| docstring-code-format = true | ||
| docstring-code-line-length = 100 | ||
|
|
||
| [tool.ruff.lint] | ||
| select = ["ALL"] | ||
| extend-select = ["PT"] | ||
| ignore = [ | ||
| "COM812", # Redundant and conflicts with ruff format | ||
| "D203", # No blank line before docstrings (D211) | ||
| "D205", # Breaks if summary is larger than one line due to wrapping or if no summary exists | ||
| "D212", # Enforce docstring summary line on the next line after quotes (D213) | ||
| "D400", # First line of docstrings may not end in period | ||
| "D401", # Docstrings should be written in present tense (not imperative) | ||
| "D415", # First line of docstrings may not end in a period, question mark, or exclamation point | ||
| "FA102", # Allow use of PEP 604 union in type annotations | ||
| "FBT", # Allow bool positional parameters since other value positions are allowed | ||
| "FIX002", # Allow todo statements | ||
| "PERF401", # Allow for loops when creating lists | ||
| "PERF403", # Allow for loops when creating dicts | ||
| "S311", # Allow usage of `random` package | ||
| "SIM102", # Allow collapsible if statements for readability | ||
| "TD002", # Author unnecessary for todo statement | ||
| "TD003", # Issue link unnecessary for todo statement | ||
| "UP015", # Explicit open modes are helpful | ||
| ] | ||
| isort.order-by-type = false | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| """Spider package root.""" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| """Spider python client.""" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| """Spider core package.""" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| """Task module for Spider.""" | ||
|
|
||
|
|
||
| class Task: | ||
| """Represents a task in Spider.""" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| """Python task executor for running Python task function.""" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| """Executes a Spider Python task.""" | ||
|
|
||
|
|
||
| def main() -> None: | ||
| """Main function to execute the task.""" | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() | ||
|
sitaowang1998 marked this conversation as resolved.
Resolved
|
||
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two comments for this task:
spider-py? We should dropbuildas it's already underbuildtask namespace.$ROOT/python/spider-py/build? (assuming u apply the comment for addingspider-pyunderpython) In this way, we can usetask build:spider-pyinside$ROOT/python/spider-py/build, treating it as a stand-alone project with the build results directly located inside the project.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
clp.