Skip to content
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- logger control of uvicorn server. can be completely silenced.
- Server: logs now printed as debug messages.
- Language switch: logs now printed as debug messages.
- Tests:
- refactored tabs-case dependency flow to `pytest-dependency`:
- `TestREvoDesignPlugin` -> `TestREvoDesignPlugin_TabPrepare` -> downstream tab/action case classes.
- dropped active `pytest.mark.order` coupling in tabs bootstrap cases and enabled dependency-aware ordering with `--order-dependencies`.
- Test env/bootstrap:
- test dependencies and `make prepare-test` now include `pytest-dependency`.
- Server (PSSM_GREMLIN):
- Public/private dashboard behavior is now configurable with `PUBLIC_DASHBOARD` (`false` by default).
- Task visibility and API access are scoped to the authenticated upload owner when `PUBLIC_DASHBOARD=false`.
Expand All @@ -78,6 +84,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- notify box and decide box now works under subthreads.
- Tests:
- `test_pm_dialog_extras_panel_expand_collapse` now waites for the dialog to fully expand before asserting.
- `test_utils` archive extraction now covers zip/tar path traversal rejection.
- Security hardening:
- archive extraction in `tools.utils.extract_archive` now validates member paths and rejects traversal entries before writing files.
- Monaco editor tarball setup now reuses the hardened archive extractor instead of raw `tar.extractall`.
- Server (PSSM_GREMLIN):
- task-id/content digest md5 calls now explicitly use `usedforsecurity=False` because they are non-cryptographic identifiers.
- Server (PSSM_GREMLIN):
- Docker daemon permission failure handling for non-root runtime users in server/worker execution paths.
- `run_stage` ORM/schema mismatch that caused sqlite update/compile failures and prevented task execution in server-image integration tests.
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ translate:
stage=release bash tools/translate.sh

prepare-test:
python -m pip install pytest pytest-cov pytest-order coverage -q --no-cache-dir
python -m pip install pytest pytest-cov pytest-order pytest-dependency coverage -q --no-cache-dir
python -m pip install "celery[redis]==5.3.4" docker==7.1.0 Flask==3.1.3 Flask-HTTPAuth==4.8.0 SQLAlchemy==2.0.46 -q --no-cache-dir

# unit test
Expand Down
220 changes: 220 additions & 0 deletions docs/dev_trick/codex_deepsource.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,220 @@
# Codex DeepSource Fix Playbook

This document records the practical approach used to process DeepSource issues in this repository with minimal complexity and high safety.

## Goal

Fix a large DeepSource backlog by prioritizing the most dangerous and complex findings first, while keeping behavior stable and tests green.

## Principles

1. Risk first, not count first.
2. Root-cause patches, not one-off local silencing.
3. Small safe diffs over broad refactors.
4. Every risky fix must have verification.
5. Keep changelog and docs updated in the same PR.

## Inputs Required

1. DeepSource issues URL.
2. Browser-like user agent for issue fetching.
3. Local runtime/test environment details (`conda` env, test commands, CI constraints).

## Access Pattern For DeepSource

Use a regular browser user agent when fetching issue pages or APIs.

```bash
curl -sS -L -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36" \
"https://app.deepsource.com/gh/YaoYinYing/REvoDesign/issues" \
-o /tmp/deepsource_issues.html
```

If static HTML is insufficient, use the same UA for authenticated GraphQL/API calls (with csrf/cookies) and save raw payloads for traceability.

## Issue Classification Model

Classify each finding on two dimensions: danger and complexity.

### Danger Levels

1. `D0 Critical`: Security, data corruption, remote execution, credential/path leakage, unsafe deserialization.
2. `D1 High`: Runtime crashes, concurrency hazards, deadlocks, silent wrong results.
3. `D2 Medium`: Reliability/perf regressions with user-visible impact.
4. `D3 Low`: Readability/style/maintainability only.

### Complexity Levels

1. `C0 Low`: Local one-file change, obvious behavior, easy regression test.
2. `C1 Medium`: Multi-file behavior but stable boundaries.
3. `C2 High`: Cross-module flow, threading/process/network interactions.
4. `C3 Very High`: Architectural shifts, migration-level changes.

## Prioritization Rule

Work in this order:

1. `D0/C0-C2`
2. `D1/C0-C2`
3. `D0/D1` items with `C3` only when isolated safely
4. `D2` batches
5. `D3` cleanup

Within the same bucket, prioritize by blast radius:

1. Core runtime modules (`src/REvoDesign/...` hot paths)
2. Shared tools/utilities
3. Optional paths and edge flows

## Batch Strategy

Use small batches (5 to 10 issues) grouped by root cause family:

1. Serialization and file safety
2. Subprocess/network/IO robustness
3. Qt/thread interaction safety
4. Path and temp-file handling
5. Test determinism and ordering

This avoids mixing unrelated risk and simplifies rollback.

## Fix Workflow (Repeatable)

1. Fetch and snapshot issue list.
2. Normalize and deduplicate findings by root cause.
3. Assign `danger` and `complexity`.
4. Select next batch by priority.
5. Patch with minimal code movement.
6. Add or adjust targeted tests.
7. Run `make clean` before test validation.
8. Run focused `kw-test` first, then broader suite if needed.
9. Update changelog.
10. Record unresolved items and blockers.

## Typical Fix Patterns Used In This Repo

### 1) Unsafe Data Loading

Preferred approach:

1. Replace unsafe loaders with constrained loaders.
2. Validate input type and expected schema early.
3. Fail closed with explicit, actionable errors.

Avoid:

1. Broad `except Exception` without context.
2. Silent fallback that hides corruption.

### 2) Subprocess and External Tools

Preferred approach:

1. Use explicit argument lists (no implicit shell parsing unless required).
2. Set timeout and capture stderr/stdout.
3. Return structured errors for UI/logging.
4. Validate executable existence before launch.

Avoid:

1. Fire-and-forget calls in critical paths.
2. Weak error propagation.

### 3) Paths, Files, and Temporary Artifacts

Preferred approach:

1. Normalize and validate paths before use.
2. Create parent dirs deterministically.
3. Use atomic write patterns where practical.
4. Guard cleanup operations carefully.

Avoid:

1. Trusting user-provided file names blindly.
2. Destructive deletes without explicit path checks.

### 4) Thread/UI Safety (Qt + Worker Threads)

Preferred approach:

1. Keep UI updates on main thread.
2. Use centralized worker orchestration.
3. Register and clean up worker state deterministically.

Avoid:

1. Cross-thread UI updates.
2. Orphan worker/process state.

### 5) Test Determinism and Ordering

Preferred approach:

1. Express logical prerequisites with explicit dependencies.
2. Keep order assumptions declarative.
3. Ensure required plugins are part of test dependencies.

Recent example:

1. Refactored tabs dependency chain to `pytest-dependency`.
2. Added `pytest-dependency` to test extras and `make prepare-test`.
3. Enabled `--order-dependencies`.

## Verification Protocol

Always verify in two layers:

1. Fast targeted checks
2. Broader regression checks

### Commands

```bash
make clean
conda run -n REvoDesignLatestDev make kw-test PYTEST_KW='single keywords'
conda run -n REvoDesignLatestDev make kw-test PYTEST_KW='"keywordA or keywordB"'
```

For dependency-chain checks, include anchor tests and downstream tests in the keyword expression.

## Documentation Checklist For Each Batch

1. What rule/finding was fixed.
2. Why it was dangerous.
3. What changed and why this is minimal.
4. Which tests validate the change.
5. Any known limitations or follow-up tasks.

## Suggested Tracking Table

Use this table format in PR descriptions or internal notes:

| DeepSource ID | Rule | Danger | Complexity | Files | Fix summary | Validation |
|---|---|---|---|---|---|---|
| example-id | PY-XXXX | D1 | C1 | `src/...` | safer error path | `kw-test ...` |

## Anti-Patterns To Avoid

1. Batch-fixing unrelated modules in one commit.
2. Silencing warnings without code-level mitigation.
3. Skipping `make clean` before validating flaky areas.
4. Reordering tests to hide dependency problems.
5. Editing test expectations without validating runtime behavior.

## Definition Of Done (DeepSource Batch)

A batch is done only if all are true:

1. Selected findings are fixed or explicitly deferred with reason.
2. No new regressions in targeted tests.
3. Changelog updated.
4. Rationale documented for non-obvious changes.
5. Diff remains minimal and reviewable.

## Notes For Future Codex Runs

1. Start from highest danger findings, not oldest findings.
2. Keep a local snapshot of issue payloads to prevent paging/context loss.
3. Prefer deterministic tests and explicit dependency wiring for UI case chains.
4. If network is restricted, document the blocker and still finish code/test/docs work that can run locally.
2 changes: 1 addition & 1 deletion playground/plotly_qt.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

import plotly.graph_objects as go
from PyQt5.QtCore import QObject, QUrl, pyqtSlot
from PyQt5.QtWebEngineWidgets import QWebEngineView
from PyQt5.QtWebEngineWidgets import QWebEngineProfile, QWebEngineView
from PyQt5.QtWidgets import QApplication, QMainWindow


Expand Down
3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,7 @@ test = [
"pytest-xdist>=3.6.1",
"pytest<=8.3.3",
"pytest-order",
"pytest-dependency",
"pytest-emoji",
"pytest-github-actions-annotate-failures",
"shellcheck-py==0.10.0.1",
Expand Down Expand Up @@ -260,6 +261,7 @@ executionEnvironments = [
]

[tool.pytest.ini_options]
addopts = "--order-dependencies"
pythonpath = [
"src"
]
Expand All @@ -274,6 +276,7 @@ markers = [
"unit: fast offline tests",
"serial: mark as serial tests that can not be run in parallel with xdist",
"very_slow: marks tests as very slow that should be run only in a independent run",
"dependency: marks tests with explicit inter-test dependencies",
]
filterwarnings = [
"ignore::Bio.PDB.PDBExceptions.PDBConstructionWarning",
Expand Down
25 changes: 25 additions & 0 deletions server/.env.example
Original file line number Diff line number Diff line change
@@ -1,23 +1,48 @@
# Copy this file to server/.env.production for production use.

## How the images will be named
SERVER_IMAGE=revodesign-pssm-gremlin-server-non-root
RUNNER_IMAGE=revodesign-pssm-gremlin-non-root

## In which directory the server will run
SERVER_DIR=/Users/yyy/Documents/protein_design/REvoDesign/playground/server_test
LOG_DIR=/Users/yyy/Documents/protein_design/REvoDesign/playground/server_test/revodesign/server/logs

## Databases for MSA searching
DB_UNIREF30=/Users/yyy/Documents/protein_design/REvoDesign/playground/miniuc/uc30/miniuc30
DB_UNIREF90=/Users/yyy/Documents/protein_design/REvoDesign/playground/miniuc/uc90/uniref90

## Users for accessing the server via basic authentication
USERS_FILE=/Users/yyy/Documents/protein_design/REvoDesign/server/pssm_gremlin/users.template.txt
Comment on lines 8 to 16

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Developer-specific absolute paths committed in the example file leak local filesystem structure (including username) and render the template unusable for any other user.

Replace every concrete path with a clearly labelled placeholder:

📝 Proposed fix
-SERVER_DIR=/Users/yyy/Documents/protein_design/REvoDesign/playground/server_test
-LOG_DIR=/Users/yyy/Documents/protein_design/REvoDesign/playground/server_test/revodesign/server/logs
+SERVER_DIR=/srv/revodesign/workdir
+LOG_DIR=/srv/revodesign/logs

 ## Databases for MSA searching
-DB_UNIREF30=/Users/yyy/Documents/protein_design/REvoDesign/playground/miniuc/uc30/miniuc30
-DB_UNIREF90=/Users/yyy/Documents/protein_design/REvoDesign/playground/miniuc/uc90/uniref90
+DB_UNIREF30=/data/uniref30/uniref30
+DB_UNIREF90=/data/uniref90/uniref90

 ## Users for accessing the server via basic authentication
-USERS_FILE=/Users/yyy/Documents/protein_design/REvoDesign/server/pssm_gremlin/users.template.txt
+USERS_FILE=/srv/revodesign/users.txt
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
SERVER_DIR=/Users/yyy/Documents/protein_design/REvoDesign/playground/server_test
LOG_DIR=/Users/yyy/Documents/protein_design/REvoDesign/playground/server_test/revodesign/server/logs
## Databases for MSA searching
DB_UNIREF30=/Users/yyy/Documents/protein_design/REvoDesign/playground/miniuc/uc30/miniuc30
DB_UNIREF90=/Users/yyy/Documents/protein_design/REvoDesign/playground/miniuc/uc90/uniref90
## Users for accessing the server via basic authentication
USERS_FILE=/Users/yyy/Documents/protein_design/REvoDesign/server/pssm_gremlin/users.template.txt
SERVER_DIR=/srv/revodesign/workdir
LOG_DIR=/srv/revodesign/logs
## Databases for MSA searching
DB_UNIREF30=/data/uniref30/uniref30
DB_UNIREF90=/data/uniref90/uniref90
## Users for accessing the server via basic authentication
USERS_FILE=/srv/revodesign/users.txt
🧰 Tools
🪛 dotenv-linter (4.0.0)

[warning] 9-9: [UnorderedKey] The LOG_DIR key should go before the SERVER_DIR key

(UnorderedKey)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@server/.env.example` around lines 8 - 16, The .env example contains
developer-specific absolute paths; replace the concrete values for SERVER_DIR,
LOG_DIR, DB_UNIREF30, DB_UNIREF90 and USERS_FILE with clearly labeled
placeholders (e.g., REPLACE_WITH_SERVER_DIR, REPLACE_WITH_LOG_DIR,
REPLACE_WITH_DB_UNIREF30, REPLACE_WITH_DB_UNIREF90, REPLACE_WITH_USERS_FILE or
<PROJECT_ROOT>/...) so the template is generic and does not leak local
filesystem or usernames; update the variable values in server/.env.example
accordingly and keep the variable names unchanged.


## redis settings
REDIS_URL=redis://redis:6379/0
BROKER_URL=redis://redis:6379/0
RESULT_BACKEND=redis://redis:6379/0

# Runner settings
## Maximum number to run the server: NPROC * WORKER_CONCURRENCY
NPROC=4

## Gunicorn settings
GUNICORN_WORKERS=2

## Celery settings
WORKER_CONCURRENCY=2

## Runner user at local machine
RUNNER_UID=1000
RUNNER_GID=1000
DOCKER_GID=998
RUNNER_USERNAME=revodesign
RUNNER_GROUP=revodesign_appgroup

## Server settings
DOMAIN=0.0.0.0
PORT=8080

## Whether to enable the dashboard user isolation
PUBLIC_DASHBOARD=false

## Admin users, comma separated
ADMIN_USERS=admin
22 changes: 11 additions & 11 deletions server/README.legacy.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The GREMLIN server stack now runs entirely inside Docker. Redis, Celery, Gunicor
```
The `server/run/restart_pssm_flask.sh` helper runs these commands for you.
3. Copy `server/.env.example` to `server/.env` and edit the values. Every absolute path listed there (server directory, SQLite DB file, databases, `users.txt`) must exist on the host. The compose file bind-mounts those directories/files into the containers under the same paths, so the application code sees the exact same values that you configure.
4. Ensure `users.txt` contains at least one `username:password` pair and is referenced by `PSSM_GREMLIN_USERS_FILE`.
4. Ensure `users.txt` contains at least one `username:password` pair and is referenced by `USERS_FILE`.
5. Initialize `server/.env` and Docker socket group settings:
```bash
bash server/run/restart_pssm_flask.sh setup
Expand Down Expand Up @@ -56,16 +56,16 @@ Key options are controlled from `server/.env`:

| Variable | Purpose |
| --- | --- |
| `PSSM_GREMLIN_SERVER_DIR` | Host directory where uploads, states, and results are stored. Mounted into the containers at the same path and created automatically if missing. |
| `PSSM_GREMLIN_LOG_DIR` | Host directory that stores persistent Gunicorn and Celery logs. Bind-mounted into the containers at the same path. |
| `PSSM_GREMLIN_DB_PATH` | Absolute path to the SQLite job-tracking database file. Create the file or its parent directory on the host; Compose bind-mounts the file so the host retains ownership and backups. |
| `PSSM_GREMLIN_DB_UNIREF30`, `PSSM_GREMLIN_DB_UNIREF90` | Absolute paths/prefixes to the sequence databases. They are mounted read-only into both containers and passed to the runner as `-U`/`-u`. |
| `PSSM_GREMLIN_USERS_FILE` | Path to the HTTP Basic Auth credentials file. |
| `PSSM_GREMLIN_RUNNER_UID`, `PSSM_GREMLIN_RUNNER_GID` | Required UID/GID pair for the GREMLIN runner container. Both must point to a dedicated non-root account; the server refuses to start if they are missing or set to `root` to avoid root-owned artifacts. |
| `SERVER_DIR` | Host directory where uploads, states, and results are stored. Mounted into the containers at the same path and created automatically if missing. |
| `LOG_DIR` | Host directory that stores persistent Gunicorn and Celery logs. Bind-mounted into the containers at the same path. |
| `DB_PATH` | Absolute path to the SQLite job-tracking database file. Create the file or its parent directory on the host; Compose bind-mounts the file so the host retains ownership and backups. |
| `DB_UNIREF30`, `DB_UNIREF90` | Absolute paths/prefixes to the sequence databases. They are mounted read-only into both containers and passed to the runner as `-U`/`-u`. |
| `USERS_FILE` | Path to the HTTP Basic Auth credentials file. |
| `RUNNER_UID`, `RUNNER_GID` | Required UID/GID pair for the GREMLIN runner container. Both must point to a dedicated non-root account; the server refuses to start if they are missing or set to `root` to avoid root-owned artifacts. |
| `DOCKER_GID` | Supplementary group id injected into `web`/`worker` so they can access `/var/run/docker.sock` (for example `$(stat -Lc '%g' /var/run/docker.sock)` on Linux). |
| `PSSM_GREMLIN_NPROC`, `PSSM_GREMLIN_WORKER_CONCURRENCY`, `PSSM_GREMLIN_GUNICORN_WORKERS` | Performance knobs for the runner, Celery worker, and Gunicorn respectively. |
| `PSSM_GREMLIN_REDIS_URL` | Broker/backend URL used by Celery. Defaults to the bundled Redis service. |
| `PSSM_GREMLIN_PORT` | External HTTP port exposed by the `web` service. |
| `NPROC`, `WORKER_CONCURRENCY`, `GUNICORN_WORKERS` | Performance knobs for the runner, Celery worker, and Gunicorn respectively. |
| `REDIS_URL` | Broker/backend URL used by Celery. Defaults to the bundled Redis service. |
| `PORT` | External HTTP port exposed by the `web` service. |
| `PUBLIC_DASHBOARD` | Optional dashboard visibility switch. Default `false` enforces per-user isolation (only task owner can list/view/download/cancel). Set `true` to expose all tasks to any authenticated user. |
| `ADMIN_USERS` | Comma-separated Basic Auth usernames treated as server admins (default `admin`). Admins can batch-delete tasks and can delete any task regardless of owner. |

Expand Down Expand Up @@ -186,7 +186,7 @@ These tests mock out the Docker client and validate the environment-driven confi
```shell
lsof -i :8080
```
If `8080` is not ocuppied, this command returns nothing, otherwise the process name and PID will be shown. When running under Docker Compose, update `PSSM_GREMLIN_PORT` inside `server/.env`. For a manual install edit the constants in `server/pssm_gremlin/pssm_gremlin.py`.
If `8080` is not ocuppied, this command returns nothing, otherwise the process name and PID will be shown. When running under Docker Compose, update `PORT` inside `server/.env`. For a manual install edit the constants in `server/pssm_gremlin/pssm_gremlin.py`.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

Now we start the stack (Gunicorn + Celery + Redis). Under Compose:
```shell
Expand Down
Loading
Loading