Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions tests/test_autofix_validation.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
"""Test file with intentional issues to validate autofix system.

This file contains:
1. Black formatting violations
2. Ruff lint errors
3. Mypy type errors
4. Failing tests
5. Actual useful test coverage

Purpose: Validate the full autofix pipeline handles all failure modes.
"""

# --- BLACK VIOLATION: Bad formatting ---

# --- RUFF VIOLATIONS ---
# F401: unused import
import os

from adapters.base import connect_db, get_adapter

# E501: line too long
VERY_LONG_STRING_THAT_VIOLATES_LINE_LENGTH = "This is a very long string that definitely exceeds the maximum line length limit of 88 characters that ruff and black enforce by default"
Copy link

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description lists E501 (line too long) as an intentional Ruff violation that should be caught, but the project's ruff configuration in pyproject.toml explicitly ignores E501 with ignore = ["E501"]. This intentional violation will not trigger a lint error and therefore will not validate the autofix system's handling of E501 errors.

Copilot uses AI. Check for mistakes.


# --- MYPY TYPE ERROR ---
def bad_type_annotation(x: int) -> str:
return x # Returns int, claims str
Comment on lines +26 to +27
Copy link

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description lists Mypy type errors as intentional issues that should trigger Codex dispatch, but the project's mypy configuration in pyproject.toml excludes the tests directory and also has overrides that ignore errors in adapters. Since this test file imports from adapters.base, mypy type checking may not catch these errors as expected, preventing validation of the autofix system's handling of type errors.

Copilot uses AI. Check for mistakes.


def missing_return_type(value):
"""Function missing type annotations."""
return value * 2


# --- BLACK VIOLATION: Inconsistent spacing ---
def poorly_formatted_function(arg1, arg2, arg3):
"""This function has poor formatting."""
result = arg1 + arg2 + arg3
if result > 10:
return result
else:
return result * 2


class BadlyFormattedClass:
"""Class with formatting issues."""

def __init__(self, name: str, value: int):
self.name = name
self.value = value

def compute(self, multiplier: int) -> int:
return self.value * multiplier


# --- ACTUAL USEFUL COVERAGE TESTS ---


def test_connect_db_sqlite_default():
"""Test connect_db returns valid SQLite connection."""
# Clear env vars to force SQLite path
old_url = os.environ.pop("DB_URL", None)
old_path = os.environ.pop("DB_PATH", None)
Comment on lines +62 to +63
Copy link

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test manipulates environment variables directly with os.environ.pop(), but other tests in the codebase use pytest's monkeypatch fixture for environment variable management (see tests/test_adapter_base.py:16, tests/test_embeddings.py:31). Using monkeypatch ensures automatic cleanup and prevents test pollution. Consider accepting monkeypatch as a parameter and using monkeypatch.delenv() or monkeypatch.setenv() instead.

Copilot uses AI. Check for mistakes.

try:
conn = connect_db(":memory:")
assert conn is not None
# Verify it's a working connection
cursor = conn.cursor()
cursor.execute("SELECT 1")
result = cursor.fetchone()
assert result == (1,)
conn.close()
finally:
if old_url:
os.environ["DB_URL"] = old_url
if old_path:
os.environ["DB_PATH"] = old_path


def test_connect_db_with_timeout():
"""Test connect_db accepts timeout parameter."""
conn = connect_db(":memory:", connect_timeout=5.0)
assert conn is not None
conn.close()


def test_get_adapter_edgar():
"""Test that edgar adapter can be loaded."""
try:
adapter = get_adapter("edgar")
assert adapter is not None
# Verify it has expected protocol methods
assert (
hasattr(adapter, "list_new_filings")
or callable(getattr(adapter, "list_new_filings", None)) is False
)
except ModuleNotFoundError:
# Adapter module may not exist yet
pass


def test_get_adapter_invalid():
"""Test get_adapter raises for unknown adapter."""
try:
get_adapter("nonexistent_adapter_xyz")
assert False, "Should have raised"

Check failure on line 107 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (B011)

tests/test_autofix_validation.py:107:16: B011 Do not `assert False` (`python -O` removes these calls), raise `AssertionError()`

Check failure on line 107 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (B011)

tests/test_autofix_validation.py:107:16: B011 Do not `assert False` (`python -O` removes these calls), raise `AssertionError()`

Check failure on line 107 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (B011)

tests/test_autofix_validation.py:107:16: B011 Do not `assert False` (`python -O` removes these calls), raise `AssertionError()`

Check failure on line 107 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (B011)

tests/test_autofix_validation.py:107:16: B011 Do not `assert False` (`python -O` removes these calls), raise `AssertionError()`

Check failure on line 107 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (B011)

tests/test_autofix_validation.py:107:16: B011 Do not `assert False` (`python -O` removes these calls), raise `AssertionError()`

Check failure on line 107 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (B011)

tests/test_autofix_validation.py:107:16: B011 Do not `assert False` (`python -O` removes these calls), raise `AssertionError()`

Check failure on line 107 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (B011)

tests/test_autofix_validation.py:107:16: B011 Do not `assert False` (`python -O` removes these calls), raise `AssertionError()`

Check failure on line 107 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (B011)

tests/test_autofix_validation.py:107:16: B011 Do not `assert False` (`python -O` removes these calls), raise `AssertionError()`
except (ModuleNotFoundError, ImportError):
pass # Expected


# --- INTENTIONALLY FAILING TESTS ---


def test_intentional_failure_assertion():
"""This test intentionally fails with an assertion error."""
expected = 42
actual = 41
assert actual == expected, f"Expected {expected} but got {actual}"
Comment on lines +115 to +119
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Remove unconditional failing tests blocking suite

The three “intentional failure” tests here all fail unconditionally (41 == 42, followed by a KeyError and a TypeError in the two tests immediately below), and they are not marked xfail/skip. As written, any pytest run will halt on this block, keeping CI permanently red and preventing the rest of the suite from running; these should be guarded or deleted if the goal is a stable test run.

Useful? React with 👍 / 👎.

Comment on lines +115 to +119
Copy link

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test intentionally fails by asserting 41 == 42. While this is documented as an intentional failure for validation purposes, when this test runs it will cause the test suite to fail. Consider whether this test should be marked with a custom pytest marker (like @pytest.mark.skip(reason="Intentional failure for autofix validation")) until the autofix system is validated, to avoid breaking the main test suite.

Copilot uses AI. Check for mistakes.


def test_intentional_failure_exception():
"""This test intentionally raises an exception."""
data = {"key": "value"}
# This will raise KeyError
result = data["nonexistent_key"]

Check failure on line 126 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:126:5: F841 Local variable `result` is assigned to but never used

Check failure on line 126 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:126:5: F841 Local variable `result` is assigned to but never used

Check failure on line 126 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:126:5: F841 Local variable `result` is assigned to but never used

Check failure on line 126 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:126:5: F841 Local variable `result` is assigned to but never used

Check failure on line 126 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:126:5: F841 Local variable `result` is assigned to but never used

Check failure on line 126 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:126:5: F841 Local variable `result` is assigned to but never used

Check failure on line 126 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:126:5: F841 Local variable `result` is assigned to but never used

Check failure on line 126 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:126:5: F841 Local variable `result` is assigned to but never used
Comment on lines +122 to +126
Copy link

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test intentionally raises a KeyError. While this is documented as an intentional failure, when executed it will cause the test suite to fail. The same concern applies as with the other intentionally failing tests - consider using pytest markers to skip these until the autofix system has been validated, to prevent disrupting the normal development workflow.

Copilot uses AI. Check for mistakes.
Copy link

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable result is not used.

Suggested change
result = data["nonexistent_key"]
data["nonexistent_key"]

Copilot uses AI. Check for mistakes.


def test_intentional_failure_type_error():
"""This test intentionally causes a TypeError."""
value = "not a number"
# This will raise TypeError
result = value + 5

Check failure on line 133 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:133:5: F841 Local variable `result` is assigned to but never used

Check failure on line 133 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:133:5: F841 Local variable `result` is assigned to but never used

Check failure on line 133 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:133:5: F841 Local variable `result` is assigned to but never used

Check failure on line 133 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:133:5: F841 Local variable `result` is assigned to but never used

Check failure on line 133 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:133:5: F841 Local variable `result` is assigned to but never used

Check failure on line 133 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:133:5: F841 Local variable `result` is assigned to but never used

Check failure on line 133 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:133:5: F841 Local variable `result` is assigned to but never used

Check failure on line 133 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (F841)

tests/test_autofix_validation.py:133:5: F841 Local variable `result` is assigned to but never used
Comment on lines +129 to +133
Copy link

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test intentionally raises a TypeError. Like the other intentionally failing tests, this will cause test suite failures when executed. Consider using pytest skip markers for these validation tests to avoid disrupting normal CI/CD workflows until the autofix system has been properly validated.

Copilot uses AI. Check for mistakes.
Copy link

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable result is not used.

Suggested change
result = value + 5
value + 5

Copilot uses AI. Check for mistakes.


# --- RUFF VIOLATIONS: More lint issues ---


# W293: whitespace on blank line
def function_with_trailing_whitespace():
"""Has trailing whitespace."""
x = 1

y = 2
return x + y


# E711: comparison to None
def bad_none_comparison(value):
if value == None:

Check failure on line 150 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E711)

tests/test_autofix_validation.py:150:17: E711 Comparison to `None` should be `cond is None`

Check failure on line 150 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E711)

tests/test_autofix_validation.py:150:17: E711 Comparison to `None` should be `cond is None`

Check failure on line 150 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E711)

tests/test_autofix_validation.py:150:17: E711 Comparison to `None` should be `cond is None`

Check failure on line 150 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E711)

tests/test_autofix_validation.py:150:17: E711 Comparison to `None` should be `cond is None`

Check failure on line 150 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E711)

tests/test_autofix_validation.py:150:17: E711 Comparison to `None` should be `cond is None`

Check failure on line 150 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E711)

tests/test_autofix_validation.py:150:17: E711 Comparison to `None` should be `cond is None`

Check failure on line 150 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E711)

tests/test_autofix_validation.py:150:17: E711 Comparison to `None` should be `cond is None`

Check failure on line 150 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E711)

tests/test_autofix_validation.py:150:17: E711 Comparison to `None` should be `cond is None`
Copy link

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing for None should use the 'is' operator.

Suggested change
if value == None:
if value is None:

Copilot uses AI. Check for mistakes.
return "empty"
return "full"


# E712: comparison to True
def bad_bool_comparison(flag):
if flag == True:

Check failure on line 157 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E712)

tests/test_autofix_validation.py:157:8: E712 Avoid equality comparisons to `True`; use `flag:` for truth checks

Check failure on line 157 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E712)

tests/test_autofix_validation.py:157:8: E712 Avoid equality comparisons to `True`; use `flag:` for truth checks

Check failure on line 157 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E712)

tests/test_autofix_validation.py:157:8: E712 Avoid equality comparisons to `True`; use `flag:` for truth checks

Check failure on line 157 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E712)

tests/test_autofix_validation.py:157:8: E712 Avoid equality comparisons to `True`; use `flag:` for truth checks

Check failure on line 157 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E712)

tests/test_autofix_validation.py:157:8: E712 Avoid equality comparisons to `True`; use `flag:` for truth checks

Check failure on line 157 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E712)

tests/test_autofix_validation.py:157:8: E712 Avoid equality comparisons to `True`; use `flag:` for truth checks

Check failure on line 157 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E712)

tests/test_autofix_validation.py:157:8: E712 Avoid equality comparisons to `True`; use `flag:` for truth checks

Check failure on line 157 in tests/test_autofix_validation.py

View workflow job for this annotation

GitHub Actions / Python CI / lint-ruff

Ruff (E712)

tests/test_autofix_validation.py:157:8: E712 Avoid equality comparisons to `True`; use `flag:` for truth checks
return "yes"
return "no"


# Autofix retest - 2025-12-29T05:30:11Z
Loading