Skip to content

Latest commit

 

History

History
94 lines (69 loc) · 3.79 KB

dev.md

File metadata and controls

94 lines (69 loc) · 3.79 KB
Raw

Developing

TODO

  • Cython correspond_pixels port
  • Cython nms port
  • Building script
  • Packaging script
  • BSDS evaluation script
  • BSDS evaluation test codes
  • Plot PR curves
  • NMS preprocess script
  • SBD evaluation script
  • "thin" GTs for SBD
  • Cityscapes evaluation script
  • Multiprocessing for evaluation
  • Set random seed for correspond_pixels
  • Move the scripts into the source code (currently moved to scripts for testing)
  • unit test coverage for important functions (would like to make tests for all functions)
  • Make a CLI interface (for evaluation/convert dataset)
  • SBDEvaluator tests are failing due to recent API updates in 0.2.7

Known Bugs and Problems

  • pyEdgeEval and MATLAB results differ slightly. This is due to various factors such as randomness in correspond_pixels and slight differences in preprocessing algorithms (thin, kill_internal, etc...). I would love to do a more comprehensive study on the differences MATLAB and Python, but I believe this benchmark code is currently robust enough to evaluate models.
  • Using multiprocessing causes a bug where every run produces different results. The seed for random number generator used in correspond_pixels.pyx causes randomness. Since this is a port of the original codebase, this problem exists on MATLAB script as well. To solve the issue, we would need to set the random seed every time we call correspond_pixels so that the results are reproducible (TODO).

Testing validity

BSDS500 provides a benchmark to test if the code is running as expected (on MATLAB). I used the same benchmark and created a test script to check if the results are the same. SBD also provides a similar benchmark which I converted to Python.

Theoratically the results should match exactly, but due to the results coming from two completely different languages as well as the dependencies being diffferent, it is understandable to have some minor diferences.

For BSDS500 and SBD, I have provided a test script inside tests. The results are very close and near identical, which means that the general functions are working. Making more detailed test code is WIP.

I also test cityscapes' edge generation code (convert_dataset/cityscapes.py) and compared it with the outputs using MATLAB code.

BSDS500

  • Test code: tests/test_bsds500.py
  • Benchmark data: data/BSDS500_bench

SBD

  • Test code: tests/test_sbd.py
  • Benchmark data: data/SBD_bench

Cityscapes

  • Test code: WIP
  • Benchmark data: data/cityscapes_test

Installation, Building, and Packaging

# Install dependencies
pip install -r requirements-dev.txt

# [try to build]
# Option 1. build as a package (`-v` for complete output)
pip install -e . -v
# Option 2. build, but don't install it as a pip package (doesn't conflict with pip installed versions)
python setup.py build_ext --inplace

# [build distribution]
python -m build

# [convert wheel (for linux)]
auditwheel -v repair dist/<package>.whl
mv wheelhouse/<new package>.whl dist/
rm dist/<old package>.whl

Acknowledgements

  • py-bsds500
    • referenced Cython ports for correspond_pixels used in the original MATLAB code
    • referenced python version of the evaluation script for BSDS500
  • cityscapes-preprocess
    • MATLAB script for preprocessing and creating GT edges
  • seal
    • semantic boundary detection protocols
  • edge_eval_python:
    • referenced implementation for bwmorph thinning algorithm and nms preprocessing
    • referenced PR-curve visualization codes
  • edges
    • tools for processing edges (written in MATLAB and C++)