Start adding type hints #1008

marcelm · 2021-04-22T16:37:54Z

This is an attempt to add a couple of type hints to pysam. I’ve started with libcalignedsegment.pyi. I’m not finished, but need to stop for the moment.

I think these annotations would already be useful in the current state, but for this to be merged, I think we would at least need to adjust the test scripts so that they run mypy (it fails at the moment).

jmarshall · 2021-04-23T14:05:11Z

Having type hints where appropriate would be good.

Is there a particular reason why you have used separate stub files rather than inline type annotations (which I believe can be used in Cython)?

Having these hints in separate files would increase the maintenance burden, as they would easily get out of sync with the definitions in the main .pyx files. So IMHO for something being merged into the main pysam source project, it would be preferable to use type annotations in the main source files instead.

That would of course require dropping Python 2.7 support. That is IMHO long overdue, but it is a decision for the primary pysam maintainers (cc @AndreasHeger @kevinjacobs-progenity).

kevinjacobs-progenity · 2021-04-23T14:24:12Z

I'm fine with dropping Python 2 support for future releases. There are many advantages to doing so and many other commonly used bioinformatics libraries have done this. All of my codebases have migrated to Python 3.7+ and use optional typing extensively, so I have plenty of test cases.

marcelm · 2021-04-23T15:30:20Z

Is there a particular reason why you have used separate stub files rather than inline type annotations (which I believe can be used in Cython)?

Using inline type annotations would be awesome, but if I’m not mistaken, it won’t actually work. Mypy can only parse annotations from a .py or .pyi file (see python/mypy#8575 (comment)). See also python/mypy#8631. (But stubgen, on the other hand, does work somewhat – I ran it on pysam to get started with the annotations).

Also, if you define def f(int x): pass in a .pyx file, Cython generates code that checks at runtime whether the passed x object is an int. If you write this as def f(x: int): pass, then this code is not generated. So the runtime behavior would change. I think PEP 484 annotations are most useful (or perhaps only meant to be used) in pure Python mode.

Another problem is that I wouldn’t know how to annotate some items, for example properties.

Separate .pyi files have an upside as well: They allow you to present the API that you want people to use. I omitted all the deprecated functions and parameters in this PR.

Perhaps keeping the .pyx and .pyi files in sync can be solved with some additional tests.

mr-c · 2021-06-28T08:15:35Z

pysam/libcbgzf.pyi

+
+BUFFER_SIZE: int
+
+class BGZFile:


Does this class implement any of the IO interfaces?

https://github.com/python/typeshed/blob/master/stdlib/io.pyi

https://github.com/python/typeshed/blob/14d7431ff1f554ae1d103b996995bfe3ecec67bd/stdlib/typing.pyi#L486

Probably. What do you suggest?

Type annotations for the gzip module are here: https://github.com/python/typeshed/blob/master/stdlib/gzip.pyi
We could copy this in the future, but I’m currently focusing on annotating the other parts of pysam.

kevinjacobs-progenity

Thanks! This is a big job. I've left some comments.

kevinjacobs-progenity · 2021-06-28T13:06:55Z

pysam/libcbcf.pyi

+
+_D = TypeVar("_D")
+
+class VariantHeaderRecord:


Should also be Mapping[str, Optional[str]]?

It isn't immediately clear why value should be Optional, but it has been a long time since I've dug into this code.

Yes, this should definitely be simplified. I tried to factor out a couple of base classes, but then noticed subtle differences where the types didn’t quite match. For example, your keys() method returns a Python-2-style list, but Mapping.keys() returns an iterator. Perhaps it works anyway, but I decided to spell everything out explicitly for now. That was necessary as a first step anyway.

The value can be None, see VariantHeaderRecord.itervalues():

pysam/pysam/libcbcf.pyx

Lines 1335 to 1343 in e340c85

def itervalues(self):

"""D.itervalues() -> an iterator over the values of D"""

cdef bcf_hrec_t *r = self.ptr

if not r:

return

cdef int i

for i in range(r.nkeys):

if r.keys[i]:

yield charptr_to_str(r.vals[i]) if r.vals[i] else None

This was all written for Python 2, so it doesn't surprise me that it was never updated to the new ABCs. Sounds like work that should be done in a follow-on PR. Same for figuring out why keys can be None in VariantHeaderRecords.

I have pushed an updated version that uses a custom _Mapping class for all the common methods.

When I try to subclass one of the ABCs like this:

class VariantHeaderRecord(Sized):

I get the error message that I haven’t implemented all abstract methods (__len__ in this case), so it doesn’t actually help me reduce the number of lines.

kevinjacobs-progenity · 2021-06-28T13:09:21Z

pysam/libcbcf.pyi

+    def pop(self, key: str, default: str = ...) -> str: ...
+    # def remove(self) -> Any: ...  # crashes
+
+class VariantHeaderRecords:


Sequence[VariantHeaderRecord]? May need to add ABC base class to get mixins (contains, reversed, index, count).

Agreed, also to remaining comments. I’ll need to have a look at collections.abc to see what can be used.

I didn’t want to touch the implementation, so I haven’t actually changed this to derive from Sequence (the mixins are missing as you noted). That can be done at a later stage.

pysam/libcbcf.pyi

kevinjacobs-progenity · 2021-06-28T13:10:51Z

pysam/libcbcf.pyi

+    def header_record(self) -> VariantHeaderRecord: ...
+    def remove_header(self) -> None: ...
+
+class VariantHeaderContigs:


As above-- Mapping[str, VariantContig]? Should the value be Optional?

Do you remember why you thought the value may need to be Optional? (__getitem__ always returns a VariantContig and never None.)

kevinjacobs-progenity · 2021-06-28T13:11:07Z

pysam/libcbcf.pyi

+    def add(self, id: str, length: Optional[int] = ..., **kwargs) -> None: ...
+
+class VariantHeaderSamples:
+    @property


As above -- Sequence[str]?

Same here, I left this as is because VariantHeaderSamples doesn’t implement all the methods that a Sequence requires.

pysam/libcbcf.pyi

marcelm · 2021-06-28T13:48:45Z

I’ve finished what I think I can do before going on vacation soon.

This PR now covers about 80% of the API. I initially wanted to gradually add type hints, as the Mypy authors recommend, but that doesn’t work so well here because all the Cython source code is invisible to Mypy. So whenever you use anything for which no .pyi file exists, you get "doesn’t have attribute/module xyz" errors.

Note that the first commit in this PR is the same as PR #1009 (adding a GitHub Actions workflow). My suggestion would be to merge #1009 or some version of it separately because it’s not strictly what this PR is about.

Testing the type hints is currently done by running mypy pysam/. This only checks internal consistency, and should be extended at some point to also cover the tests/ directory.

marcelm · 2021-06-28T22:56:49Z

See also the test results

This is only for type checking: Mypy (and PyCharm) complain that libchtslib is undefined (within the definition of __all__). This is actually not true because "from pysam.x import y", when used within pysam/__init__.py, will actually add not only y, but also x into the list of available symbols.

marcelm · 2021-11-27T23:35:49Z

I’ve now annotated the remaining files. I added an empty libcvcf.pyi as it appears that libcvcf.pyx is outdated. Or is it?

The only remaining problem that I get now when running mypy pysam/ is this:

pysam/__init__.py:26: error: Incompatible import of "TabixIterator" (imported name has type "Type[pysam.libcbcf.TabixIterator]", local name has type "Type[pysam.libctabix.TabixIterator]")

And this is potentially the first "bug" that Mypy has found: There’s indeed a TabixIterator both in libcbcf.pyx and libctabix.pyx. I’m not sure what to do, they cannot both be exposed beneath the pysam. namespace.

HTSFile and HFile need a trick described in PEP 673 https://www.python.org/dev/peps/pep-0673/

marcelm · 2021-11-30T14:55:55Z

I took the liberty of publishing a pysam-stubs package to PyPI for those who want to easily test the type annotations.

AndreasHeger · 2022-03-24T19:54:11Z

Thanks! I will work on removing python2 next.

marcelm mentioned this pull request Apr 22, 2021

Mypy annotations #857

Closed

marcelm force-pushed the type-hints branch from cd627b9 to d236753 Compare April 23, 2021 13:28

marcelm force-pushed the type-hints branch from ecc20fd to bc98b06 Compare April 23, 2021 15:33

marcelm force-pushed the type-hints branch 3 times, most recently from 0cd21dc to 43d0fdf Compare June 28, 2021 07:40

mr-c reviewed Jun 28, 2021

View reviewed changes

kevinjacobs-progenity reviewed Jun 28, 2021

View reviewed changes

marcelm force-pushed the type-hints branch from 3ea3381 to 13b23ad Compare June 28, 2021 13:25

marcelm marked this pull request as ready for review June 28, 2021 13:48

marcelm added 15 commits November 26, 2021 13:35

Start to add static typing

a61935e

Add type hints for AlignedSegment, PileupRead, PileupColumn

e86ed7d

Python before 3.8 does not have Literal

2310b98

Make mypy happier by providing namedtuple attributes as tuple

a2fdf67

Add type hints for libchtslib

4003a84

Do not use deprecated tags attribute

236630c

Add type hints for libcutils

a8745aa

With these ignores, "mypy pysam/" passes

7e9ba2b

Add type hints for libcalignmentfile

35ee3f1

Add type hints for libcfaidx

e043c4a

Add type hints for libcbgzf

02955a9

Add type hints for libcbcf

97796f9

Type HTSFile filename more correctly

8a76014

Extract a _Mapping generic base class

fdf3a75

marcelm added 3 commits November 26, 2021 14:01

Add type hints for libcsamfile

0b7b6ca

Annotate get/set_encoding_error_handler

2cb76ab

Add type hints for libcbcftools and libcsamtools

c41650b

marcelm force-pushed the type-hints branch from a6683a7 to c41650b Compare November 26, 2021 17:34

marcelm added 3 commits November 27, 2021 23:41

Add type hints for remaining files

9eab03f

VariantRecordSamples can be indexed by int and str

283d532

Fix: VariantHeaderSamples is indexed by int

0c8d597

marcelm added 3 commits November 28, 2021 22:39

Format with black

4595703

Some type hint fixes

3957183

Fix __enter__

ebc7346

HTSFile and HFile need a trick described in PEP 673 https://www.python.org/dev/peps/pep-0673/

marcelm added 2 commits November 30, 2021 16:33

Use Union instead of | for now to be consistent with other annotations

b6eaab8

Fix _Mapping

83e4873

AndreasHeger merged commit 2a5e55e into pysam-developers:master Mar 24, 2022

marcelm deleted the type-hints branch March 24, 2022 20:03

jmarshall mentioned this pull request Apr 19, 2024

fix: add missing properties to AlignedSegment type stubs #1273

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start adding type hints #1008

Start adding type hints #1008

marcelm commented Apr 22, 2021

jmarshall commented Apr 23, 2021

kevinjacobs-progenity commented Apr 23, 2021

marcelm commented Apr 23, 2021

mr-c Jun 28, 2021

marcelm Jun 28, 2021

marcelm Jun 28, 2021

kevinjacobs-progenity left a comment

kevinjacobs-progenity Jun 28, 2021

kevinjacobs-progenity Jun 28, 2021

marcelm Jun 28, 2021

kevinjacobs-progenity Jun 28, 2021

marcelm Jun 28, 2021

kevinjacobs-progenity Jun 28, 2021

marcelm Jun 28, 2021

marcelm Nov 27, 2021

kevinjacobs-progenity Jun 28, 2021

marcelm Nov 27, 2021

kevinjacobs-progenity Jun 28, 2021

marcelm Nov 27, 2021

marcelm commented Jun 28, 2021

marcelm commented Jun 28, 2021

marcelm commented Nov 27, 2021

marcelm commented Nov 30, 2021

AndreasHeger commented Mar 24, 2022

	def itervalues(self):
	"""D.itervalues() -> an iterator over the values of D"""
	cdef bcf_hrec_t *r = self.ptr
	if not r:
	return
	cdef int i
	for i in range(r.nkeys):
	if r.keys[i]:
	yield charptr_to_str(r.vals[i]) if r.vals[i] else None

Start adding type hints #1008

Start adding type hints #1008

Conversation

marcelm commented Apr 22, 2021

jmarshall commented Apr 23, 2021

kevinjacobs-progenity commented Apr 23, 2021

marcelm commented Apr 23, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinjacobs-progenity left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcelm commented Jun 28, 2021

marcelm commented Jun 28, 2021

marcelm commented Nov 27, 2021

marcelm commented Nov 30, 2021

AndreasHeger commented Mar 24, 2022