Scikit-learn has autocomplete type "module" when it should seemingly be "function"

[nscozzaro]

Hi, I'm wondering why, for example, load_iris in sklearn is being autocompleted with a type "module" instead of a "function"? I'm building an application that uses this type information, so it causes me a problem that the type isn't what I expect.

load_iris documentation: https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html#sklearn.datasets.load_iris

Below I have a screenshot showing use of the autocomplete in JupyterLab:

Screenshot from load_iris source code (https://github.com/scikit-learn/scikit-learn/blob/b194674c42d54b26137a456c510c5fdba1ba23e0/sklearn/datasets/_base.py#L326):

[davidhalter]

Jedi version?

[davidhalter]

I doubt that this is a Jedi issue. jedi.Interpreter('from sklearn.datasets import load_ir', []).complete()[0].type returns function.

This might be a JupyterLab issue or you might have an old version.

Closing, because this is probably not a Jedi issue, but happy to help you find your issue, so feel free to keep responding.

[nscozzaro]

Thank you @davidhalter -- Jupyter installed Jedi version 13 automatically, so manually upgrading to version 16 solved the problem for me.
One question, what is the easiest way to use Jedi (or similar tool) to parse an entire python library, e.g. sklearn, into one JSON file? Given just the top-level package name (e.g. "sklearn") I need to achieve a format like:

{ 
    libraries: [
        {
            name: sklearn,
            type: module,
            children: [
                {
                      name: module1,
                      type: module,
                      children: [...more modules...]
                }, 
                {
                      name: function1, 
                      type: function
                      params: [
                           {
                                 name: param1,
                                 type: Boolean,
                                 default: False
                           }
                      ]
                },
                {
                      name: class1, 
                      type: class
                      methods: [
                           {
                                 name: method1,
                                 params: [...]
                           }
                      ], 
                      attributes: [...]
                }
            ]
        },
       {...other libraries...}
    ]
}

For some background, I'm working on a drag-n-drop graphical programming application that first converts functions and classes into a set of "nodes" with ports representing inputs/outputs. The first step is to create the full set of nodes, so it would be great to have this JSON file to cache and work with. It would also be very useful to have both the parameter type and value... I browsed issue #1039 and it seems that both parameter type and value might not be available yet?

[davidhalter]

I'm not sure you need type inference. If you don't, you don't need to use Jedi. Just use https://github.com/davidhalter/parso. It's what Jedi uses as well to parse source code. Does that make sense?

[nscozzaro]

The type inference is necessary for my application in order to know whether an output is compatible with an input on these nodes. For example, if one function returns type "bool" and another accepts type "int" for a specific parameter, I need to leverage that type information in order to prevent a user connecting ports with different types.

[davidhalter]

Yeah, I can probably not help you with that. In some distant future, #1059 might be a good option for you. But for now this is just not really working. Mostly for performance reasons I decided not to make certain APIs public. Maybe get_type_hint() helps you (I just added that yesterday), but I doubt that it will help you with all cases.

[gramster]

Speaking of get_type_hint, that doesn't seem to be defined on Definition? At least I have a Definition which has type 'function' and there is not get_type_hint method (I installed jedi from head of repo today).

[davidhalter]

It's definitely defined on master. I just checked again.

[gramster]

Yes, you're right - I foolishly did not uninstall my PyPI installed version before installing the repo-built version and my environment was still picking up the old install. Thanks!

[davidhalter]

@gramster Do you have a specific usage in mind for this?

[nscozzaro]

Hi @davidhalter,

I'm also not seeing the type (module/class/function) for this Tensorflow autocomplete, and the Interpreter returns empty:
e.g. https://www.tensorflow.org/api_docs/python/tf/keras/Sequential

I'm using Jedi 0.16.0

Please let me know what could be the reason/workaround, as it's important for my application, thank you!

[davidhalter]

Can you give me jedi.Interpreter('from tensorflow import ', [globals()]).complete()?

[nscozzaro]

Can you give me jedi.Interpreter('from tensorflow import ', [globals()]).complete()?

[<Completion: abs>,
 <Completion: acos>,
 <Completion: acosh>,
 <Completion: add>,
 <Completion: add_n>,
 <Completion: AggregationMethod>,
 <Completion: argmax>,
 <Completion: argmin>,
 <Completion: argsort>,
 <Completion: as_dtype>,
 <Completion: as_string>,
 <Completion: asin>,
 <Completion: asinh>,
 <Completion: Assert>,
 <Completion: assert_equal>,
 <Completion: assert_greater>,
 <Completion: assert_less>,
 <Completion: assert_rank>,
 <Completion: atan>,
 <Completion: atan2>,
 <Completion: atanh>,
 <Completion: audio>,
 <Completion: autodiff>,
 <Completion: autograph>,
 <Completion: batch_to_space>,
 <Completion: bfloat16>,
 <Completion: bitcast>,
 <Completion: bitwise>,
 <Completion: bool>,
 <Completion: boolean_mask>,
 <Completion: broadcast_dynamic_shape>,
 <Completion: broadcast_static_shape>,
 <Completion: broadcast_to>,
 <Completion: case>,
 <Completion: cast>,
 <Completion: clip_by_global_norm>,
 <Completion: clip_by_norm>,
 <Completion: clip_by_value>,
 <Completion: compat>,
 <Completion: complex>,
 <Completion: complex128>,
 <Completion: complex64>,
 <Completion: concat>,
 <Completion: cond>,
 <Completion: config>,
 <Completion: constant>,
 <Completion: constant_initializer>,
 <Completion: control_dependencies>,
 <Completion: convert_to_tensor>,
 <Completion: cos>,
 <Completion: cosh>,
 <Completion: CriticalSection>,
 <Completion: cumsum>,
 <Completion: custom_gradient>,
 <Completion: data>,
 <Completion: debugging>,
 <Completion: device>,
 <Completion: DeviceSpec>,
 <Completion: distribute>,
 <Completion: divide>,
 <Completion: double>,
 <Completion: DType>,
 <Completion: dtypes>,
 <Completion: dynamic_partition>,
 <Completion: dynamic_stitch>,
 <Completion: edit_distance>,
 <Completion: eig>,
 <Completion: eigvals>,
 <Completion: einsum>,
 <Completion: ensure_shape>,
 <Completion: equal>,
 <Completion: errors>,
 <Completion: estimator>,
 <Completion: executing_eagerly>,
 <Completion: exp>,
 <Completion: expand_dims>,
 <Completion: experimental>,
 <Completion: extract_volume_patches>,
 <Completion: eye>,
 <Completion: feature_column>,
 <Completion: fill>,
 <Completion: fingerprint>,
 <Completion: float16>,
 <Completion: float32>,
 <Completion: float64>,
 <Completion: floor>,
 <Completion: foldl>,
 <Completion: foldr>,
 <Completion: function>,
 <Completion: gather>,
 <Completion: gather_nd>,
 <Completion: get_logger>,
 <Completion: get_static_value>,
 <Completion: grad_pass_through>,
 <Completion: gradients>,
 <Completion: GradientTape>,
 <Completion: Graph>,
 <Completion: graph_util>,
 <Completion: greater>,
 <Completion: greater_equal>,
 <Completion: group>,
 <Completion: guarantee_const>,
 <Completion: half>,
 <Completion: hessians>,
 <Completion: histogram_fixed_width>,
 <Completion: histogram_fixed_width_bins>,
 <Completion: identity>,
 <Completion: identity_n>,
 <Completion: image>,
 <Completion: import_graph_def>,
 <Completion: IndexedSlices>,
 <Completion: IndexedSlicesSpec>,
 <Completion: init_scope>,
 <Completion: initializers>,
 <Completion: int16>,
 <Completion: int32>,
 <Completion: int64>,
 <Completion: int8>,
 <Completion: io>,
 <Completion: is_tensor>,
 <Completion: keras>,
 <Completion: less>,
 <Completion: less_equal>,
 <Completion: linalg>,
 <Completion: linspace>,
 <Completion: lite>,
 <Completion: load_library>,
 <Completion: load_op_library>,
 <Completion: logical_and>,
 <Completion: logical_not>,
 <Completion: logical_or>,
 <Completion: lookup>,
 <Completion: losses>,
 <Completion: make_ndarray>,
 <Completion: make_tensor_proto>,
 <Completion: map_fn>,
 <Completion: math>,
 <Completion: matmul>,
 <Completion: matrix_square_root>,
 <Completion: maximum>,
 <Completion: meshgrid>,
 <Completion: metrics>,
 <Completion: minimum>,
 <Completion: mixed_precision>,
 <Completion: mlir>,
 <Completion: Module>,
 <Completion: multiply>,
 <Completion: name_scope>,
 <Completion: negative>,
 <Completion: nest>,
 <Completion: newaxis>,
 <Completion: nn>,
 <Completion: no_gradient>,
 <Completion: no_op>,
 <Completion: nondifferentiable_batch_function>,
 <Completion: norm>,
 <Completion: not_equal>,
 <Completion: numpy_function>,
 <Completion: one_hot>,
 <Completion: ones>,
 <Completion: ones_initializer>,
 <Completion: ones_like>,
 <Completion: Operation>,
 <Completion: optimizers>,
 <Completion: OptionalSpec>,
 <Completion: pad>,
 <Completion: parallel_stack>,
 <Completion: pow>,
 <Completion: print>,
 <Completion: py_function>,
 <Completion: qint16>,
 <Completion: qint32>,
 <Completion: qint8>,
 <Completion: quantization>,
 <Completion: queue>,
 <Completion: quint16>,
 <Completion: quint8>,
 <Completion: ragged>,
 <Completion: RaggedTensor>,
 <Completion: RaggedTensorSpec>,
 <Completion: random>,
 <Completion: random_normal_initializer>,
 <Completion: random_uniform_initializer>,
 <Completion: range>,
 <Completion: rank>,
 <Completion: raw_ops>,
 <Completion: realdiv>,
 <Completion: recompute_grad>,
 <Completion: reduce_all>,
 <Completion: reduce_any>,
 <Completion: reduce_logsumexp>,
 <Completion: reduce_max>,
 <Completion: reduce_mean>,
 <Completion: reduce_min>,
 <Completion: reduce_prod>,
 <Completion: reduce_sum>,
 <Completion: register_tensor_conversion_function>,
 <Completion: RegisterGradient>,
 <Completion: repeat>,
 <Completion: required_space_to_batch_paddings>,
 <Completion: reshape>,
 <Completion: resource>,
 <Completion: reverse>,
 <Completion: reverse_sequence>,
 <Completion: roll>,
 <Completion: round>,
 <Completion: saturate_cast>,
 <Completion: saved_model>,
 <Completion: scalar_mul>,
 <Completion: scan>,
 <Completion: scatter_nd>,
 <Completion: searchsorted>,
 <Completion: sequence_mask>,
 <Completion: sets>,
 <Completion: shape>,
 <Completion: shape_n>,
 <Completion: sigmoid>,
 <Completion: sign>,
 <Completion: signal>,
 <Completion: sin>,
 <Completion: sinh>,
 <Completion: size>,
 <Completion: slice>,
 <Completion: sort>,
 <Completion: space_to_batch>,
 <Completion: space_to_batch_nd>,
 <Completion: sparse>,
 <Completion: SparseTensor>,
 <Completion: SparseTensorSpec>,
 <Completion: split>,
 <Completion: sqrt>,
 <Completion: square>,
 <Completion: squeeze>,
 <Completion: stack>,
 <Completion: stop_gradient>,
 <Completion: strided_slice>,
 <Completion: string>,
 <Completion: strings>,
 <Completion: subtract>,
 <Completion: summary>,
 <Completion: switch_case>,
 <Completion: sysconfig>,
 <Completion: tan>,
 <Completion: tanh>,
 <Completion: Tensor>,
 <Completion: tensor_scatter_nd_add>,
 <Completion: tensor_scatter_nd_sub>,
 <Completion: tensor_scatter_nd_update>,
 <Completion: TensorArray>,
 <Completion: TensorArraySpec>,
 <Completion: tensordot>,
 <Completion: TensorShape>,
 <Completion: TensorSpec>,
 <Completion: test>,
 <Completion: tile>,
 <Completion: timestamp>,
 <Completion: tpu>,
 <Completion: train>,
 <Completion: transpose>,
 <Completion: truediv>,
 <Completion: truncatediv>,
 <Completion: truncatemod>,
 <Completion: tuple>,
 <Completion: TypeSpec>,
 <Completion: uint16>,
 <Completion: uint32>,
 <Completion: uint64>,
 <Completion: uint8>,
 <Completion: UnconnectedGradients>,
 <Completion: unique>,
 <Completion: unique_with_counts>,
 <Completion: unravel_index>,
 <Completion: unstack>,
 <Completion: Variable>,
 <Completion: variable_creator_scope>,
 <Completion: VariableAggregation>,
 <Completion: VariableSynchronization>,
 <Completion: variant>,
 <Completion: vectorized_map>,
 <Completion: version>,
 <Completion: where>,
 <Completion: while_loop>,
 <Completion: xla>,
 <Completion: zeros>,
 <Completion: zeros_initializer>,
 <Completion: zeros_like>,
 <Completion: _absolute_import>,
 <Completion: _API_MODULE>,
 <Completion: _compat>,
 <Completion: _current_file_location>,
 <Completion: _current_module>,
 <Completion: _distutils>,
 <Completion: _division>,
 <Completion: _fi>,
 <Completion: _forward_module>,
 <Completion: _importlib>,
 <Completion: _inspect>,
 <Completion: _LazyLoader>,
 <Completion: _ll>,
 <Completion: _logging>,
 <Completion: _m>,
 <Completion: _module_util>,
 <Completion: _names_with_underscore>,
 <Completion: _os>,
 <Completion: _plugin_dir>,
 <Completion: _print_function>,
 <Completion: _root_estimator>,
 <Completion: _running_from_pip_package>,
 <Completion: _s>,
 <Completion: _site>,
 <Completion: _site_packages_dirs>,
 <Completion: _sys>,
 <Completion: _tf_api_dir>,
 <Completion: _top_level_modules>,
 <Completion: _types>,
 <Completion: __all__>,
 <Completion: __compiler_version__>,
 <Completion: __cxx11_abi_flag__>,
 <Completion: __doc__>,
 <Completion: __file__>,
 <Completion: __git_version__>,
 <Completion: __monolithic_build__>,
 <Completion: __name__>,
 <Completion: __package__>,
 <Completion: __path__>,
 <Completion: __version__>]

[davidhalter]

I'll get back to you. But please don't take pictures anymore of stuff. I want to copy things, this way I have to re-type it ;-).

[nscozzaro]

Thanks for the support. And you got it, no more screenshots.

[davidhalter]

Can you retry with 0.17.0, just so we are sure that it wasn't fixed along the line?

[nscozzaro]

Hi, I just tried 0.17.0, but unfortunately jedi.Interpreter('from tensorflow.keras import ', [globals()]).complete() still returns empty when I would expect it to return the classes and functions that are in that module.

[davidhalter]

Just checked again why this is happening: Tensorflow 2.1.0 uses a directory that pretty much contains nothing (especially no keras folder). It then modifies the sys modules (see code below). Sys module modifications are not understood by Jedi. And if you read the code below, you can probably understand why. So if don't undo this change, I guess they should just write stubs.

For now I'm not going to do anything. This is an issue with tensorflow code IMO. Most of the time these kind of structures are pretty stupid and don't really provide a benefit.

def _forward_module(old_name):
  parts = old_name.split(".")
  parts[0] = parts[0] + "_core"
  local_name = parts[-1]
  existing_name = ".".join(parts)
  _module = _LazyLoader(local_name, globals(), existing_name)
  return _sys.modules.setdefault(old_name, _module)

[davidhalter]

Please let us continue the tensorflow discussion in #1391.