Keras Keras __init__.py structure isn't readable by a static type checker

In keras 3.2.1, if you import say keras.models, you get no errors in VS code:

In keras 3.3.1, this same code shows an error:

I believe the reason for this was the refactoring of the __init__.py that happened in this commit:

https://github.com/keras-team/keras/commit/04891e89da87c2a433beb12ff7dad59403c71671

The code added in this commit dynamically adds the path to api to the sys path. This isn't something a static type checker can handle.

Could this change be reverted? Or perhaps marked with a if TYPECHECKING special case?

Comment From: rchiodo

Sorry I should have said, the code runs fine, it just can't be interpreted by a static type checker. Our users are confused by this because they can execute their code just fine, but Pylance doesn't think there's a module called keras.models. This makes it harder for them to write code using keras.models because there's no intellisense for it anymore.

Comment From: fchollet

@mattdangerw @sampathweb can we resolve this?

Comment From: fchollet

The code added in this commit dynamically adds the path to api to the sys path. This isn't something a static type checker can handle.

Basically, publicly-exported APIs are in keras/api/, rather than in the top-level keras/ folder. This is not unlike, e.g. TF APIs, which live in tf._api.v2. Can PyLance only handle packages where the directory structure matches the public namespace?

Comment From: rchiodo

Can PyLance only handle packages where the directory structure matches the public namespace?

We should be able to handle anything that's through static imports. The old code was importing all of the api modules, so Pylance could tell that those were imported in the __init__.py.

The code in the commit I referenced changed to add the api path to the sys.path at runtime. We don't execute any python code, so we can't actually figure out that api should be on the sys.path.

It's kind of similar to how we don't support PEP 660 because we can't execute the dynamic code in the .pth files.

Here's a similar example: https://github.com/microsoft/pyright/issues/6272

Comment From: rchiodo

We also have a problem with tensorflow too when it imports keras like this:

from tensorflow.keras import activations, initializers, layers, losses, metrics, models, optimizers, regularizers

This is because keras is added to the tensorflow package dynamically.

Comment From: mattdangerw

@rchiodo can you say more about the if TYPECHECKING special case?

Comment From: rchiodo

I'd be happy to.

If you've never seen it before, it's described here: https://typing.readthedocs.io/en/latest/spec/directives.html#type-checking

It basically tells type checkers extra information that the runtime doesn't care about.

So you might for example have this in your __init__.py

if typing.TYPE_CHECKING:
  from keras.api import _tf_keras
  from keras.api import activations
  from keras.api import applications
  from keras.api import backend
  from keras.api import callbacks
  from keras.api import config
  from keras.api import constraint
  # etc
else:
   # Import everything from /api/ into keras.
   from keras.api import *  # noqa: F403
   from keras.api import __version__  # Import * ignores names start with "_".

   # Add everything in /api/ to the module search path.
   __path__.append(os.path.join(os.path.dirname(__file__), "api"))  # noqa: F405

Comment From: mattdangerw

Thanks!

I guess if the issue is just with import *, we can probably just make sure we individually import everything from api -> toplevel. I don't think this needs to be done with if TYPE_CHECKING at all. We could even extend our api gen script to generate that block in the __init__.py file. @rchiodo @fchollet does that sounds like a plan?

Comment From: fchollet

Sounds like a plan! This is clean solution.

Comment From: google-ml-butler[bot]

Are you satisfied with the resolution of your issue? Yes No

Comment From: google-ml-butler[bot]

Are you satisfied with the resolution of your issue? Yes No

Comment From: rchiodo

Sorry but I don't know if there's a way to resolve this. Please see the comment I left in the PR: https://github.com/keras-team/keras/pull/19864#issuecomment-2174204607

Comment From: Splines

Does that mean that imports like the following are officially not supported by keras/tensorflow? In VSCode, something like this is not working for me. Both Pylint and Pylance complain in a virtual environment (venv). Other libraries like matplotlib work fine.

import keras

from keras.datasets import cifar10
from keras.utils import to_categorical
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D, Dense, Flatten, Dropout, Input
from keras.optimizers import SGD

import tensorflow as tf

What is the recommended workaround for this? It's a pretty big deal for me not having autocompletion available available in VSCode.

Comment From: artemgur

I found a simple workaround. You can import the same objects from keras.api and its submodules. Pylance correctly works with such imports, and intellisense fully works. For example, replace from keras.optimizers import SGD with from keras.api.optimizers import SGD

Comment From: AlanBogarin

I'm sorry I wasn't there when this was open, as I had the same problem during the development of my packages.

In short, the reason pylance doesn't recognize exported symbols is because it doesn't follow the Python convention for public symbol interfaces.

you can read it here

To solve this you have two ways to do it

1. create the __all__ symbol that will contain all the public symbols of the module or package

# keras/__init__.py
from ... import activations, applications, ...

__all__ = (
    "activations",
    "applications",
    "backend",
    ...  # all the symbols that will be public
)

1. Imports should use a redundant alias (this tells type checkers to treat them as public, since they are private by default)

from ... import activations as activations
from ... import applications as applications
...

Comment From: rivershah

This api/src division is causing problems and breaks conventions. It breaks IDE autocompletion and import resolution and forces developers to implement hacks or use non-standard imports and creates unnecessary confusion

pytorch, numpy and other major libraries separate implementation from API without breaking standard import patterns. Can we please consider reverting to a conventional package structure so all linting and analysis tools just work out of the box.

These kinds of imports should work seamless in vscode and other ides but don't:

from keras.layers import Dense

Comment From: AlanBogarin

This problem can be solved not by changing the structure, but by creating a script that fixes the formatting of modules generated by namex.

But I also think it's a good opportunity to significantly reduce the package size, considering @rivershah 's suggestion.

I opened an issue in another repository that appears to be a version compatible with type checking, so I became quite interested and reported this same problem with the src/api structure.

https://github.com/keras-team/keras-rs/issues/38