Install a python requirement with an --extra-index-url argument #1270
Comments
mocaccano
added
the
enhancement
|
Agreed that this is something we should formally support. In addition to the PIP_EXTRA_INDEX_URL workaround, you could download the wheel manually, and add a direct reference to the wheel in your requires list. Another workaround - although one that is a lot more fragile, and I wouldn't really want to rely on long term - would be to add [ |
|
Thank you for your responses. I tried adding |
To add to the fragility of this workaround, you can't have the space separating the option and its value...an equal sign must be used. This should be true for any requires = [
"--extra-index-url=https://download.pytorch.org/whl/cpu",
"torch==2.0.1",
"toga-gtk~=0.3.1",
] |
freakboy3742
added
the
good first issue
New here, but I just got through the tutorial and I'm looking to contribute. Wanted some clarity about what the most desirable solution would be:
|
|
The most flexible way would be to allow arbitrary pip options, like Chaquopy does. @freakboy3742 @rmartin16: any thoughts? |
|
That's definitely an option; my primary hesitation is the possibility that at some point in the future, we might provide an install option other than pip (the most likely alternative being Conda; but in theory, there could be others), as well as (potentially) lock file formats that are bound to specific tools (e.g., poetry). Baking pip options directly into the config could back us into a corner, whereas "repository url" and/or "extra repository URL" is less likely to be a problem. |
@freakboy3742, how about I then add |
|
Sounds like a good approach to me; with the caveat that the extra needs to be a list, not just a single URL (since you can specify multiple extra URLs) |
|
It would be fantastic if we could simply opt to disable briefcases package resolution and use an existing system like setup tools (boring tech FTW). from setuptools import setup
setup(
name='somepackage',
install_requires=[
'somedep'
],
dependency_links=[
'https://pypi.example.org/pypi/somedep/'
]
# ...
)pip install .
briefcase devI mean why re-invent the whl;) |
|
@rfletchr Briefcase will honor PEP621 metadata (now formally documented here). That means Beyond that, setuptools doesn't have sufficient flexibility to define the packaging needs of a bundled Python app, so it's not possible to fully specify a Briefcase configuration purely with setuptools. |
|
Adding to the feature list - we probably also want to support |
This comment was marked as off-topic.
This comment was marked as off-topic.
|
@Koffilo, please open a discussion if you have a question about BeeWare. Also please stop posting other questions in existing issues. |
|
I've got most of a PR ready for this change, however, I wanted to clarify something: pip's Additionally, to add support for a Solving this for remote URLs was relatively straightforward. The local paths are tripping me up a bit 😅. |
|
I'm not sure excessive testing of As for IIRC, the difference between Something that I hadn't considered before as a possibility that would avoid the need for an extra |
|
TL;DR: There are problems with using local directory references for Interesting to know about the
The fact that pip supports these formats are, I agree, outside of Briefcase's concerns, and Briefcase shouldn't test pip, just that it's giving pip the right options. To clarify my concern, it is that if the intention behind So, the question is this: is the intention for Briefcase to support local path references in these options the way pip does? Or, should Briefcase explicitly only support URLs? In other words, in the spirit of pip being an implementation detail, can Briefcase take a more conservative approach, and thereby avoid the complexity of needing to manage local path references for each of these contexts? As things currently stand, because this isn't an officially supported option in Briefcase, it makes sense that the developer experience of trying to use a local directory reference is broken and that users would need to come up with their own workarounds. It doesn't seem to me that Briefcase is necessarily bound to support those edge cases, and by using On the other hand, if the intention is to support all potential uses of those two pip options, then Briefcase needs to ensure local directory references work in all these cases. My point about testing was that I wasn't sure how to verify if the local directory references would be an issue (I'd never used these options with Briefcase before), but it's clear that they would be. I assume all of this all holds for an officially supported version of |
|
Quick clarification: handling the relative vs absolute path references is trivial, and would just need to follow the same logic as is used for local references in |
In short, no. In long... the longer term goal is that the "environment management" aspect of Briefcase will become pluggable. Although Briefcase currently uses This also means that shouldn't be aiming to expose every possible feature of pip literally. As with Toga, the goal should be to abstract the high level ideas that transcend any specific environment management implementation; longer term, if we need to add support for a specific pip option, it would be in a pip-specific configuration block (similar to how we have platform-generic permission management options, but iOS- and Android-specific mechanisms to output platform-specific permissions). In this context we need to be able to specify "where do you get your packages from" in a way that isn't a hack dependent on a specific interpretation of That said - if you provide a list of package repositories, it's then up to the installer how those sources are interpreted. I wouldn't expect a conda installer to be able to make sense of a pip-layout repository. But conda might need to be pointed at a different root conda repository, or supplemental repositories, or a local filesystem with pre-build conda packages. As for how to handle the limitations of local directory references - I'd be entirely comfortable if v1 of this feature didn't allow local filesystem references for Flatpak or Docker builds. It should be possible (if somewhat complex) to expose a file location for Docker by mapping the filesystem mount - but that could be added as a future enhancement. It might even be possible to resolve this for Flatpak by linking filesystems; but again, that can be a future enhancement. In the short term, I'm entirely comfortable documenting the limitation as a platform-specific quirk, and the platforms that have the limitation validate and report the configuration issue in a helpful way.
Interesting... I guess that upgrades the test to "filesystem location exists and contains an index.html" over whether a path is used as an
I'd say the guiding principle here is "does the underlying idea of the configuration option being exposed make sense regardless of the specific implementation?". That doesn't mean the feature is implemented everywhere - but there needs to at least be a potential interpretation that would make sense. Beyond that, we take a "best effort" approach. A local filesystem path will work in a lot of situations; there are time that it won't, and those edge cases we can document (and treat as future enhancements). Does that make sense? (Also - in the spirit of late-design bike sheds - I'm also wondering whether the name of the option should be |
|
Okay, thanks for the thorough response. This makes perfect sense, and is in line with what I was thinking as well. I'll get that branch cleaned up and ready for review by sometime tomorrow afternoon 🙂. We can bike shed the option names in the PR 😀 |
|
I've got the PR up as a draft here: #2057 However, in finalising it with documentation, and thinking further about how to support So: rather than trying to abstract these package installer options into a few generic names, and considering that support for different installers is on the roadmap, what if briefcase instead added support for explicit configuration of those installers' options? For now, only pip would have it, but future installers, however they are implemented, could rely on a loose API with a uniform entrypoint for retrieving configuration options. For example, could something like this work better to separate these concerns? [tool.briefcase.app.helloworld]
# This would be the default and for now the only option
requirement_installer = "pip"
[tool.briefcase.pip_options]
index_url = "" # maps explicitly to the option of the same name in pip, no ambiguity, no abstraction
extra_index_urls = [] # ditto!
find_links = "" # ditto!Then, turn This is, admittedly, a huge change, as the methods used on A similar approach could be taken to selecting the environment manager, if relevant. @freakboy3742 what do you think? |
|
Addendum: it strikes me that the installer options might be best suited on |
It's taken me a while to convince myself, but I think I'm inclined to agree. As much as I'd like to keep generic options to "keep it simple", every abstraction layer inevitably leads to gaps; and in this case, we're already talking about niche customisation, so the gaps are where we're going to get bitten. The real tipping point for me was digging into the full list of values that The only edge case I can think of is backends (or backend-specific app configuration) that need to provide specific overrides. For example, the iOS backend needs to inject the However, my inclination is that we're likely OK on both those counts. The iOS backend already defines the extra arguments as part of |
I agree with this approach, but I do have one suggestion: [tool.briefcase.pip_options]
index_url = "" # maps explicitly to the option of the same name in pip, no ambiguity, no abstraction
extra_index_urls = [] # ditto!
find_links = "" # ditto!I think mapping the options to a structured TOML representation is more trouble than it's worth, and probably can't even be done unambiguously:
I suggest we avoid all this complexity by taking the options as a flat list of strings, as Chaquopy does. For example: requirement_installer = "pip"
requirement_options = ["--extra-index-url", "https://whatever", "--no-deps"]
Yes, this will definitely be needed. If the options appear in multiple levels of the config, I suggest concatenating them with the less-specific sections coming first, as we already do with the |
Those are all reasonable points. Passing through arguments as literals has the added benefit of being the least possible implementation effort (it's a single new list-of-strings setting), and the least prone to error on our part. It also means that if pip ever adds a new option, end-users can add support for that option without any interaction from us. Exposing flags like My one concern/downside is that it doesn't provide any protection to the end user - so if a user gets creative and decides to add
Agreed. I think the use case is fairly narrow - there aren't many options that you would need to enable on a per-platform basis. The best I can think of would be enabling a specific index URL - but those should be able co-exist globally based purely on platform specifiers. However, I can't rule out that eccentricities will exist, so it might as well be cumulative. The good news is that by making it a single |
|
Point taken regarding a set list of options being ambiguous viz booleans, and potentially limited depending on the precise level of flexibility users wish to have moving forward. I agree that an option for users to pass whatever arguments they like seems like a good idea. However, there are still some edge cases to sort out. First, the path references issue I remarked on before potentially becomes a bit hairier if explicit options aren't defined (even if support for them is pushed off), but I think I have a solution that is better than anything we've discussed so far, would require minimal code changes to support, and would work for all build targets. Second, Path referencesExplicitly named options (as opposed to only a list of ambiguous argument strings) makes it possible for Briefcase to transform values as needed to match the requirements of a given build target's context. For example, local paths used for Relying solely on a simple list of string arguments to pass to Keep in mind that BeeWare already relies on relative paths being transformed to absolute ones in this configuration. Support for it is absolutely required for this to work (unless macOS builds happen at the project's root directory, I don't have a macOS device to test it, but based on my reading of the code, that is not the case, and an absolute path transformation is what makes this work). A sort of workaround exists using pip_install_arguments = ["--extra-index-url=helloworld-0.0.1/usr/lib/helloworld/app/packages"]It's a workaround, and maybe it falls under the arm-hurting analogy, but it's pretty ugly. The path just changes to match whatever Because if you use pip_install_arguments = ["--extra-index-url={app_path}/packages"]As far I as I can tell, that would work for everything except Flatpak, but should be fixable in that target's template. Other options I can think of that I like a lot less involve defining entirely new configuration options to facilitate easily identifying directories to copy to locations relative to where The most flexible and least complicated solution to me is adding the template variable for the new pip install arguments option and explicitly documenting the approach using Use of
|
Have we established exactly what forms of local paths are accepted by
That doesn't seem very intuitive, to use As you've noticed, we already allow
I agree. I think requirements files are only used by two platforms:
|
|
As for the name of the option, the reason I suggested |
Good catch on both of these. We definitely need to be careful in how we modify/transform any arguments to make sure we don't introduce a new class of problems - and we definitely don't want to get into the game of trying to parse and re-interpret pip's own arguments so we can work out how to "pair" them. However, I don't think it's necessarily as complex as you might think.
This isn't quite right. When Briefcase is invoking pip directly, it's always doing so in a working directory that is the project directory (i.e., the folder containing The three exceptions to this are:
Case 1 and 2 aren't a problem for relative paths - we know that any value is either a package name, a path, or a URL. There's a The complication for arbitrary arguments is that it's a little harder to say that any given argument is a filesystem path - but if we say "contains a / and the location is an existent file or directory", that should be safe enough (I think?) Case 3 is a lot harder to manage, because it requires adding a new filesystem mount and rewriting any filesystem path. We currently manage this by looking for any filesystem location in the requirements, and building an sdist locally, putting that into the Docker container, and then installing the sdist. That approach obviously won't work for
I don't have any fundamental objection to making these arguments templated... but I'm not sure I understand the use case. What case are you envisaging where a user will need to reference a location inside the transient build folder as a source of packages? Also - the
Is there any reason that we can't pass them in as arguments to pip install - even in the In the case for Flatpak, the invocation of pip is part of the flatpak manifest, which is templated - so we should be able to include any extra pip arguments in that template. The one downside is that the templated value is only generated during the create call - but we could address that by writing a pip invocation into a script every time requirements are updated, and then invoking that script as part of the flatpak manifest. That might even make the transition to other installers easier, because "install-requirements.sh" becomes something that is generated externally to the base template. The same is true of Android - Chaquopy allows you to pass in the pip invocation. The "write a script" approach won't work here, but once |
Specifically, it has access to /home, which we can reasonably assume most of the relevant local paths are within.
I think so. We would need to document that local paths in pip arguments need to contain a slash (e.g.
If I understand correctly, it was just a workaround for the difficulty of using relative paths in the pip arguments, but it looks like converting them to absolute paths automatically is a better solution, because it doesn't require the user to do anything special.
The The reason for this is that Chaquopy runs pip once for each ABI. So it needs to distinguish which arguments are options, which get passed to every pip invocation, and which ones are requirements sources. For the first ABI, it installs all requirements, and then for the remaining ABIs it installs native requirements only. |
|
Apologies for the tardy response, been a busy weekend. @freakboy3742 @rmartin16, please confirm if this summary matches your desired path forward:
If that is accurate, then I think the right order of PRs to support this would be:
This order ensures the option isn't available until the templates can actually use it. I'm making an assumption that this is important, particularly for Android which I suspect is a popular target, but it's just an assumption. If it doesn't need to be in this order, then I can group the first and last change into a single changeset if desired... though separate PRs might be easier to review and work with. Let me know. Just want to clarify a few things which shouldn't have any bearing on the above, but I feel were misunderstood in my previous comment. I've put them in a details/summary block to prevent distracting too much, but I did want to make sure I wasn't being misunderstood or causing confusion.Regarding automatic transformation of paths in the arguments string to absolute paths: my concern here is solved if Briefcase only transforms arguments that start with Regarding template variables in
The existence of This is an extremely minor point, but I fear I've misunderstood something about the intention of
I suggested there may be a bug in the Flatpak template, because it does not copy
Yes, there is. I had a detailed explanation of this in my comment originally, but I seem to have left it out when I re-wrote it after further exploration. I apologise that it was unclear. The requirements file only allows a single Given this file, It is fixed by splitting the first line into two, one for each argument. If the option is supplied as a flat list of strings, it becomes difficult to know for sure how to interpret the input such that the requirements file can be written with only a single argument and value per line. Given the following example I cannot think of an easy way to split it into arguments per-line: This is a valid list of You cannot merely group every 2 items, or else Furthermore, taking this specific example, |
No problems at all - we'll take whatever contributions we can get, on whatever timeline your schedule allows. I think you probably meant @mhsmith here...
👍
I don't think we can be as specific as "starts with I'd be more inclined to use a 3 part check:
Part (1) and (2) of that check are already implemented as part of
That's a good catch... I hadn't thought of that one. I can think of 2 possible approaches:
Given the inherent ambiguity associated with interpreting arguments, I'd argue keeping an escape hatch available to allow for a "file like" argument to not be treated as a file is probably worthwhile - but I'm open to be convinced I'm wrong on this one.
👍
This is a general problem that Briefcase has at present - see #472. There are a range of features that we can't currently update after initial project generation. In this case, it might be worth exploring whether we can factor the pip arguments out into a standalone gradle file and "import" that file into the top-level one. I'm not especially familiar with Gradle, but a quick search seems to indicate there are a few options for re-using Gradle logic in a way that will be compatible with Briefcase regenerating the part of the file that is imported, without regenerating the "host" file that does the importing.
That sounds like a good path forward to me. |
|
(apologies - I somehow managed to hit a magic key combination that submitted my comment and closed the issue... currently editing my response)... |
|
(Responding to the rest in a standalone comment because of the "hit close accidentally" debacle..)
Completely agreed there's an ambiguity here. However, even "starts with However, as I noted in my previous comment - as long as we've got a way to explicitly say "don't interpret as a file", I think we can call that a win - and the ambiguity over
I think you've correctly understood the general relationship between Although the name of the option is Your usage of
Right - that makes sense. FWIW, I think that's in the territory of "not exactly a bug, but also not an intended use case either". As with my previous comment, the intention isn't that
My apologies - I think we're talking at crossed purposes here. I completely agree with what you've written here - I was advocating for not putting these arguments into |
It seems unlikely that a feature flag would contain a slash, and if it did, I think @freakboy3742's comments above provide a good way of stopping it (or any other argument) from being interpreted as a filename.
Gradle files can contain arbitrary Groovy code, so we could make Briefcase write the pip options to a simple text file, one option per line, and put some code in the template's build.gradle to read the options from the file and pass them to the Can we deal with Flatpak in a similar way? If so, we can avoid doing any "template re-processing whenever requirements would update". |
Yes, that's the core of the |
|
Awesome, thanks for the clarifications all around, and apologies for the mixed up ping @mhsmith. For writing a file like |
Correct - although the template might output an "empty" or "default" version of the file that Briefcase will overwrite so that the "empty template" will continue to work as expected. This will be particular important during the transition period where the template supports the new feature, but Briefcase doesn't (yet).
AFAICT, it should be safe to write the file. If you've got a project generated with Briefcase 0.3.20, and you use Briefcase 0.3.21 to build, then it will write the file, but the project won't pick up the new option. That's not ideal, but it's not entirely surprising as an outcome. FWIW, we do have the ability to define a template epoch - for example, Briefcase 0.3.15 introduced a backwards incompatible change for Android templates, so the template contains this marker, and the Android backend contains the corresponding marker. If you had a project generated with 0.3.14 or earlier, it would raise an error if you tried to build the project with 0.3.15. However, we only really use this capability when we make a major change that is fundamentally incompatible, rather than something like this where and existing project will still be compatible, it just won't be able to use features from the new release. |
No, that issue was implemented pretty much as discussed. For example. you can see that the Android template repository has one branch for each Briefcase version released in the last 2 years. So there's no risk of an old Briefcase version using a new template. And the template But if the user runs |
|
Alright, thanks for clarifying that. Sounds like it should be fine to not worry about it in this case 🙂 To amend my earlier list of tasks for this, template re-processing isn't necessary, so I'll work on PRs to the Flatpak and Android templates to add the new files. I'll probably start with the Flatpak one. |
re: beeware/briefcase#1270 Facilitates configuration of the pip install command arguments by Briefcase or the use of a different package installer altogether without needing to re-process the template or add complex logic to the template (as would be required if the command was templated directly into the manifest).
What is the problem or limitation you are having?
Hi, I have a project which requires PyTorch CPU version, not the default GPU version. Normally, it's possible to install this in one of two ways using pip:
or
where the requirements look like:
Describe the solution you'd like
How do I translate this to the Briefcase environment? This is a question rather than a solution, but I'm new to Beeware and don't see any obvious way of including
--extra-index-urlin therequiressection of thepyproject.tomlfile.Describe alternatives you've considered
Thanks!
Additional context
No response
The text was updated successfully, but these errors were encountered: