[Feature]: auto-discover Java sources bundled in pip installed packages

[T-Dynamos]

If a pip-installable package ships Java source files alongside its Python code, p4a has no way to pick those up automatically. The only supported path is Buildozer's android.add_src, which requires the app developer to add files manually.

Proposed solution

Right after the pip install stage in build.py scan the newly installed top level package directories for a fixed marker folder: android_java/ (for example). If a package contains one, register its path and send to gradle.

Optionally also delete that folder as it's not required/used.

Original idea by @kuzeyron.

CC: @AndreMiras

[kuzeyron]

fixed marker folder, I'd suggest it just uses glob.glob that scans all pip packages for java codes and includes them. Just awkward for the package maintainer to remember to state a fixed path when they have other things to think of while configuring pyproject.toml ('package = ["java/**/*", "java/*"]). glob.glob handles that nicely without complication.

[T-Dynamos]

@kuzeyron pyproject.toml will not exist in install directory.

[kuzeyron]

@kuzeyron pyproject.toml will not exist in install directory.

'package = ["java/**/", "java/"]` is being used when you are uploading to pypi so yeah

[T-Dynamos]

@kuzeyron I see I misread the original message.

Using glob.glob won't be reliable according to me as glob won't be able to determine at which depth folder needs to be included.

Eg: android_java/com/test/Some.java correct include should be android_java in this case but glob might return test.

[kuzeyron]

@kuzeyron I see I misread the original message.

Using glob.glob won't be reliable according to me as glob won't be able to determine at which depth folder needs to be included.

Eg: android_java/com/test/Some.java correct include should be android_java in this case but glob might return test.

Almost all java folders I've seen is like java/org/kivy/android/myjava.java. So that could be standard. Once when that is done I can documentate it.

Could also be without the java folder so the pyproject.toml would just pick up org

[ElliotGarbus]

I’ve started working on a proposal for adding Java source code to a wheel. I’ll share it in a few days, for feedback. It would be ideal if we could align on a common proposed.

I want to include hashing & locking the content and versioning the schema.

[ElliotGarbus]

Here is what I'm thinking. The problem is larger than just adding Java to a wheel. The wheel needs to specify all of its native dependencies. The reader needs to resolve the dependencies into a gradle project.

Native sidecar: letting a PyPI package declare what it needs from the Android build

The problem. pip install delivers a package's Python and drops everything else. [KivMob]is the clearest case: installing it works, but the app developer must hand-copy five settings into buildozer.spec — a Maven coordinate, a custom Maven repo, manifest meta-data, two permissions, and an AndroidX flag. Every app repeats that transcription. Every version bump risks silent drift. And when the package is a transitive dependency, the person on the hook may not know it's in the tree.

The package author knows what the package needs. The app author is the one obliged to say it. This inversion is the problem.

The proposal, mechanically

The package author adds two things to their own pyproject.toml — an entry point, and the module it points to:

# kivmob's pyproject.toml — the library's, not the app's
[project.entry-points."native-integration.v1"]
native = "kivmob._native"

[tool.setuptools.package-data]
kivmob = ["_native/**/*"]

Both ride inside the package's ordinary wheel. Nothing new is published, and no separate artifact exists:

kivmob-2.0.0-py3-none-any.whl
├── kivmob/
│   ├── __init__.py
│   └── _native/
│       ├── __init__.py
│       ├── native.toml                     ← the declaration, at a fixed name
│       └── java/
│           └── org/kivmob/AdBridge.java    ← glue source, compiled by the app's Gradle
└── kivmob-2.0.0.dist-info/
    └── entry_points.txt                    ← [native-integration.v1] native = kivmob._native

The wheel stays py3-none-any, because TOML and .java are text, not platform binaries.

native.toml declares what the Android build must do:

contract = "1"

[android]
java_namespace = "org.kivmob"          # everything this package contributes lives under this

[android.gradle]
dependencies = ["com.google.android.gms:play-services-ads:25.2.0"]

[android.src]
java = ["java"]                        # relative to native.toml

[android.permissions]
uses = ["INTERNET", "ACCESS_NETWORK_STATE"]

At build time the Android tool enumerates installed distributions, reads entry_points.txt from each .dist-info/, maps kivmob._native to kivmob/_native/native.toml via Distribution.locate_file(), and stages what it declares into the generated Gradle project.

the entry_points.txt file is created by the build backend (setuptools, hatchling, flit, …) from the [project.entry-points] table in the package's pyproject.toml, and ships inside the wheel's .dist-info/ directory. It contains metadata that any tool can read through importlib.metadata to discover what a package offers — without importing it. This is all existing packaging infrastructure.

The app author's pyproject.toml gets simplifed :

dependencies = ["kivmob"]

Why a "sidecar"

The declaration travels with the package but stays separate from everything else — which is forced, not stylistic:

• Not in pyproject.toml. That's a build-time input; arbitrary [tool.*] tables never reach site-packages, so no consumer can read them. This is exactly why designs that author the schema there need a custom build backend to copy it into the wheel.
• Not in wheel metadata. There's no field for it, and adding one means amending a packaging standard before anything can be tried.
• Not merged into a shared tree. The tempting design is to have every package write into one common .java/ directory and let pip do the merging. That destroys provenance at install time — once overlaid, you cannot tell which package contributed which file, and everything below depends on knowing that.

Key advantages

• It composes. pip install and the native half comes too. No README archaeology.
• Provenance survives. Every contribution is attributable, so diagnostics name the package that caused the problem.
• Reviewable. The build tool records each package's resolved contribution, so a version bump shows up as + RECEIVE_BOOT_COMPLETED in a pull request rather than as an opaque hash change.
• The app keeps authority. A package may request a permission or register a component, but only the app may mark a feature required or a component exported. A dependency can't silently shrink your Play device reach or open an IPC surface.
• Namespace ownership. A package declares a Java package prefix and may only write under it. Bootstrap prefixes are reserved, so no dependency can replace PythonActivity.
• No new packaging machinery. Entry points plus static package data — works with setuptools, hatchling, flit, pdm, maturin. No custom build backend, no PEP required to start.

Scope

"Why not just publish the Java to Maven?" You should, for anything substantial — Maven coordinates are the recommended path, and the sidecar's main job is letting a package declare the coordinate instead of making the app author transcribe it. But KivMob shows the cost: they published org.kivmob:kivmob-android-bridge separately, and it now versions independently of the Python package it exists to serve. For a single glue class, shipping .java source keeps both halves in one artifact that can't skew. The example above shows how this code can be shipped as source in the wheel.

Deliberately out of scope: prebuilt .aars (they carry their own manifest and merge into yours — a binary nobody reviews contributing permissions and exported components).

Next Steps

• Continue the discussion here, I'd be very interested in feedback. Is the declaration set right, does the app-keeps-authority split lands where it should, and — for anyone maintaining another toolchain — whether this is something you'd plausibly read.

• I'll create a separate repo that includes the spec (a full fleshed out version of the ideas here), a reference "reader", and a compliance test suite. I will let you know when this is done. Expect within 2 weeks.

• IOS has similar issues to the ones described here, the full spec will comprehend both IOS and Andriod.

• Invite the Briefcase and psproject developers to join the discussion. python-for-android, KivyForge, Briefcase and psproject have the identical gap, and a convention only one tool reads isn't worth defining. I'm hoping can build alignment and demonstrate value that could end in a python mobile packaging standard.

@Novfensec

[Novfensec]

You may wanna look at https://github.com/kivy-school/ksp-builder

And the packages using this build system that are explicitly focused for ksproject

https://github.com/PyPlatformPackages/PyGMA

https://youtu.be/mWzGmjfY3iw?si=-UaGV_mdfMFhgF3y

https://github.com/PyPlatformPackages/PyOneSignal

[ElliotGarbus]

@Novfensec
I did review ksp-builder. There are some very clever ideas, and importantly it solves this problem-end to end.

My concerns with ksp-builder/ksproject:

• Merge by filesystem overlay. Every package writes into one shared site_packages//.java/ tree and pip does the merging for free. Elegant, but it destroys provenance at install time — after which collision detection, per-package hashing, a review gate, and "which package added this permission?" all become impossible. Concretely: pip_install.py uses shutil.copytree(..., dirs_exist_ok=True) with the project copied first, so a local dependency overwrites the app's own Java; and the generated Gradle Copy stages site-packages .java/ into src/main/java/ after the bootstrap is written, so a package shipping .java/org/kivy/android/PythonActivity.java silently replaces the app's Activity.

• The backend is the transform. Nice authoring ergonomics, but it needs a wrapper per build backend forever — flit, pdm, maturin, scikit-build are all excluded. Static package data plus an entry point gets the same result with no backend involvement.

Some smaller issues:

• No schema version in the JSON
• Permissions auto-merge with no gate
• No <uses-feature>, so a camera or BLE plugin silently shrinks Play device reach
• .libs// encodes the ABI as a subdirectory, which only makes sense in a fat py3-none-any wheel and contradicts PEP 738 tagging.

I think the ksp-builder approach has a number of good ideas, and it attacking the same problem - I'm hoping together we can develop a more robust solution for the eco-system.

[ElliotGarbus]

As promised — the spec is ready: https://github.com/ElliotGarbus/native-integration

Full spec (Android + iOS), a reference, and ten worked examples against real packages — including PyOneSignal and PyGMA from @Novfensec's links above, plus examples/pystripe/ showing both halves — package declaration and app reply — side by side.

@T-Dynamos @kuzeyron — would value your eyes on it, especially whether it fits how p4a actually resolves dependencies.
@Novfensec @psychowasp - You have real experience from ksp-builder, what do you think?

Don't start with the spec — it's ~17k words.
README first, then
one example:
pystripe
to see both halves of one integration,
Appendix D
is a one-line description of every key if you'd rather scan the surface.

[ElliotGarbus]

I created a discussion in the spec repo: ElliotGarbus/native-integration#1. this might be a more appropriate spot for discussions on the spec.

I received a substantive response over on Briefcase's repo, and it raised some genuinely interesting design threads: whether this should eventually cover macOS (sandboxed app permissions have the same shape as iOS's), an overlap with how Briefcase already handles web-deploy content contributions from packages, and even a possible longer-term path toward a PEP. Worth a read: beeware/briefcase#2999

Still early days — more feedback expected as people dig into the actual spec.

[Novfensec]

https://github.com/ElliotGarbus/native-integration/blob/main/development/examples/

Yeah I also think there must be standard to follow like PEP for this.
Quite deep research though, and I am absolutely in for anything that can handle these integrations in a more better way.

One more similar package https://github.com/Novfensec/nfswebview