Repositories of schema v4 can have arbitrary includes

[deathaxe]

With regards to

thecrawl/lib-spec.md

Line 15 in a42b390

"libraries": [ ... ]

Any repository.json of schema v4 may include other repositories recursively via "includes": ["other-repository.json"].

{
  "$schema": "sublime://packagecontrol.io/schemas/repository",
  "schema_version": "4.0.0",
  "packages": [],
  "libraries": [ ... ],
  "includes": []
}

I haven't investigated the details, but as a general note (If it is already the case than please nevermind): The crawler should be able to fully crawl a repository and extract both, packages and libraries. The default channel for instance just includes a dependencies.json aka libraries.json with libraries into its main repository.json. Beyond that any (3rd-party?) repository may contain both as well. It should resolve any value of "includes": [] and include its content into results.

[kaste]

The registry crawler does indeed do the recursive/nested thing, but here I just grab the one file from packagecontrol/channel because that's the current state of affairs. I made a note about this #72 (comment) if we even want to open this feature (or trap) here, we're not fans of all the sub-repositories either, or if we should forbid this esp. for dependencies as 3rd parties could claim popular Pypi names. Where to discuss these ideas? A sub-page on the Sublime (https://forum.sublimetext.com/) would have been nice.

[deathaxe]

My initial comment on how libraries are crawled, primarily targets the goal to bring them home again to sublimehq/package_control_channel and get rid of the separate packagecontrol/channel repository. thecrawl should at least be prepared to support it, to make that switch effortless easy, once the day comes.

As "includes" are not dedicated to asset based releases or libraries, I'd be fine with this issue as a discussion plattform.

Background about "includes":

Originally PC3 supported a single level of "includes" in the default channel's main repository, only. I guess primarily to drive the "a.json", ... separations.

PC4 added support for arbitrary includes, primarily to consolidate code base - not making any difference between crawling main and sub-repositories implementation wise. It adds some meassures to avoid circular imports and avoid infinite loops.

But this doesn't mean, the official crawler must support this as well. It should for sure take some meassures to keep the infrastructure sane. Hence a strategy to support includes only in default channel's main repository and skip it from any included (3rd-party) repo.json is a reasonable strategy to go.

I've not seen any 3rd-party repository.json making use of "includes" so far and I don't see it being required. Most of them only contain a couple of packages and only sublimelsp/repository defines a library so far.