Multiple Repository Management¶
This page introduces basic concepts related to west and its multiple repository management features, and gives an overview of the associated commands. See History and Motivation and Zephyr issue #6770 for additional discussion, rationale, and motivation.
Note
West’s multi-repo commands are meant to augment Git in minor ways for multi-repo work, not to replace it. For tasks that only operate on one repository, just use plain Git commands.
Introduction¶
West’s built-in commands allow you to work with projects composed of multiple Git repositories installed under a common parent directory, which we call a west workspace (before west 0.7, this was called a west installation). This works similarly to Git Submodules and Google’s repo.
A west workspace is the result of running the west init
command, which
is described in more detail below. For upstream Zephyr, the workspace looks
like this:
zephyrproject/ # west topdir
├── .west/
│ └── config
├── zephyr/ # .git/ │
│ ├── west.yml # manifest │ never modified by west
│ └── [... other files ...] │
├── modules/
│ └── lib/
│ └── tinycbor/ # .git/ project
├── net-tools/ # .git/ project
└── [ ... other projects ...]
Above, zephyrproject
is the name of the west workspace’s root
directory. This name is just an example – it could be anything, like z
,
my-zephyr-workspace
, etc. The file .west/config
is the
workspace’s local configuration file.
Every west workspace contains exactly one manifest repository, which is a
Git repository containing a file named west.yml
, which is the west
manifest. The location of the manifest repository is given by the
manifest.path configuration option in the local
configuration file. The manifest file, along with west’s configuration files,
controls the workspace’s behavior. For upstream Zephyr, zephyr
is
the manifest repository, but you can configure west to use any Git repository
in the workspace as the manifest repository. The only requirement is that it
contains a valid manifest file. See West Manifests for more details on
what this means.
Both of the tinycbor
and net-tools
directories are projects
managed by west, and configured in the manifest file. A west workspace can
contain arbitrarily many projects. As shown above, projects can be located
anywhere in the workspace. They don’t have to be subdirectories of the
manifest directory, and they can be inside of arbitrary subdirectories inside
the workspace’s root directory. By default, the Zephyr build system uses
west to get the locations of all the projects in the workspace, so any code
they contain can be used by applications. This behavior can be overridden using
the ZEPHYR_MODULES
CMake variable; see Modules (External projects) for details.
Finally, any repository managed by a west workspace can contain extension commands, which are extra west commands provided by that project. This includes the manifest repository and any project repository.
Topologies supported¶
The following are example source code topologies supported by west.
T1: star topology, zephyr is the manifest repository
T2: star topology, a Zephyr application is the manifest repository
T3: forest topology, freestanding manifest repository
T1: Star topology, zephyr is the manifest repository¶
The zephyr repository acts as the central repository and specifies its Modules (External projects) in its
west.yml
Analogy with existing mechanisms: Git submodules with zephyr as the super-project
This is the default. See Introduction for how mainline Zephyr is an example of this topology.
T2: Star topology, application is the manifest repository¶
Useful for those focused on a single application
A repository containing a Zephyr application acts as the central repository and names other projects required to build it in its
west.yml
. This includes the zephyr repository and any modules.Analogy with existing mechanisms: Git submodules with the application as the super-project, zephyr and other projects as submodules
A workspace using this topology looks like this:
west-workspace/
│
├── application/ # .git/ │
│ ├── CMakeLists.txt │
│ ├── prj.conf │ never modified by west
│ ├── src/ │
│ │ └── main.c │
│ └── west.yml # main manifest with optional import(s) and override(s)
│ │
├── modules/
│ └── lib/
│ └── tinycbor/ # .git/ project from either the main manifest or some import.
│
└── zephyr/ # .git/ project
└── west.yml # This can be partially imported with lower precedence or ignored.
# Only the 'manifest-rev' version can be imported.
Here is an example application/west.yml
which uses
Manifest Imports, available since west 0.7, to import Zephyr v2.2.0
and its modules into the application manifest file:
# Example T2 west.yml, using manifest imports.
manifest:
remotes:
- name: zephyrproject-rtos
url-base: https://github.com/zephyrproject-rtos
projects:
- name: zephyr
remote: zephyrproject-rtos
revision: v2.2.0
import: true
self:
path: application
You can still selectively “override” individual Zephyr modules if you use
import:
in this way; see Example 1.3: Downstream of a Zephyr release, with module fork for an example.
Another way to do the same thing is to copy/paste zephyr/west.yml
to application/west.yml
, adding an entry for the zephyr
project itself, like this:
# Equivalent to the above, but with manually maintained Zephyr modules.
manifest:
remotes:
- name: zephyrproject-rtos
url-base: https://github.com/zephyrproject-rtos
defaults:
remote: zephyrproject-rtos
projects:
- name: zephyr
revision: v2.2.0
west-commands: scripts/west-commands.yml
- name: net-tools
revision: some-sha-goes-here
path: tools/net-tools
# ... other Zephyr modules go here ...
self:
path: application
(The west-commands
is there for Building, Flashing and Debugging and other
Zephyr-specific Extensions. It’s not necessary when using
import
.)
The main advantage to using import
is not having to track the revisions of
imported projects separately. In the above example, using import
means
Zephyr’s module versions are automatically determined from the
zephyr/west.yml
revision, instead of having to be copy/pasted (and
maintained) on their own.
T3: Forest topology¶
Useful for those supporting multiple independent applications or downstream distributions with no “central” repository
A dedicated manifest repository which contains no Zephyr source code, and specifies a list of projects all at the same “level”
Analogy with existing mechanisms: Google repo-based source distribution
A workspace using this topology looks like this:
west-workspace/
├── app1/ # .git/ project
│ ├── CMakeLists.txt
│ ├── prj.conf
│ └── src/
│ └── main.c
├── app2/ # .git/ project
│ ├── CMakeLists.txt
│ ├── prj.conf
│ └── src/
│ └── main.c
├── manifest-repo/ # .git/ never modified by west
│ └── west.yml # main manifest with optional import(s) and override(s)
├── modules/
│ └── lib/
│ └── tinycbor/ # .git/ project from either the main manifest or
│ # frome some import
│
└── zephyr/ # .git/ project
└── west.yml # This can be partially imported with lower precedence or ignored.
# Only the 'manifest-rev' version can be imported.
Here is an example T3 manifest-repo/west.yml
which uses
Manifest Imports, available since west 0.7, to import Zephyr
v2.2.0 and its modules, then add the app1
and app2
projects:
manifest:
remotes:
- name: zephyrproject-rtos
url-base: https://github.com/zephyrproject-rtos
- name: your-git-server
url-base: https://git.example.com/your-company
defaults:
remote: your-git-server
projects:
- name: zephyr
remote: zephyrproject-rtos
revision: v2.2.0
import: true
- name: app1
revision: SOME_SHA_OR_BRANCH_OR_TAG
- name: app2
revision: ANOTHER_SHA_OR_BRANCH_OR_TAG
self:
path: manifest-repo
You can also do this “by hand” by copy/pasting zephyr/west.yml
as shown above for the T2 topology, with the same caveats.
West Structure¶
West’s code is distributed via PyPI in a Python package named west
. See
West APIs for API documentation.
This distribution also includes a launcher executable, also named west
.
When west is installed, the launcher is placed by pip3
somewhere in the
user’s filesystem (exactly where depends on the operating system, but should be
on the PATH
environment variable). This launcher is the
command-line entry point.
The manifest-rev
branch¶
West creates a branch named manifest-rev
in each project, pointing to the
commit the project’s revision resolves to. The branch is updated whenever
project data is fetched by west update
. Other multi-repo commands also use
manifest-rev
as a reference for the upstream revision as of the most recent
update. See Multi-Repo Commands, below, for more information.
manifest-rev
is a normal Git branch, but if you delete or otherwise modify
it, west will recreate and/or reset it as if with git reset --hard
on the
next update (though git update-ref
is used internally). For this reason, it
is normally a bad idea to modify it yourself. manifest-rev
was added to
allow SHAs as project revisions in the manifest, and to give a consistent
reference for the current upstream revision regardless of how the manifest
changes over time.
Note
West does not create a manifest-rev
branch in the manifest repository,
since west does not manage the manifest repository’s branches or revisions.
Multi-Repo Commands¶
This section gives a quick overview of the multi-repo commands, split up by
functionality. Some commands loosely mimic the corresponding Git command, but
in a multi-repo context (e.g. west diff
shows local changes on all
repositories).
Project arguments can be the names of projects in the manifest, or (as
fallback) paths to them. Omitting project arguments to commands which accept a
list of projects (such as west list
, west forall
, etc.) usually
defaults to using all projects in the manifest file plus the manifest
repository itself.
For help on individual commands, run west <command> -h
(e.g. west diff
-h
).
Main Commands¶
The west init
and west update
multi-repo commands are the most
important to understand.
west init [-l] [-m URL] [--mr REVISION] [PATH]
: create a west workspace in directoryPATH
(i.e..west
etc. will be created there). If thePATH
argument is not given, the current working directory is used. This command does not clone any of the projects in the manifest; that is done the next timewest update
is run.This command can be invoked in two ways:
If you already have a local clone of the zephyr repository and want to create a west workspace around it, you can use the
-l
switch to pass its path to west, as in:west init -l path/to/zephyr
. This is the only reason to use-l
.Otherwise, omit
-l
to create a new workspace from a remote manifest repository. You can give the manifest URL using the-m
switch, and its revision using--mr
. For example, invoking west with:west init -m https://github.com/zephyrproject-rtos/zephyr --mr v1.15.0
would clone the upstream official zephyr repository at the tagged release v1.15.0 (-m
defaults to https://github.com/zephyrproject-rtos/zephyr, and the-mr
default is overridden tov1.15.0
).
west update [--fetch {always,smart}] [--rebase] [--keep-descendants] [PROJECT ...]
: clone and update the specified projects based on the current west manifest.By default, this command:
Parses the manifest file,
west.yml
Clones any project repositories that are not already present locally
Fetches any project revisions in the manifest file which are not already pulled from the remote
Sets each project’s manifest-rev branch to the current manifest revision
Checks out those revisions in local working trees
To operate on a subset of projects only, specify them using the
PROJECT
positional arguments, which can be either project names as given in the manifest file, or paths to the local project clones.To force this command to fetch from project remotes even if the revisions appear to be available locally, either use
--fetch always
or set theupdate.fetch
configuration option to"always"
.For safety,
west update
usesgit checkout --detach
to check out a detachedHEAD
at the manifest revision for each updated project, leaving behind any branches which were already checked out. This is typically a safe operation that will not modify any of your local branches. See the help for the--rebase
/-r
and--keep-descendants
/-k
options for ways to influence this.
Miscellaneous Commands¶
West has a few more commands for managing the multi-repo, which are briefly
discussed here. Run west <command> -h
for detailed help.
west forall -c COMMAND [PROJECT ...]
: Runs the shell commandCOMMAND
within the top-level repository directory of each of the specified projects (default: all cloned projects). IfCOMMAND
consists of more than one word, it must be quoted to prevent it from being split up by the shell.To run an arbitrary Git command in each project, use something like
west forall -c 'git <command> --options'
. Note thatwest forall
can be used to run any command, though, not just Git commands.west help <command>
: this is equivalent towest <command> -h
.west list [-f FORMAT] [PROJECT ...]
: Lists project information from the manifest file, such as URL, revision, path, etc. The printed information can be controlled using the-f
option.west manifest
: Manipulates manifest files. See Manifest Command.west manifest --validate
: Ensure the current manifest file is well-formed. Print information about what’s wrong and fail the process in case of error.west diff [PROJECT ...]
: Runs a multi-repogit diff
for the specified projects.west status [PROJECT ...]
: Likewest diff
, for runninggit status
.west topdir
: Prints the top directory of the west workspace.
Private repositories¶
You can use west to fetch from private repositories. There is nothing west-specific about this.
The west update
command essentially runs git fetch YOUR_PROJECT_URL
when a project’s manifest-rev
branch must be updated to a newly fetched
commit. It’s up to your environment to make sure the fetch succeeds.
You can either enter the password manually or use any of the credential helpers built in to Git. Since Git has credential storage built in, there is no need for a west-specific feature.
The following sections cover common cases for running west update
without
having to enter your password, as well as how to troubleshoot issues.
Fetching via HTTPS¶
On Windows when fetching from GitHub, recent versions of Git prompt you for your GitHub password in a graphical window once, then store it for future use (in a default installation). Passwordless fetching from GitHub should therefore work “out of the box” on Windows after you have done it once.
In general, you can store your credentials on disk using the “store” git credential helper. See the git-credential-store manual page for details.
To use this helper for all the repositories in your workspace, run:
west forall -c "git config credential.helper store"
To use this helper on just the projects foo
and bar
, run:
west forall -c "git config credential.helper store" foo bar
To use this helper by default on your computer, run:
git config --global credential.helper store
On GitHub, you can set up a personal access token to use in place of your account password. (This may be required if your account has two-factor authentication enabled, and may be preferable to storing your account password in plain text even if two-factor authentication is disabed.)
If you don’t want to store any credentials on the file system, you can store them in memory temporarily using git-credential-cache instead.
Fetching via SSH¶
If your SSH key has no password, fetching should just work. If it does have a password, you can avoid entering it manually every time using ssh-agent.
On GitHub, see Connecting to GitHub with SSH for details on configuration and key creation.
Troubleshooting¶
One good way to troubleshoot fetching issues is to run west update
in
verbose mode, like this:
west -v update
The output includes Git commands run by west and their outputs. Look for something like this:
=== updating your_project (path/to/your/project):
west.manifest: your_project: checking if cloned
[...other west.manifest logs...]
--- your_project: fetching, need revision SOME_SHA
west.manifest: running 'git fetch ... https://github.com/your-username/your_project ...' in /some/directory
The git fetch
command example in the last line above is what needs to
succeed. Go to /some/directory
, copy/paste and run the entire git fetch
command, then debug from there using the documentation for your credential
storage helper.
If you can get the git fetch
command to run successfully without prompting
for a password when you run it directly, you will be able to run west
update
without entering your password in that same shell.