This CL adds a new way of generating splittings, which clusters files
based on their directory structure (while respecting `set noparent` in
OWNERS files). The user provides a minimum and maximum acceptable
number of files per CL, and the algorithm attempts to create CLs in
that range with a single unique reviewer for each. I've tested it on
some example CLs, and it seems to work well -- certainly better than
the existing algorithm.
Usage of the new algorithm is triggered by passing the
`--target-range` option to `git cl split`. A second new argument,
`--expect-owners-override`, may also be used to ignore OWNERS during
the clustering algorithm.
Bug: 389069356
Change-Id: I321e6424b2bdb7624b370af23b73759fda261161
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6324373
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Devon Loehr <dloehr@google.com>
This CL defines a trie-based datastructure for representing files based
on their path. It directly mirrors the structure of a file system,
keeping track of directories and the files inside them. It also stores
some information about OWNERS files, for use during clustering (we
won't cluster files together if there's a "break" in ownership due to
`set noparent). Optionally, the ownership information can be overridden;
this will be done via a command-line flag when the algorithm is fully
implemented.
Bug: 335797528
Change-Id: I5dcdf36695a1da5714ec021e5e18b6c36855a4f1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6321290
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Devon Loehr <dloehr@google.com>
This CL adds a function which takes a set of files, and attempts to
select a single owner for all of them. If it cannot, it falls back to
the standard owner selection algorithm, which may result in more owners
being chosen than necessary, but guarantees that a valid set of owners
is always returned.
Bug: 389069356
Change-Id: I985804040f149a02bfb5b3c6b946602a81334e7c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6321289
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Devon Loehr <dloehr@google.com>
roll_dep_test is currently using file:///path/to/repo as git url.
Interestingly, the test is not failing in presubmit bot but
failing locally because it is trying to clone using sso protocol
(see: b/391455129#comment11). I think the presubmit bot is not
equipped with git-remote-sso so the tests are able to pass. However,
on a local workstation, git-remote-sso is available but it should
not be used as the file:// scheme should indicate using local
transport protocol. Quoted from
https://git-scm.com/docs/git-clone#_git_urls
> When Git doesn’t know how to handle a certain transport
protocol, it attempts to use the remote-<transport> remote helper,
if one exists.
Removing the file:// scheme appears to fix the issue even though
the only difference is the usage of --local flag according to the
doc.
fwiw, I did try to inject `--local` flag but it gives me a warning
`--local is ignored`. I haven't dig deep enough to figure out why
the flag is ignored.
Bug: 391455129
Change-Id: I8fab9aee5198b2eedfacdb60161197b58f7533ea
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6313297
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Yiwei Zhang <yiwzhang@google.com>
`git cl split` allows users to parameterize their CL descriptions
by including the string $directory, which is replaced by the list of
directories covered by each generated CL. However, as we add more ways
of generating splittings, a list of directories may not always be the
most appropriate description. For example, the --from-file option lets
users write arbitrary description strings.
This CL renames the $directory string to $description, to reflect this
new generality, and refactors all places in the code that expected a
directory list instead of a string. Since users may beused to the
current version, it does not remove $directory, but instead notes that
it is deprecated and emits a warning when it is used. After some time,
we can remove it entirely.
Bug: 389069356
Change-Id: If8c947fdcbbb4897675b015a377cf21123e51467
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6299688
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Devon Loehr <dloehr@google.com>
Currently, the dry run setting causes the script to print out extremely
detailed information about the splitting it generated. While this is
sometimes useful, often users only care about the overall shape of the
splitting. This CL adds a flag which lets them see the same brief
summary that's printed during full runs.
Bug: 389069356
Change-Id: I473740e50f382e63b1289101f3253ff8ae9e6bbe
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6298096
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Python has started warning about invalid escape sequences in strings.
This fixes the warning by escaping the backslash in question. It seems
Python was already (correctly) printing the literal backslash despite
the warning, so no behavioral change.
Bug: None
Change-Id: I2d5d8d56b989d9a1bb87aba69af0ebac0209a7a0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6299490
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Brian Ryner <bryner@google.com>
`git cl split` claims to have built-in auto-resume capabilities, but
the nondeterminism in its splitting algorithm means they don't work
in practice. Now that we have the ability to load splittings from
files, we can ensure the user is using the same splitting as before,
and resume safely.
This CL does two things:
- It checks that any existing split branches match the splitting we're
about to upload, and complains if they don't match. This relies on
the fact that our branch names are unique and deterministic.
- It changes the auto-resume message to mention that --from-file is
required, and includes the relevant filename. This requires tracking
that filename from earlier in the program.
Bug: 389069356
Change-Id: Ic1d8964e96193ca93e05a9a39e286b84ffb61b06
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6280953
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This reverts commit 58625e82c6.
Reason for revert: Several comments on crbug.com/40264739 that this broke their workflow. I plan to revert this and put patchset #2 back up for review, with an opt-in instead of a change to the default behavior.
Original change's description:
> Remove "attempt to squash" behavior from `git rebase-update`
>
> In common workflows, this step only succeeds ~5% of the time, and
> in the other 95% of cases, simply adds ~30 seconds of wasted time
> to the execution time of `rebase-update`. Therefore, this CL removes
> the automated functionality, and leaves squashing as a manual option
> to the user.
>
> Fixed: 40264739
> Bug: 40390274
> Change-Id: Ib2a3ffe4b0e5d0b74323a2a0d34ab0e1b7fafb08
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6104282
> Auto-Submit: Mason Freed <masonf@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Bug: 40390274
Change-Id: I91210a5a21b3751695553433389f346ba443bb1e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6254237
Auto-Submit: Mason Freed <masonf@chromium.org>
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
This CL implements the proposal (from the design doc in the cited bug)
to allow `git cl split` to save the generated splitting to a
human-readable and editable text file. This allows users to save the
splitting in case something goes wrong and they need to retry, and also
lets them tweak a generated splitting if they get something mostly, but
not quite, usable.
It also allows users to edit the generated splitting interactively, like
they do for e.g. CL descriptions during `git cl upload`.
Bug: 389069356
Change-Id: I8ae21f2eb7c022ba181ae5af273ce09605b5acec
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6235727
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This CL changes the way `git cl split` generates branch names during
upload. Specifically, it attempts to derive a common directory for all
the files in each CL, and includes that dirname as part of the branch
name, in addition to a hash of the files in the CL. This makes it a bit
easier to guess what's in a branch, given its name.
Since the hash is still included, branch names should remain unique.
Finding a common directory should be deterministic, so we can still rely
on branches having the same name if the script is re-run with the same
splitting.
Bug: 389069356
Change-Id: I70490258755f13962f3db5c835619caa24176af9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6243377
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
`git cl split` operates by creating a new git branch for each CL
it uploads. It names this branch based on the directory of the files
in the CL. It uses this convention when resuming an interrupted upload:
it skips branches that already exist.
However, this approach is brittle for several reasons:
- The splitting is nondeterministic, so there's no guarantee that CLs
with the same directories will be generated when re-run. Even if they
are, there's no guarantee they'll have exactly the same files.
- For CLs with multiple directories, it uses an arbitrary one, so it
may not choose the same name a second time
- It bakes in an assumption that every CL is defined by the directories
it draws files from.
This CL alleviates this issues by instead naming branches using the hash
of their contained files. This ensures that, with extremely high
probability, branches will have the same name if and only if they have
the same (sorted) list of files. Thus resumed uploads can safely skip
existing branches.
Bug: 389069356
Change-Id: I19bf7cd6ff8b0127f5906384f7f43b3e40ac8fee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6236622
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Devon Loehr <dloehr@google.com>
This adds AffectedFile.UnixLocalPath(), Change.UnixLocalPaths(), and
input_api.UnixLocalPaths() to the presubmit support APIs, so that we can
check paths in a platform-independent manner (i.e., not have to worry
about whether we're running on Windows and might need to handle
backslashes).
Change-Id: I1a424825384cb950ef2d4d3008c9c6e60028cf07
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6236404
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Introduces a new git cl command `squash-closed`. Similar to archive,
this will operate on all closed branches. Unlike archive, rather than
removing these branches and potentially leaving their downstreams in a
wedged state, this will squash the branch and appropriately reparent the
downstream. This makes this a good function to run in preparation of a
`git rebase-update`, especially if there are chained git branches, as
this will then shrink any merged branches to a single commit which
should cleanly apply and drop off.
Bug: 40264739
Change-Id: I1836f944d43f5f3dcebbebf06437bef890da96ca
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6230362
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Alexander Cooper <alcooper@chromium.org>
Auto-Submit: Alexander Cooper <alcooper@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
`git cl split` has the potential to do a lot of operations at once,
which is problematic if it didn't end up splitting things in a way
the user approves of. This changes it to print a summary of the
splitting, and asking the user to review it briefly before proceeding
with the upload.
Bug: 389069356, 40642562, 40269201
Change-Id: I90bead0147badbaa3afcce2264d805ae498f101c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6163905
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Peter Kotwicz <pkotwicz@chromium.org>
Reviewed-by: Andy Perelson <ajp@google.com>
This is a reland of https://crrev.com/c/6168707 with fixes.
The CL adds a new param, unified, to git_diff() and create_diffs()
as a required param.
The functions themselves are tested in presubmit_diffs_test.py, but
their usages in presubmit_support.py were not, and the missing param
caused a failure in CiderG (b/389876151)
This CL basically carries the same patch, but makes the params
optional with a default value and adds unit tests for the usage
in presubmit_support.py
Tested with CiderG: https://screenshot.googleplex.com/PYEDZ3NGn5cYGtV
Change-Id: Ic747c6a133c016dbcd80034e521c76a3db3a3f60
Bug: 389876151, 379902295
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6182703
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Gary Tong <gatong@chromium.org>
Sometimes I want to add a message without touching the attention set.
The default review call will automatically update it regardless. Add
a flag to explicitly control behavior.
Test:
# No attention set option.
$ ./gerrit_client.py ...
-> ignore_automatic_attention_set_rules not included in body of request
$ ./gerrit_client.py ... --automatic-attention
-> body has {'ignore_automatic_attention_set_rules': False}
$ ./gerrit_client.py ... --no-automatic-attention
-> body has {'ignore_automatic_attention_set_rules': True}
Change-Id: Idcf7975ba691942cfe385a2156ef3a3ec064a0b7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6172486
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
When adding tests, the LoadDescription function accidentally got called
with the wrong arguments in the main function. This fixes that; a future
CL (uploading shortly) adds more robust tests that would have caught this.
Bug: 389568463, 389069356
Change-Id: Icaf5e83cd8caa9f3975173f8c8ee7d92ef44ee56
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6170638
Commit-Queue: Andy Perelson <ajp@google.com>
Reviewed-by: Brian Ryner <bryner@google.com>
Reviewed-by: Andy Perelson <ajp@google.com>
Auto-Submit: Devon Loehr <dloehr@google.com>
I frequently find myself running `git cl split` just to see how the
files are going to be split up, without actually intending to submit the
resulting CLs immediately. Doing so requires creating a file, putting a
dummy description in it, and passing that to the command line, which is
unnecessary work. This CL simply allows dry runs to use a dummy
description if none is provided, to make checking the tool's output
easier.
Bug: None
Change-Id: I47e06c6e6da26701e07dcae81ab605edac2e2ca6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6163904
Reviewed-by: Andy Perelson <ajp@google.com>
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Brian Ryner <bryner@google.com>
gclient normally runs hooks in a pseudo terminal which has no
sense of the terminal size. If we want hooks to be able to change
what they output based on the terminal size, we have to explicitly
propagate the terminal size info down through the subprocess
environment.
Change-Id: I08f7c48ef78ea4eb9f5b791abb2a7e5ef8870050
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6161596
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
When running pylint tools in depot_tools itself, we default to the
pylintrc file all the time even though we have versioned ones we
want to use instead. This breaks things like `./pylint-2.17 foo.py`.
Rename the existing pylintrc to 2.6 and duplicate it to 2.7 so we
clear out the 'pylintrc' file and the wrappers always default to the
versioned ones.
We've been riding on pylint being a bit compatible across versions,
but it seems to be accelerating in adding/removing diagnostics and
settings, so this is more churn to add new versions, but we don't
have much choice.
Change-Id: I5c20a41ed8aead3e90b6dda8f5f2fc3b87cfb251
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6150453
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
In common workflows, this step only succeeds ~5% of the time, and
in the other 95% of cases, simply adds ~30 seconds of wasted time
to the execution time of `rebase-update`. Therefore, this CL removes
the automated functionality, and leaves squashing as a manual option
to the user.
Fixed: 40264739
Bug: 40390274
Change-Id: Ib2a3ffe4b0e5d0b74323a2a0d34ab0e1b7fafb08
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6104282
Auto-Submit: Mason Freed <masonf@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
metrics_xml_format.py has a check to ensure that the given xml
is located in the current checkout using GetPrimarySolutionPath().
The problem is that GetPrimarySolutionPath() adds "src" if the path
doesn't end with "src".
https://source.chromium.org/chromium/chromium/tools/depot_tools/+/main:gclient_paths.py;l=78-79
The workspace in Cog doesn't have "src" in the path. Instead, it creates
a workspace at /..../$workspace_name/$repo_name, which doesn't match
GetPrimarySolutionPath() because the workspace root doesn't end with
'src'.
This must be a common problem in other modules, and Cog gets around
this issue by creating a symlink, 'src', to the workspace root.
This CL replaces os.path.abspath() with os.path.realpath() so that
the check is done properly.
Bug: b/369827156
Change-Id: Iaf56de0a9ccbd168004c4c80672c9dd18211bf5e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6094779
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Scott Lee <ddoman@chromium.org>
