This reverts commit eb60ab38de.
Reason for revert: re-landing with an additional patch
-------------
* Problem
Browser infra runs ci.*-presubmit builders, such as linux-presubmit,
with --all to ensure that the entire chromium/src passes
presubmit checks.
crrev.com/c/6842238 changed the finding type for License Check
from warning to error, but the CI presubmit builders failed because
there are many files without valid CopyRight.
Not only the existing files, all the new files that are added with
`Bypass-Check-License: <reason> footer could also cause the presubmit
builder to fail.
* This CL
In addition to the original patch from crrev.com/c/6842238,
this CL makes additional patches to turn the CopyRight errors into
warnings, if --file or --all is given.
The same approach is used in
https://source.chromium.org/chromium/chromium/tools/depot_tools/+/main:presubmit_canned_checks.py;l=982-987;drc=eb60ab38deeda6975c9b0fef883978f2a9f69120
Bug: 435696543,40237859
Original change's description:
> Revert "presubmit: emit errors instead of warnings for bad copyright headers"
>
> This reverts commit fa62515ecb.
>
> Reason for revert: it seems that the existing files without valid copyright headers are causing linux-presubmit builds to fail. b/438791294
>
> Bug: 435696543,40237859
> Original change's description:
> > presubmit: emit errors instead of warnings for bad copyright headers
> >
> > This is an effective revert of https://crrev.com/c/4895337 with
> > additional patches to support a footer.
> >
> > https://crrev.com/c/3887721 updated CheckLicense() to emit errors
> > for bad copyright headers. However, https://crrev.com/c/4895337
> > was changed the finding type from error to warning, claiming that
> > the check is N/A for moved third files, but it's not so easy
> > to programatiically distinguish moved third-party files.
> >
> > After discussions, it was decided to change the finding type back
> > to error to prevent accidental submissions for new files without
> > correct CopyRight headers.
> >
> > To mitigate moved, third-party files, this CL adds support for
> > "Bypass-Check-License: <reason>" footer.
> >
> > If the check should be ignored in a given CL, CL authors should
> > use the footer instead.
> >
> > Bug: 435696543,40237859
> > Change-Id: I177915c65932a3d76ea60ee6a0e396f726bc400d
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6842238
> > Reviewed-by: Gavin Mak <gavinmak@google.com>
> > Commit-Queue: Scott Lee <ddoman@chromium.org>
>
> Bug: 435696543,40237859
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Change-Id: Ibedf8d13e3742249947e29e625a14cceaf89879c
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6852377
> Commit-Queue: Scott Lee <ddoman@chromium.org>
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
Bug: 435696543,40237859
Change-Id: Iafdb29b928c016eb3949e29fd43a2ba5f53e0ba0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6852108
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Scott Lee <ddoman@chromium.org>
This reverts commit fa62515ecb.
Reason for revert: it seems that the existing files without valid copyright headers are causing linux-presubmit builds to fail. b/438791294
Bug: 435696543,40237859
Original change's description:
> presubmit: emit errors instead of warnings for bad copyright headers
>
> This is an effective revert of https://crrev.com/c/4895337 with
> additional patches to support a footer.
>
> https://crrev.com/c/3887721 updated CheckLicense() to emit errors
> for bad copyright headers. However, https://crrev.com/c/4895337
> was changed the finding type from error to warning, claiming that
> the check is N/A for moved third files, but it's not so easy
> to programatiically distinguish moved third-party files.
>
> After discussions, it was decided to change the finding type back
> to error to prevent accidental submissions for new files without
> correct CopyRight headers.
>
> To mitigate moved, third-party files, this CL adds support for
> "Bypass-Check-License: <reason>" footer.
>
> If the check should be ignored in a given CL, CL authors should
> use the footer instead.
>
> Bug: 435696543,40237859
> Change-Id: I177915c65932a3d76ea60ee6a0e396f726bc400d
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6842238
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Scott Lee <ddoman@chromium.org>
Bug: 435696543,40237859
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Change-Id: Ibedf8d13e3742249947e29e625a14cceaf89879c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6852377
Commit-Queue: Scott Lee <ddoman@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This is an effective revert of https://crrev.com/c/4895337 with
additional patches to support a footer.
https://crrev.com/c/3887721 updated CheckLicense() to emit errors
for bad copyright headers. However, https://crrev.com/c/4895337
was changed the finding type from error to warning, claiming that
the check is N/A for moved third files, but it's not so easy
to programatiically distinguish moved third-party files.
After discussions, it was decided to change the finding type back
to error to prevent accidental submissions for new files without
correct CopyRight headers.
To mitigate moved, third-party files, this CL adds support for
"Bypass-Check-License: <reason>" footer.
If the check should be ignored in a given CL, CL authors should
use the footer instead.
Bug: 435696543,40237859
Change-Id: I177915c65932a3d76ea60ee6a0e396f726bc400d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6842238
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Scott Lee <ddoman@chromium.org>
The current gsutil download code silently fails when the connection
drops mid-download, as read() returns an empty buffer instead of raising
an exception. This may lead to errors such as "zipfile.BadZipFile: File
is not a zip file" on Chromium sync with freshly-bootstrapped
depot_tools when downloading gcs deps.
This change solves this by hardening the process:
- Use retry mechanism with exponential backoff for gsutil download
- Switch to urlretrieve, which looks at Content-Length
- Compare MD5 of the downloaded file with the value from API
- Move exponential_backoff_retry from git_cache.py to gclient_utils.py
Change-Id: I25242948399e01373eb2afd9352e5c78a889051d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6485485
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Auto-Submit: Aleksei Khoroshilov <akhoroshilov@brave.com>
Reviewed-by: Scott Lee <ddoman@chromium.org>
The current implementation relies on the current working directory,
when it traverses the file tree to find the nearest OWNERS. It also
causes an infinite loop when it cannot find any OWNERS.
This CL changes the implementation so that it works
no matter what the cwd is.
Bug: 412904761
Change-Id: Ic4e25217aa64bd2eb6514ccdd486fe3b57a82312
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6484531
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
These tests get broken by If0a214fe074d7e7591b5d37417ca447525c983b6 so
delete them.
Now, you might raise your eyebrows at that, but hear me out.
These tests are so heavily mocked that I argue that they aren't
providing any value. What's the point of tests? To save time by
catching regressions early.
Being dumped a large list of expected mocked calls like:
@10: (['os.path.isfile', 'TEMP_DIR/trace-packet'],)
@11: (['FileRead', 'TEMP_DIR/trace-packet'],)
@12: (['FileWrite', 'TEMP_DIR/trace-packet', 'git-hash: 012345\ngit-hash: abcdea\n'],)
@13: (['make_archive', 'TRACES_DIR/20170316T200041.000000-traces', 'zip', 'TEMP_DIR'],)
@14: (['FileWrite', 'TEMP_DIR/git-config', <ANY>],)
provides little value. Like, what the heck is this even supposed to
be testing?
I argue that:
the cost of potentially having a regression in the future, fixing it,
and adding a new, sane test for that case
is LESS than:
the cost of someone looking at one of these tests failures and
trying to figure out what exactly broke, whether it is an actual
breakage or do they just need to wade into the mess of expected mocked
calls and figure out how to update the mocked calls to reflect a
perfectly fine change that they are trying to make
Bug: 404613530
Change-Id: Id8e2f5ebb9ad55ee32291c2f70b2d26fb479b195
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6428049
Reviewed-by: Gregory Nisbet <gregorynisbet@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Allen Li <ayatane@chromium.org>
Attempt to update the README.chromium file with the new submodule
revision after rolling it in a DEPS update. This update is flag-guarded
and will only run when explicitly specified.
Currently, only README.chromium files with a single "Revision:" line are
supported. Multiple lines and files with the divider are not handled.
These are left as future TODOs.
Bug: 390067679
Change-Id: Ib776564ae94360cc72dd633fc7ed7b3f84b5b9d2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6173767
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Jordan Brown <rop@google.com>
Reviewed-by: Jiewei Qian <qjw@chromium.org>
The existing licence regex assumed each line would start with an unknown
and optional comment delimiter, followed by a required space. However
the standard way to write the licence in XML files uses a multi-line
comment, each individual line simply starts with the licence text, no
delimiter or space. The license check thus needed to be updated to make
the space optional as well.
Bug: 407060754
Change-Id: I4a83dd9408de890593952e441ba6b650f726907c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6409392
Commit-Queue: Sky Malice <skym@chromium.org>
Reviewed-by: Andy Perelson <ajp@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Make this available for testing first.
This should be functionally roughly analogous with the current auto
configuration logic, and it prints all actions it performs so it
should be pretty safe to play with, even if it hypothetically
misbehaves.
Bug: b/401338175
Change-Id: I803c7e167e355ec8cca1f5959099138c2fee305e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6334614
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Allen Li <ayatane@chromium.org>
This CL adds a new way of generating splittings, which clusters files
based on their directory structure (while respecting `set noparent` in
OWNERS files). The user provides a minimum and maximum acceptable
number of files per CL, and the algorithm attempts to create CLs in
that range with a single unique reviewer for each. I've tested it on
some example CLs, and it seems to work well -- certainly better than
the existing algorithm.
Usage of the new algorithm is triggered by passing the
`--target-range` option to `git cl split`. A second new argument,
`--expect-owners-override`, may also be used to ignore OWNERS during
the clustering algorithm.
Bug: 389069356
Change-Id: I321e6424b2bdb7624b370af23b73759fda261161
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6324373
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Devon Loehr <dloehr@google.com>
This CL defines a trie-based datastructure for representing files based
on their path. It directly mirrors the structure of a file system,
keeping track of directories and the files inside them. It also stores
some information about OWNERS files, for use during clustering (we
won't cluster files together if there's a "break" in ownership due to
`set noparent). Optionally, the ownership information can be overridden;
this will be done via a command-line flag when the algorithm is fully
implemented.
Bug: 335797528
Change-Id: I5dcdf36695a1da5714ec021e5e18b6c36855a4f1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6321290
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Devon Loehr <dloehr@google.com>
This CL adds a function which takes a set of files, and attempts to
select a single owner for all of them. If it cannot, it falls back to
the standard owner selection algorithm, which may result in more owners
being chosen than necessary, but guarantees that a valid set of owners
is always returned.
Bug: 389069356
Change-Id: I985804040f149a02bfb5b3c6b946602a81334e7c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6321289
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Devon Loehr <dloehr@google.com>
roll_dep_test is currently using file:///path/to/repo as git url.
Interestingly, the test is not failing in presubmit bot but
failing locally because it is trying to clone using sso protocol
(see: b/391455129#comment11). I think the presubmit bot is not
equipped with git-remote-sso so the tests are able to pass. However,
on a local workstation, git-remote-sso is available but it should
not be used as the file:// scheme should indicate using local
transport protocol. Quoted from
https://git-scm.com/docs/git-clone#_git_urls
> When Git doesn’t know how to handle a certain transport
protocol, it attempts to use the remote-<transport> remote helper,
if one exists.
Removing the file:// scheme appears to fix the issue even though
the only difference is the usage of --local flag according to the
doc.
fwiw, I did try to inject `--local` flag but it gives me a warning
`--local is ignored`. I haven't dig deep enough to figure out why
the flag is ignored.
Bug: 391455129
Change-Id: I8fab9aee5198b2eedfacdb60161197b58f7533ea
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6313297
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Yiwei Zhang <yiwzhang@google.com>
`git cl split` allows users to parameterize their CL descriptions
by including the string $directory, which is replaced by the list of
directories covered by each generated CL. However, as we add more ways
of generating splittings, a list of directories may not always be the
most appropriate description. For example, the --from-file option lets
users write arbitrary description strings.
This CL renames the $directory string to $description, to reflect this
new generality, and refactors all places in the code that expected a
directory list instead of a string. Since users may beused to the
current version, it does not remove $directory, but instead notes that
it is deprecated and emits a warning when it is used. After some time,
we can remove it entirely.
Bug: 389069356
Change-Id: If8c947fdcbbb4897675b015a377cf21123e51467
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6299688
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Devon Loehr <dloehr@google.com>
Currently, the dry run setting causes the script to print out extremely
detailed information about the splitting it generated. While this is
sometimes useful, often users only care about the overall shape of the
splitting. This CL adds a flag which lets them see the same brief
summary that's printed during full runs.
Bug: 389069356
Change-Id: I473740e50f382e63b1289101f3253ff8ae9e6bbe
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6298096
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Python has started warning about invalid escape sequences in strings.
This fixes the warning by escaping the backslash in question. It seems
Python was already (correctly) printing the literal backslash despite
the warning, so no behavioral change.
Bug: None
Change-Id: I2d5d8d56b989d9a1bb87aba69af0ebac0209a7a0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6299490
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Brian Ryner <bryner@google.com>
`git cl split` claims to have built-in auto-resume capabilities, but
the nondeterminism in its splitting algorithm means they don't work
in practice. Now that we have the ability to load splittings from
files, we can ensure the user is using the same splitting as before,
and resume safely.
This CL does two things:
- It checks that any existing split branches match the splitting we're
about to upload, and complains if they don't match. This relies on
the fact that our branch names are unique and deterministic.
- It changes the auto-resume message to mention that --from-file is
required, and includes the relevant filename. This requires tracking
that filename from earlier in the program.
Bug: 389069356
Change-Id: Ic1d8964e96193ca93e05a9a39e286b84ffb61b06
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6280953
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
