Since fetches involve multiple subprocess calls, any of which can be
slow, the per-subprocess caffeination strategy does not seem suitable --
the Mac might sleep as soon as the wake lock is dropped, before it
starts a new one. This instead implements a context manager to allow
caffeinating a scope.
To allow flag control, caffeinate.scope takes an argument that decides
whether or not it should actually do anything useful; it looks silly,
but the alternative is to interfere with flag parsing more or to require
users to write separate codepaths to decide whether to enter the context
manager scope or not; the "use the context manager in a mode where it
does not do anything" prevents this.
Bug: 462507017
Change-Id: Icc5bb9cadda30b5a120f112b10bf96ffd3b6550f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7183647
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Adam Norberg <norberg@google.com>
The diff hunk can omit the number of lines after the line position.
For example, the following hunks are all valid
- @@ -1,2 +1,3 @@
- @@ -1 +1,3 @@
- @@ -1,2 +1 @@
- @@ -1 +1 @@
However, the existing regex assumes that the number of lines is always
given. It wasn't a problem before crrev.com/c/6952890, because -U3 was
applied always. However, since it changed the context lines to -U0,
git diff started generating hunks without the number of lines, if
a given changed section contains only a single line.
This CL fixes the regex.
Bug: 453641840
Change-Id: I67f680bc1d8fa046e373b065e8ccbb6ce652eb3f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7158599
Auto-Submit: Scott Lee <ddoman@chromium.org>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Scott Lee <ddoman@chromium.org>
This CL implements ensure_authenticated() for GitCredsAuthenticator, and
adds ReAuth support.
This check performs best-effort credential checks before performing
expensive operations such as presubmit hooks. This allows us to fail fast when an ReAuth token is missing for a contributor subject to
ReAuth requirements.
If you suspect this CL has caused a breakage on CL upload, please
revert.
Bug: 451651615
Change-Id: I42f3ffe3c830ed7bd2322f6cf2ad8adbba7e7251
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7036772
Reviewed-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Jiewei Qian <qjw@chromium.org>
This CL makes it possible for ensure_authenticated to check whether
the current credential would satisfy ReAuth requirement.
Right now, only GitCredsAuthenticator performs additional ReAuth
check. For other Authenticator, either ReAuth is not relevant
(e.g. SSO), or not supported or implied (e.g. .gitcookies).
This CL does nothing on its own, because ensure_autheticated with
ReAuth is opt-in. Subsequent CL will update callsites to pass in
ReAuthContext to make the check effective.
Bug: 451651615
Change-Id: Idb1e70c52aaa844e672331fad8d462a2a7577d45
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7104363
Commit-Queue: Jiewei Qian <qjw@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>
This CL refactors _test_gerrit_ensure_authenticated_common by splitting
it into:
- The actual mock and common code path
- Test case specific setup (e.g. scm.GIT.SetConfig)
In addition, fixes a few issues with the existing code:
- _Authenticator.get() is now mocked correctly, so a CookiesAuthenticator
is returned in relevant tests. Previously it returned the "production"
ChainedAuthenticator, which could call other authenticators (such as
GitCredsAuthenticator based on the runtime environment)
- Adds mock patch cleanup functions. Previously, cleanup functions
aren't registered, so the mock leaks and got overwritten by other test
cases.
- Rewrite 'Bearer' in Authorization header check to be more robust.
Previously, the test case raise a KeyError exception when the
Authorization header isn't set at all in request headers.
This is a preparation CL for implementing ensure_authenticated() method
in ChainedAuthenticator and GitCredsAuthenticator in future CLs.
Bug: 348024314
Change-Id: I28a7fabbc6cf2dc33dccc1339b89d20a22dc12ea
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7082265
Commit-Queue: Jiewei Qian <qjw@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
This CL implements attempt_authenticate_with_reauth in LuciContextAuthenticator according to the base _Authenticator's
interface definition.
This allows LuciContextAuthenticator to be used where an action that needs to meet ReAuth requirement (instead of raising an exception
saying "ReAuth
is required").
Luci context access tokens should already satisfy ReAuth requirement.
Bug: 442666611
Change-Id: I2e4d43a75b230932fc779c805f75b5828c9d0980
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6967915
Reviewed-by: Allen Li <ayatane@chromium.org>
Commit-Queue: Jiewei Qian <qjw@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
This CL implements attempt_authenticate_with_reauth in GceAuthenticator
according to the base _Authenticator's interface definition.
This allows GceAuthenticator to be used where an action that needs to
meet ReAuth requirement (instead of raising an exception saying "ReAuth
is required").
GceAuthenticator's credential already satisfies ReAuth requirement.
ReAuth is satisfied if the GCE bot is trusted (controlled by Gerrit
server side config).
Bug:442666611
Change-Id: I9a801e3fd6ab9fb446e7842a93bcd9ee1ff953c7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6965527
Reviewed-by: Allen Li <ayatane@chromium.org>
Commit-Queue: Jiewei Qian <qjw@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
git cl format uses `git diff -U0` when it has to find the files
and ranges to format. Similarly, CiderG generates diffs with
-U0 and run presubmits for the checks.
However, the scm.py:GenerateDiff(), which CheckPatchFormatted() uses,
doesn't specify -U and it defaults to -U3.
This CL updates CheckPatchFormatted() to pass -U0 to
scm.py:GenerateDiff() so that Presubmit and git cl format are
always given diffs generated with -U0.
Bug: 386840832
Change-Id: I19479ab139b7dba8ba4e5e5fed89eca4d2e66412
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6952890
Auto-Submit: Scott Lee <ddoman@chromium.org>
Commit-Queue: Brian Ryner <bryner@google.com>
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Brian Ryner <bryner@google.com>
This CL adds ReAuth support to GerritAuthenticator. ReAuth token can be
obtained with a new get_authorization_header() call.
The task of obtaining such a token is delegated to different
authenticators to check if ReAuth is necessary, and if the existing
authentication token already satisfies ReAuth requirements.
Bug: 442666611
Change-Id: Ic661b868f1c61c653de0da43eb784ad5938342f2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6914237
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Jiewei Qian <qjw@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>
Historically, git_cl.py and presubmit_support.py generated diffs
inconsistently.
==== git_cl.py ====
1) git_cl.py:_RunClangFormatDiff() assumed the input diff was
generated with --no-prefix. Therefore, it hard-coded -p0 as a param
for clang_format_diff.py. If the diff was generated without --no-prefix,
it would have to pass -p1 instead.
2) git_cl.py _RunYapf() and _RunGoogleJavaFormat assumed that
the input diff was generated with prefixes. Therefore,
it parsed the diffs, assuming that the diff was generated with
a/ and b/ prefixes.
This discrepancy wasn't an issue because each of the Run functions
generated and consumed the diffs within themselves. It became an issue
when https://crrev.com/c/6931775 was developed. The CL consolidated
the diff generation into a common function, so that all _Run functions
need to agree with the presence of the prefixes.
The CL changed _RunClangFormatDiff() to use -p1 instead of -p0,
mainly because
- it's the p level for the default git-diff option
- it's the p level for the unix diff util
- CiderG Chromium also generates the diff with the prefixes to
emulate the default diff output.
- I highly believe that --no-prefix was chosen for no reasons.
It seems to be a random choice to work with -p0.
- Either Java/Python or Clang wrapper should be changed.
==== presubmit_support.py ====
presubmit_support.py can be given a diff_file via --diff_file.
Otherwise, it generates the diff based on --upstream_commit.
a) it doesn't enforce the presence of the prefixes in the input
diff_file. As stated above, the Chromium extension generates
a diff with prefixes.
b) in contrast, when it is not given --diff_file, it generates
the diff with --no-prefix, and I don't find any reasons for it.
I believe that it's from a copy-pasted random choice.
==== the problem ====
This discarepancy became a problem, when crrev.com/c/6937365
was landed. It enforces git_cl.py and presubmit_support.py to agree
for the format of the input diff. That is, the presence of
the prefixes must be agreed by all the following.
- presubmit_support.py with --diff_file /tmp/abc
- presubmit_suppory.py without --diff_file /tmp/abc
- git cl format
- git cl format --input_diff_file /tmp/abc
==== possible solutions ====
Obviously, there are 3 choices.
1) update the regex and parsers of git_cl.py to auto detect the best
-p{num}.
This was the least preferred option, as it can be fragile.
2) update Chromium extension and git_cl.py to use --no-prefix.
i.e.,
- update the Chromium CiderG to generate diffs without the prefix.
- update the regex used in _ComputeFormatDiffLineRanges() to assume
that the input is generated with --no-prefix.
- change _RunClangFormatDiff() to pass -p0 instead of -p1.
3) update presubmit_support.py to generate diff without --no-prefix.
==== What this CL does ====
It implements (3).
- That is the default option for git-diff.
- I don't find obvious reasons to use --no-prefix.
- CiderG has been generating diffs with prefixes for weeks.
- It implies that presubmit_support.py works fine with a prefixed diff.
Bug: 386840832
Change-Id: Iac8a4fc30f101e70e3ccc56f9f8ee48198dfa833
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6939737
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This makes the formatter runners to be git-free if --input_diff_file
is given. They still run `git diff`, if --diff is given. However,
for the purpose of making the presubmit check for format verification
git-free, it's ok as --diff is not used by the presubmit check.
Bug: 386840832
Change-Id: If5ab68fa4e2fec1aafa22e15ddeabb744993342b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6931775
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
If the option is given, git cl format uses the input diff file to
determine the files and sections to format instead of git-diff.
The intended use case is to remove git dependencies in case
presubmit_support.py is executed with a diff file and it calls
presubmit_canned_checks.CheckPatchFormatted().
Bug: 386840832
Change-Id: Ia1eacd1bdb81731e07753df19ebf1d2470c9f1aa
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6912693
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Scott Lee <ddoman@chromium.org>
gclient_paths.GetPrimarySolutionPath() finds primary
solution path from current directory, so it would fail
if it is invoked from outside of workspace.
Pass directory to gclient_paths.GetPrimarySolutionPath
so it could find workspace correctly if it is invoked
ninja outside of workspace.
Bug: 441240584
Change-Id: I873f7883873e143ec8a64ee0e636042ac2336a2a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6898614
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
This reverts commit eb60ab38de.
Reason for revert: re-landing with an additional patch
-------------
* Problem
Browser infra runs ci.*-presubmit builders, such as linux-presubmit,
with --all to ensure that the entire chromium/src passes
presubmit checks.
crrev.com/c/6842238 changed the finding type for License Check
from warning to error, but the CI presubmit builders failed because
there are many files without valid CopyRight.
Not only the existing files, all the new files that are added with
`Bypass-Check-License: <reason> footer could also cause the presubmit
builder to fail.
* This CL
In addition to the original patch from crrev.com/c/6842238,
this CL makes additional patches to turn the CopyRight errors into
warnings, if --file or --all is given.
The same approach is used in
https://source.chromium.org/chromium/chromium/tools/depot_tools/+/main:presubmit_canned_checks.py;l=982-987;drc=eb60ab38deeda6975c9b0fef883978f2a9f69120
Bug: 435696543,40237859
Original change's description:
> Revert "presubmit: emit errors instead of warnings for bad copyright headers"
>
> This reverts commit fa62515ecb.
>
> Reason for revert: it seems that the existing files without valid copyright headers are causing linux-presubmit builds to fail. b/438791294
>
> Bug: 435696543,40237859
> Original change's description:
> > presubmit: emit errors instead of warnings for bad copyright headers
> >
> > This is an effective revert of https://crrev.com/c/4895337 with
> > additional patches to support a footer.
> >
> > https://crrev.com/c/3887721 updated CheckLicense() to emit errors
> > for bad copyright headers. However, https://crrev.com/c/4895337
> > was changed the finding type from error to warning, claiming that
> > the check is N/A for moved third files, but it's not so easy
> > to programatiically distinguish moved third-party files.
> >
> > After discussions, it was decided to change the finding type back
> > to error to prevent accidental submissions for new files without
> > correct CopyRight headers.
> >
> > To mitigate moved, third-party files, this CL adds support for
> > "Bypass-Check-License: <reason>" footer.
> >
> > If the check should be ignored in a given CL, CL authors should
> > use the footer instead.
> >
> > Bug: 435696543,40237859
> > Change-Id: I177915c65932a3d76ea60ee6a0e396f726bc400d
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6842238
> > Reviewed-by: Gavin Mak <gavinmak@google.com>
> > Commit-Queue: Scott Lee <ddoman@chromium.org>
>
> Bug: 435696543,40237859
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Change-Id: Ibedf8d13e3742249947e29e625a14cceaf89879c
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6852377
> Commit-Queue: Scott Lee <ddoman@chromium.org>
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
Bug: 435696543,40237859
Change-Id: Iafdb29b928c016eb3949e29fd43a2ba5f53e0ba0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6852108
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Scott Lee <ddoman@chromium.org>
This reverts commit fa62515ecb.
Reason for revert: it seems that the existing files without valid copyright headers are causing linux-presubmit builds to fail. b/438791294
Bug: 435696543,40237859
Original change's description:
> presubmit: emit errors instead of warnings for bad copyright headers
>
> This is an effective revert of https://crrev.com/c/4895337 with
> additional patches to support a footer.
>
> https://crrev.com/c/3887721 updated CheckLicense() to emit errors
> for bad copyright headers. However, https://crrev.com/c/4895337
> was changed the finding type from error to warning, claiming that
> the check is N/A for moved third files, but it's not so easy
> to programatiically distinguish moved third-party files.
>
> After discussions, it was decided to change the finding type back
> to error to prevent accidental submissions for new files without
> correct CopyRight headers.
>
> To mitigate moved, third-party files, this CL adds support for
> "Bypass-Check-License: <reason>" footer.
>
> If the check should be ignored in a given CL, CL authors should
> use the footer instead.
>
> Bug: 435696543,40237859
> Change-Id: I177915c65932a3d76ea60ee6a0e396f726bc400d
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6842238
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Scott Lee <ddoman@chromium.org>
Bug: 435696543,40237859
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Change-Id: Ibedf8d13e3742249947e29e625a14cceaf89879c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6852377
Commit-Queue: Scott Lee <ddoman@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This is an effective revert of https://crrev.com/c/4895337 with
additional patches to support a footer.
https://crrev.com/c/3887721 updated CheckLicense() to emit errors
for bad copyright headers. However, https://crrev.com/c/4895337
was changed the finding type from error to warning, claiming that
the check is N/A for moved third files, but it's not so easy
to programatiically distinguish moved third-party files.
After discussions, it was decided to change the finding type back
to error to prevent accidental submissions for new files without
correct CopyRight headers.
To mitigate moved, third-party files, this CL adds support for
"Bypass-Check-License: <reason>" footer.
If the check should be ignored in a given CL, CL authors should
use the footer instead.
Bug: 435696543,40237859
Change-Id: I177915c65932a3d76ea60ee6a0e396f726bc400d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6842238
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Scott Lee <ddoman@chromium.org>
The current gsutil download code silently fails when the connection
drops mid-download, as read() returns an empty buffer instead of raising
an exception. This may lead to errors such as "zipfile.BadZipFile: File
is not a zip file" on Chromium sync with freshly-bootstrapped
depot_tools when downloading gcs deps.
This change solves this by hardening the process:
- Use retry mechanism with exponential backoff for gsutil download
- Switch to urlretrieve, which looks at Content-Length
- Compare MD5 of the downloaded file with the value from API
- Move exponential_backoff_retry from git_cache.py to gclient_utils.py
Change-Id: I25242948399e01373eb2afd9352e5c78a889051d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6485485
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Auto-Submit: Aleksei Khoroshilov <akhoroshilov@brave.com>
Reviewed-by: Scott Lee <ddoman@chromium.org>
The current implementation relies on the current working directory,
when it traverses the file tree to find the nearest OWNERS. It also
causes an infinite loop when it cannot find any OWNERS.
This CL changes the implementation so that it works
no matter what the cwd is.
Bug: 412904761
Change-Id: Ic4e25217aa64bd2eb6514ccdd486fe3b57a82312
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6484531
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
