This CL refactors _test_gerrit_ensure_authenticated_common by splitting
it into:
- The actual mock and common code path
- Test case specific setup (e.g. scm.GIT.SetConfig)
In addition, fixes a few issues with the existing code:
- _Authenticator.get() is now mocked correctly, so a CookiesAuthenticator
is returned in relevant tests. Previously it returned the "production"
ChainedAuthenticator, which could call other authenticators (such as
GitCredsAuthenticator based on the runtime environment)
- Adds mock patch cleanup functions. Previously, cleanup functions
aren't registered, so the mock leaks and got overwritten by other test
cases.
- Rewrite 'Bearer' in Authorization header check to be more robust.
Previously, the test case raise a KeyError exception when the
Authorization header isn't set at all in request headers.
This is a preparation CL for implementing ensure_authenticated() method
in ChainedAuthenticator and GitCredsAuthenticator in future CLs.
Bug: 348024314
Change-Id: I28a7fabbc6cf2dc33dccc1339b89d20a22dc12ea
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7082265
Commit-Queue: Jiewei Qian <qjw@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
This CL implements attempt_authenticate_with_reauth in LuciContextAuthenticator according to the base _Authenticator's
interface definition.
This allows LuciContextAuthenticator to be used where an action that needs to meet ReAuth requirement (instead of raising an exception
saying "ReAuth
is required").
Luci context access tokens should already satisfy ReAuth requirement.
Bug: 442666611
Change-Id: I2e4d43a75b230932fc779c805f75b5828c9d0980
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6967915
Reviewed-by: Allen Li <ayatane@chromium.org>
Commit-Queue: Jiewei Qian <qjw@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
This CL implements attempt_authenticate_with_reauth in GceAuthenticator
according to the base _Authenticator's interface definition.
This allows GceAuthenticator to be used where an action that needs to
meet ReAuth requirement (instead of raising an exception saying "ReAuth
is required").
GceAuthenticator's credential already satisfies ReAuth requirement.
ReAuth is satisfied if the GCE bot is trusted (controlled by Gerrit
server side config).
Bug:442666611
Change-Id: I9a801e3fd6ab9fb446e7842a93bcd9ee1ff953c7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6965527
Reviewed-by: Allen Li <ayatane@chromium.org>
Commit-Queue: Jiewei Qian <qjw@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
git cl format uses `git diff -U0` when it has to find the files
and ranges to format. Similarly, CiderG generates diffs with
-U0 and run presubmits for the checks.
However, the scm.py:GenerateDiff(), which CheckPatchFormatted() uses,
doesn't specify -U and it defaults to -U3.
This CL updates CheckPatchFormatted() to pass -U0 to
scm.py:GenerateDiff() so that Presubmit and git cl format are
always given diffs generated with -U0.
Bug: 386840832
Change-Id: I19479ab139b7dba8ba4e5e5fed89eca4d2e66412
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6952890
Auto-Submit: Scott Lee <ddoman@chromium.org>
Commit-Queue: Brian Ryner <bryner@google.com>
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Brian Ryner <bryner@google.com>
This CL adds ReAuth support to GerritAuthenticator. ReAuth token can be
obtained with a new get_authorization_header() call.
The task of obtaining such a token is delegated to different
authenticators to check if ReAuth is necessary, and if the existing
authentication token already satisfies ReAuth requirements.
Bug: 442666611
Change-Id: Ic661b868f1c61c653de0da43eb784ad5938342f2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6914237
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Jiewei Qian <qjw@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>
Historically, git_cl.py and presubmit_support.py generated diffs
inconsistently.
==== git_cl.py ====
1) git_cl.py:_RunClangFormatDiff() assumed the input diff was
generated with --no-prefix. Therefore, it hard-coded -p0 as a param
for clang_format_diff.py. If the diff was generated without --no-prefix,
it would have to pass -p1 instead.
2) git_cl.py _RunYapf() and _RunGoogleJavaFormat assumed that
the input diff was generated with prefixes. Therefore,
it parsed the diffs, assuming that the diff was generated with
a/ and b/ prefixes.
This discrepancy wasn't an issue because each of the Run functions
generated and consumed the diffs within themselves. It became an issue
when https://crrev.com/c/6931775 was developed. The CL consolidated
the diff generation into a common function, so that all _Run functions
need to agree with the presence of the prefixes.
The CL changed _RunClangFormatDiff() to use -p1 instead of -p0,
mainly because
- it's the p level for the default git-diff option
- it's the p level for the unix diff util
- CiderG Chromium also generates the diff with the prefixes to
emulate the default diff output.
- I highly believe that --no-prefix was chosen for no reasons.
It seems to be a random choice to work with -p0.
- Either Java/Python or Clang wrapper should be changed.
==== presubmit_support.py ====
presubmit_support.py can be given a diff_file via --diff_file.
Otherwise, it generates the diff based on --upstream_commit.
a) it doesn't enforce the presence of the prefixes in the input
diff_file. As stated above, the Chromium extension generates
a diff with prefixes.
b) in contrast, when it is not given --diff_file, it generates
the diff with --no-prefix, and I don't find any reasons for it.
I believe that it's from a copy-pasted random choice.
==== the problem ====
This discarepancy became a problem, when crrev.com/c/6937365
was landed. It enforces git_cl.py and presubmit_support.py to agree
for the format of the input diff. That is, the presence of
the prefixes must be agreed by all the following.
- presubmit_support.py with --diff_file /tmp/abc
- presubmit_suppory.py without --diff_file /tmp/abc
- git cl format
- git cl format --input_diff_file /tmp/abc
==== possible solutions ====
Obviously, there are 3 choices.
1) update the regex and parsers of git_cl.py to auto detect the best
-p{num}.
This was the least preferred option, as it can be fragile.
2) update Chromium extension and git_cl.py to use --no-prefix.
i.e.,
- update the Chromium CiderG to generate diffs without the prefix.
- update the regex used in _ComputeFormatDiffLineRanges() to assume
that the input is generated with --no-prefix.
- change _RunClangFormatDiff() to pass -p0 instead of -p1.
3) update presubmit_support.py to generate diff without --no-prefix.
==== What this CL does ====
It implements (3).
- That is the default option for git-diff.
- I don't find obvious reasons to use --no-prefix.
- CiderG has been generating diffs with prefixes for weeks.
- It implies that presubmit_support.py works fine with a prefixed diff.
Bug: 386840832
Change-Id: Iac8a4fc30f101e70e3ccc56f9f8ee48198dfa833
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6939737
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This makes the formatter runners to be git-free if --input_diff_file
is given. They still run `git diff`, if --diff is given. However,
for the purpose of making the presubmit check for format verification
git-free, it's ok as --diff is not used by the presubmit check.
Bug: 386840832
Change-Id: If5ab68fa4e2fec1aafa22e15ddeabb744993342b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6931775
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
If the option is given, git cl format uses the input diff file to
determine the files and sections to format instead of git-diff.
The intended use case is to remove git dependencies in case
presubmit_support.py is executed with a diff file and it calls
presubmit_canned_checks.CheckPatchFormatted().
Bug: 386840832
Change-Id: Ia1eacd1bdb81731e07753df19ebf1d2470c9f1aa
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6912693
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Scott Lee <ddoman@chromium.org>
gclient_paths.GetPrimarySolutionPath() finds primary
solution path from current directory, so it would fail
if it is invoked from outside of workspace.
Pass directory to gclient_paths.GetPrimarySolutionPath
so it could find workspace correctly if it is invoked
ninja outside of workspace.
Bug: 441240584
Change-Id: I873f7883873e143ec8a64ee0e636042ac2336a2a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6898614
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
This reverts commit eb60ab38de.
Reason for revert: re-landing with an additional patch
-------------
* Problem
Browser infra runs ci.*-presubmit builders, such as linux-presubmit,
with --all to ensure that the entire chromium/src passes
presubmit checks.
crrev.com/c/6842238 changed the finding type for License Check
from warning to error, but the CI presubmit builders failed because
there are many files without valid CopyRight.
Not only the existing files, all the new files that are added with
`Bypass-Check-License: <reason> footer could also cause the presubmit
builder to fail.
* This CL
In addition to the original patch from crrev.com/c/6842238,
this CL makes additional patches to turn the CopyRight errors into
warnings, if --file or --all is given.
The same approach is used in
https://source.chromium.org/chromium/chromium/tools/depot_tools/+/main:presubmit_canned_checks.py;l=982-987;drc=eb60ab38deeda6975c9b0fef883978f2a9f69120
Bug: 435696543,40237859
Original change's description:
> Revert "presubmit: emit errors instead of warnings for bad copyright headers"
>
> This reverts commit fa62515ecb.
>
> Reason for revert: it seems that the existing files without valid copyright headers are causing linux-presubmit builds to fail. b/438791294
>
> Bug: 435696543,40237859
> Original change's description:
> > presubmit: emit errors instead of warnings for bad copyright headers
> >
> > This is an effective revert of https://crrev.com/c/4895337 with
> > additional patches to support a footer.
> >
> > https://crrev.com/c/3887721 updated CheckLicense() to emit errors
> > for bad copyright headers. However, https://crrev.com/c/4895337
> > was changed the finding type from error to warning, claiming that
> > the check is N/A for moved third files, but it's not so easy
> > to programatiically distinguish moved third-party files.
> >
> > After discussions, it was decided to change the finding type back
> > to error to prevent accidental submissions for new files without
> > correct CopyRight headers.
> >
> > To mitigate moved, third-party files, this CL adds support for
> > "Bypass-Check-License: <reason>" footer.
> >
> > If the check should be ignored in a given CL, CL authors should
> > use the footer instead.
> >
> > Bug: 435696543,40237859
> > Change-Id: I177915c65932a3d76ea60ee6a0e396f726bc400d
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6842238
> > Reviewed-by: Gavin Mak <gavinmak@google.com>
> > Commit-Queue: Scott Lee <ddoman@chromium.org>
>
> Bug: 435696543,40237859
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Change-Id: Ibedf8d13e3742249947e29e625a14cceaf89879c
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6852377
> Commit-Queue: Scott Lee <ddoman@chromium.org>
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
Bug: 435696543,40237859
Change-Id: Iafdb29b928c016eb3949e29fd43a2ba5f53e0ba0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6852108
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Scott Lee <ddoman@chromium.org>
This reverts commit fa62515ecb.
Reason for revert: it seems that the existing files without valid copyright headers are causing linux-presubmit builds to fail. b/438791294
Bug: 435696543,40237859
Original change's description:
> presubmit: emit errors instead of warnings for bad copyright headers
>
> This is an effective revert of https://crrev.com/c/4895337 with
> additional patches to support a footer.
>
> https://crrev.com/c/3887721 updated CheckLicense() to emit errors
> for bad copyright headers. However, https://crrev.com/c/4895337
> was changed the finding type from error to warning, claiming that
> the check is N/A for moved third files, but it's not so easy
> to programatiically distinguish moved third-party files.
>
> After discussions, it was decided to change the finding type back
> to error to prevent accidental submissions for new files without
> correct CopyRight headers.
>
> To mitigate moved, third-party files, this CL adds support for
> "Bypass-Check-License: <reason>" footer.
>
> If the check should be ignored in a given CL, CL authors should
> use the footer instead.
>
> Bug: 435696543,40237859
> Change-Id: I177915c65932a3d76ea60ee6a0e396f726bc400d
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6842238
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Scott Lee <ddoman@chromium.org>
Bug: 435696543,40237859
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Change-Id: Ibedf8d13e3742249947e29e625a14cceaf89879c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6852377
Commit-Queue: Scott Lee <ddoman@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This is an effective revert of https://crrev.com/c/4895337 with
additional patches to support a footer.
https://crrev.com/c/3887721 updated CheckLicense() to emit errors
for bad copyright headers. However, https://crrev.com/c/4895337
was changed the finding type from error to warning, claiming that
the check is N/A for moved third files, but it's not so easy
to programatiically distinguish moved third-party files.
After discussions, it was decided to change the finding type back
to error to prevent accidental submissions for new files without
correct CopyRight headers.
To mitigate moved, third-party files, this CL adds support for
"Bypass-Check-License: <reason>" footer.
If the check should be ignored in a given CL, CL authors should
use the footer instead.
Bug: 435696543,40237859
Change-Id: I177915c65932a3d76ea60ee6a0e396f726bc400d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6842238
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Scott Lee <ddoman@chromium.org>
The current gsutil download code silently fails when the connection
drops mid-download, as read() returns an empty buffer instead of raising
an exception. This may lead to errors such as "zipfile.BadZipFile: File
is not a zip file" on Chromium sync with freshly-bootstrapped
depot_tools when downloading gcs deps.
This change solves this by hardening the process:
- Use retry mechanism with exponential backoff for gsutil download
- Switch to urlretrieve, which looks at Content-Length
- Compare MD5 of the downloaded file with the value from API
- Move exponential_backoff_retry from git_cache.py to gclient_utils.py
Change-Id: I25242948399e01373eb2afd9352e5c78a889051d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6485485
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Auto-Submit: Aleksei Khoroshilov <akhoroshilov@brave.com>
Reviewed-by: Scott Lee <ddoman@chromium.org>
The current implementation relies on the current working directory,
when it traverses the file tree to find the nearest OWNERS. It also
causes an infinite loop when it cannot find any OWNERS.
This CL changes the implementation so that it works
no matter what the cwd is.
Bug: 412904761
Change-Id: Ic4e25217aa64bd2eb6514ccdd486fe3b57a82312
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6484531
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
These tests get broken by If0a214fe074d7e7591b5d37417ca447525c983b6 so
delete them.
Now, you might raise your eyebrows at that, but hear me out.
These tests are so heavily mocked that I argue that they aren't
providing any value. What's the point of tests? To save time by
catching regressions early.
Being dumped a large list of expected mocked calls like:
@10: (['os.path.isfile', 'TEMP_DIR/trace-packet'],)
@11: (['FileRead', 'TEMP_DIR/trace-packet'],)
@12: (['FileWrite', 'TEMP_DIR/trace-packet', 'git-hash: 012345\ngit-hash: abcdea\n'],)
@13: (['make_archive', 'TRACES_DIR/20170316T200041.000000-traces', 'zip', 'TEMP_DIR'],)
@14: (['FileWrite', 'TEMP_DIR/git-config', <ANY>],)
provides little value. Like, what the heck is this even supposed to
be testing?
I argue that:
the cost of potentially having a regression in the future, fixing it,
and adding a new, sane test for that case
is LESS than:
the cost of someone looking at one of these tests failures and
trying to figure out what exactly broke, whether it is an actual
breakage or do they just need to wade into the mess of expected mocked
calls and figure out how to update the mocked calls to reflect a
perfectly fine change that they are trying to make
Bug: 404613530
Change-Id: Id8e2f5ebb9ad55ee32291c2f70b2d26fb479b195
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6428049
Reviewed-by: Gregory Nisbet <gregorynisbet@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Allen Li <ayatane@chromium.org>
Attempt to update the README.chromium file with the new submodule
revision after rolling it in a DEPS update. This update is flag-guarded
and will only run when explicitly specified.
Currently, only README.chromium files with a single "Revision:" line are
supported. Multiple lines and files with the divider are not handled.
These are left as future TODOs.
Bug: 390067679
Change-Id: Ib776564ae94360cc72dd633fc7ed7b3f84b5b9d2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6173767
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Jordan Brown <rop@google.com>
Reviewed-by: Jiewei Qian <qjw@chromium.org>
The existing licence regex assumed each line would start with an unknown
and optional comment delimiter, followed by a required space. However
the standard way to write the licence in XML files uses a multi-line
comment, each individual line simply starts with the licence text, no
delimiter or space. The license check thus needed to be updated to make
the space optional as well.
Bug: 407060754
Change-Id: I4a83dd9408de890593952e441ba6b650f726907c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6409392
Commit-Queue: Sky Malice <skym@chromium.org>
Reviewed-by: Andy Perelson <ajp@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Make this available for testing first.
This should be functionally roughly analogous with the current auto
configuration logic, and it prints all actions it performs so it
should be pretty safe to play with, even if it hypothetically
misbehaves.
Bug: b/401338175
Change-Id: I803c7e167e355ec8cca1f5959099138c2fee305e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6334614
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Allen Li <ayatane@chromium.org>
