https://crrev.com/c/3674981 installed CIPD ninja packages.
This CL switches from the old ninja binaries to them.
Note that this CL doesn't change ninja version. the old/CIPD ninja binaries are v1.8.2.
The differences from the old binaries are
- the CIPD packages don't use chromium's clang.
- the CIPD package for Linux doesn't use chromium's sysroot.
- the min macOS version is 10.13, which was 10.6.
Please also see https://crsrc.org/i/3pp/ninja/install_bootstrap.sh
On Windows, ninja.exe will be installed under .cipd_bin.
But there are many places that assume that ninja.exe exists on depot_tools root. So this CL also copies the ninja.exe from .cipd_bin to root in cipd_bin_setup.bat.
Bug: 931218
Change-Id: Ib67eee5e9b6ad9b2937b789626970d9c85867dbd
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3674982
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
This reverts commit 7c4f7ec408.
Reason for revert: bugs reported by users (see comments in Gerrit).
error: block. was unexpected at this time.
Original change's description:
> Make depot_tools scripts return exit codes properly on all platforms.
>
> Changes:
> 1. Windows: exit /b %errorlevel% should be used instead of goto :EOF to get valid exit codes during cmd /c <script>.bat invocation.
> 2. Windows: delayed var expansion is required in update_depot_tools.bat exit code generation.
> 3. Posix: update_depot_tools returns exit code from update_git_repo function in case of a failure.
>
> A rule of thumb on Windows: goto :EOF should not be used if %errorlevel% must be returned for all possible invocations.
>
> Test case for update_depot_tools changes:
> 1. Make a change to depot_tools sources that will conflict with next depot_tools update
> 2. Run update_depot_tools either directly or via gclient
> 3. Expect a git error is triggered
> 4. Inspect %errorlevel% or $? depending on platform
> 5. Expected 1, but the actual result is 0.
>
> Test case for changes in other .bat files:
> 1. Make a change to depot_tools sources that will conflict with next depot_tools update
> 2. Run cmd /c gclient
> 3. Expect a git error is triggered
> 4. Inspect %errorlevel%
> 5. Expected 1, but the actual result is 0.
>
> Change-Id: I64459982bcd9cc3db1319a9b39224b7a7af8c5aa
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3688632
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
> Auto-Submit: Aleksey Khoroshilov <akhoroshilov@brave.com>
> Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Change-Id: I85d598af01d75588cdee77165d6af22270ee031d
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3694139
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Changes:
1. Windows: exit /b %errorlevel% should be used instead of goto :EOF to get valid exit codes during cmd /c <script>.bat invocation.
2. Windows: delayed var expansion is required in update_depot_tools.bat exit code generation.
3. Posix: update_depot_tools returns exit code from update_git_repo function in case of a failure.
A rule of thumb on Windows: goto :EOF should not be used if %errorlevel% must be returned for all possible invocations.
Test case for update_depot_tools changes:
1. Make a change to depot_tools sources that will conflict with next depot_tools update
2. Run update_depot_tools either directly or via gclient
3. Expect a git error is triggered
4. Inspect %errorlevel% or $? depending on platform
5. Expected 1, but the actual result is 0.
Test case for changes in other .bat files:
1. Make a change to depot_tools sources that will conflict with next depot_tools update
2. Run cmd /c gclient
3. Expect a git error is triggered
4. Inspect %errorlevel%
5. Expected 1, but the actual result is 0.
Change-Id: I64459982bcd9cc3db1319a9b39224b7a7af8c5aa
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3688632
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Auto-Submit: Aleksey Khoroshilov <akhoroshilov@brave.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
A run of "git cl presubmit --all" shows that PyLint 1.5 is being run
from nine locations, however the deprecation warnings don't say where.
This makes it difficult to file bugs or fix the remaining instances.
This changes the message to list the path to the presubmit that is
running PyLint 1.5.
This also adds instructions on how to change to version 2.7.
Before:
pylint-1.5 is deprecated, please switch to 2.7 before 2022-07-11
After (with word wrapping):
pylint-1.5 is being run on ...src\tools\find_runtime_symbols
and is deprecated, please switch to 2.7 before 2022-07-11 (add
version='2.7' to RunPylint call)
Change-Id: Iece2cb904f5d26ad66e3ab78f7ce7aef1878bfd1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3688839
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
This reverts commit 013e47be09.
Reason for revert: This will be relanded as a warning, although the
bugs which the original change exposed should be fixed soon.
Original change's description:
> Revert "Fix not-run-test detection."
>
> This reverts commit 41691abe86.
>
> Reason for revert: I thought https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683378 would solve the problem but turns out this is the issue where it now reports it as a critical error.
>
> Original change's description:
> > Fix not-run-test detection.
> >
> > In crrev.com/c/2986400 the skip_shebang_check flag was added to address
> > the problem of some tests being run neither on Python 2 or Python 3. At
> > the same time reporting was added to warn if tests were run on neither
> > Python version.
> >
> > Unfortunately the reporting code was incorrect. A PresubmitPromptWarning
> > object was created but was not added to the list of results, so it had
> > no effect. Also the skipped tests name was added where a list was
> > expected which meant that each character of the test name showed up on
> > its own line.
> >
> > This change fixes the reporting. It also changes the format of the
> > report and makes it an error - a failure to run tests at all deserves
> > that severity level.
> >
> > A test run with the fixed check showed that no errors have crept in -
> > all tests were being run with one or the other. However the fix did
> > find a bug in an in-progress change I was working on.
> >
> > Bug: 1223478
> > Change-Id: Ibb44b5e60e7a7a5de08302f19ee4035cdfac5212
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3674199
> > Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
> > Commit-Queue: Fabrice de Gans <fdegans@chromium.org>
> > Reviewed-by: Fabrice de Gans <fdegans@chromium.org>
>
> Bug: 1223478
> Change-Id: If4395e90784c3f91692f398b4e5230a51139f78a
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683105
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Bug: 1223478, 1330859
Change-Id: I6a0a78e70db77c9c5fda5bab27960cd4e71279e5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3685349
Reviewed-by: Fabrice de Gans <fdegans@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Currently cipd packages listed in src/DEPS are being created and
installed in symlink mode (see bug for more info). This CL adds `$OverrideInstallMode copy` to the ensure files for cipd packages installed during gclient runhooks. `$OverrideInstallMode` was
implemented here, https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/3124202,
so that every cipd package can be installed directly to the installation
target, instead of being symlinked to an executable inside the cache/builder/.cipd dir.
The added `$OverrideInstallMode copy` can be removed once all cipd
packages themselves or the default mode is updated to be created in
copy mode.
Bug: 1329641
Change-Id: I68dc718e20f193c384d005c9b6b87603753943d4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3673698
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
This reverts commit 41691abe86.
Reason for revert: I thought https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683378 would solve the problem but turns out this is the issue where it now reports it as a critical error.
Original change's description:
> Fix not-run-test detection.
>
> In crrev.com/c/2986400 the skip_shebang_check flag was added to address
> the problem of some tests being run neither on Python 2 or Python 3. At
> the same time reporting was added to warn if tests were run on neither
> Python version.
>
> Unfortunately the reporting code was incorrect. A PresubmitPromptWarning
> object was created but was not added to the list of results, so it had
> no effect. Also the skipped tests name was added where a list was
> expected which meant that each character of the test name showed up on
> its own line.
>
> This change fixes the reporting. It also changes the format of the
> report and makes it an error - a failure to run tests at all deserves
> that severity level.
>
> A test run with the fixed check showed that no errors have crept in -
> all tests were being run with one or the other. However the fix did
> find a bug in an in-progress change I was working on.
>
> Bug: 1223478
> Change-Id: Ibb44b5e60e7a7a5de08302f19ee4035cdfac5212
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3674199
> Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
> Commit-Queue: Fabrice de Gans <fdegans@chromium.org>
> Reviewed-by: Fabrice de Gans <fdegans@chromium.org>
Bug: 1223478
Change-Id: If4395e90784c3f91692f398b4e5230a51139f78a
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683105
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
This reverts commit dfc71bbe01.
Reason for revert: Breaks all presubmits that do not have skip_shebang_check=True set.
Original change's description:
> Add Python 3 support to GetPythonUnitTests
>
> GetPythonUnitTests (and by extension RunPythonUnitTests) always ran
> scripts using Python 2. This change adds a python3=False parameter to
> GetPythonUnitTests to give the option of switching to Python 3, while
> leaving the default as Python 2. The child scripts are run under one or
> the other, based on this parameter.
>
> This change is needed in support of crrev.com/c/3679801.
>
> Bug: 1313804
> Change-Id: Ic59287352d4941707adaf7981ed7af4201b7d526
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3680099
> Reviewed-by: Jesse McKenna <jessemckenna@google.com>
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Bug: 1313804
Change-Id: I1bd7096b6cfdfed6205e6dc0b5d4498b5b821b97
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683378
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Christoffer Jansson <jansson@chromium.org>
Owners-Override: Ben Pastene <bpastene@chromium.org>
Reviewed-by: Tomas Gunnarsson <tommi@chromium.org>
GetPythonUnitTests (and by extension RunPythonUnitTests) always ran
scripts using Python 2. This change adds a python3=False parameter to
GetPythonUnitTests to give the option of switching to Python 3, while
leaving the default as Python 2. The child scripts are run under one or
the other, based on this parameter.
This change is needed in support of crrev.com/c/3679801.
Bug: 1313804
Change-Id: Ic59287352d4941707adaf7981ed7af4201b7d526
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3680099
Reviewed-by: Jesse McKenna <jessemckenna@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
In crrev.com/c/2986400 the skip_shebang_check flag was added to address
the problem of some tests being run neither on Python 2 or Python 3. At
the same time reporting was added to warn if tests were run on neither
Python version.
Unfortunately the reporting code was incorrect. A PresubmitPromptWarning
object was created but was not added to the list of results, so it had
no effect. Also the skipped tests name was added where a list was
expected which meant that each character of the test name showed up on
its own line.
This change fixes the reporting. It also changes the format of the
report and makes it an error - a failure to run tests at all deserves
that severity level.
A test run with the fixed check showed that no errors have crept in -
all tests were being run with one or the other. However the fix did
find a bug in an in-progress change I was working on.
Bug: 1223478
Change-Id: Ibb44b5e60e7a7a5de08302f19ee4035cdfac5212
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3674199
Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Fabrice de Gans <fdegans@chromium.org>
Reviewed-by: Fabrice de Gans <fdegans@chromium.org>
Some presubmit failures result in uncaught exceptions which historically
have halted presubmit runs. This behavior makes it more difficult to
find all failures, measure presubmit times, or detect other behavior
over time. This is particularly problematic when running "git cl
presubmit --all".
As an example, presubmit --all was broken and would not complete from
when crrev.com/3642633 landed until crrev.com/c/3657600. The minimal
repro was this presubmit command:
git cl presubmit --files ui\accessibility\ax_mode.h
This command, or presubmit --all, triggered this error, which halts the
presubmits:
Evaluation of CheckChangeOnCommit failed: [Errno 2] No such file or directory: 'chrome/browser/resources/accessibility/accessibility.js'
Traceback (most recent call last):
File "presubmit_support.py", line 2038, in <module>
sys.exit(main())
File "presubmit_support.py", line 2015, in main
return DoPresubmitChecks(
File "presubmit_support.py", line 1738, in DoPresubmitChecks
results += executer.ExecPresubmitScript(presubmit_script, filename)
File "presubmit_support.py", line 1602, in ExecPresubmitScript
self._run_check_function(function_name, context, sink,
File "presubmit_support.py", line 1640, in _run_check_function
six.reraise(e_type, e_value, e_tb)
File "C:\Users\brucedawson\.vpython-root\e726d2\lib\site-packages\six.py", line 686, in reraise
raise value
File "presubmit_support.py", line 1630, in _run_check_function
result = eval(function_name + '(*__args)', context)
File "<string>", line 1, in <module>
File "PRESUBMIT.py", line 314, in CheckChangeOnCommit
errs.extend(CheckModesMatch(input_api, output_api))
File "PRESUBMIT.py", line 283, in CheckModesMatch
ax_modes_in_js = GetAccessibilityModesFromFile(
File "PRESUBMIT.py", line 245, in GetAccessibilityModesFromFile
for line in open(fullpath).readlines():
FileNotFoundError: [Errno 2] No such file or directory: 'chrome/browser/resources/accessibility/accessibility.js'
With this change the exception is handled and presubmits continue after
printing this message:
Evaluation of CheckChangeOnCommit failed: [Errno 2] No such file or directory: 'chrome/browser/resources/accessibility/accessibility.js', Traceback (most recent call last):
File "presubmit_support.py", line 1630, in _run_check_function
result = eval(function_name + '(*__args)', context)
File "<string>", line 1, in <module>
File "PRESUBMIT.py", line 314, in CheckChangeOnCommit
errs.extend(CheckModesMatch(input_api, output_api))
File "PRESUBMIT.py", line 283, in CheckModesMatch
ax_modes_in_js = GetAccessibilityModesFromFile(
File "PRESUBMIT.py", line 245, in GetAccessibilityModesFromFile
for line in open(fullpath).readlines():
FileNotFoundError: [Errno 2] No such file or directory: 'chrome/browser/resources/accessibility/accessibility.js'
That is, the crucial information regarding the failure is printed, but
subsequent presubmits are not skipped.
Bug: 1309977
Change-Id: I7cfeda85c830e6c8e567c0df3c50f27af1dbe835
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3653978
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
A typical cpplint error report during a presubmit looks like this:
.../linux_ui.cc:64: Almost always, snprintf is better than strcpy [runtime/printf] [4]
...
Changelist failed cpplint.py check.
The problem is that the details of the error go to stderr when
CheckChangeLintsClean is run, whereas the "Changelist failed cpplint.py
check." message is printed at the end of the run. If there a lot of
errors, warnings, or messages then the association between them can be
completely lost.
This change adds the tag (cpplint) to the error messages, and adds a
comment to the "failed cpplint.py check" message to suggest looking for
the tag.
Bug: 1309977
Change-Id: I0d073b57f215e28495cbc3e009cab6ac2f23b65e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3632947
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Some of the expensive checks when running presubmit --all, such as
CheckStableMojomChanges (~300 s) and CheckAddedDepsHaveTargetApprovals
(~200 s) only look at diffs and are therefore guaranteed to be NOPs when
running presubmit --all or --files=. Passing along the no_diffs state
lets these expensive checks be skipped, thus allowing for faster
iteration times.
Initial testing suggests that (with some supporting changes in the
Chromium repo) this reduces "presubmit --all" times by about ten
minutes, or a bit more than 10%, and additional improvements may be
possible.
Special handling for the no-diffs case also offers a simple way to avoid
presubmit failures that happen whenever all files are flagged as being
changed.
Finally, and perhaps most importantly for having a presubmit --all bot,
when --no_diffs is passed we can treat errors like "Issue wasn't
uploaded" and "Add a description to the CL" as messages, thus making it
possible to have zero presubmit errors when run on origin/main.
Bug: 1320937, 1322936
Change-Id: I0d09dd4aae8fdaa48c8b2f89337441cf96dcff72
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3628368
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Presubmit message types were printed in an inconsistent order, generally
determined by which type was encountered first. This unpredictability
can be confusing when comparing runs. This change enforces a consistent
order, in increasing order of severity. ERRORS always go last so that
they will be the most visible when running presubmits locally.
This also adds a "There were Python %d presubmit errors." message to
the end to make it easier to tell when there were presubmit errors.
This is particularly useful when manually running presubmits.
Bug: 1309977
Change-Id: Ib2b73c7625789bad5b21ae12abf238b746cd11e1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3668724
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
In https://crbug.com/1327371, we started encountering hanging gsutil
calls. These were triggered under very specific circumstances, but
were essentially due to a multi-processing bug in python on MacOS:
https://bugs.python.org/issue33725
When running gsutil on mac, it explicitly suggests disabling
multi-processing for this reason:
https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331
So this CL simply puts that suggestion into practice. There are two
options for doing so: adding a line to an active Boto file, or adding
a "-o" arg to every gsutil invocation. Since chromium has multiple
copies of gsutil and multiple different locations where `gsutil` is
invoked, the latter option is considered intractable.
Instead, we choose the Boto file option by adding a context manager
to the gsutil recipe module that will insert a custom Boto file into
the environment that only disables multi-threading. Due to the fact
that the BOTO_PATH env var takes multiple paths, we can safely
incorporate both our custom Boto along with LUCI's per-build Boto file
in the same gsutil invocation. And when the context manager exits, the
environment should revert back to its original state.
Chromium incorporates this method in crrev.com/c/3662023. See it take
effect during the "gclient runhooks" step in this led build:
https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com
Bug: 1327371
Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
While enabling cpplint on base I found an extraneous error and an overly
complicated invocation of CheckChangeLintsClean. This change disables
the extraneous error and improves the file filtering defaults to make it
easier to invoke cpplint.
The warning was:
base\win\win_util.cc(193): Missing username in TODO; it should look
like "// TODO(my_username): Stuff." [readability/todo] [2]
Chromium has over 10,000 TODO comments that are links to bugs instead of
users so we clearly want to suppress this warning.
Most calls to CheckChangeLintsClean in Chromium pass in custom file
filtering in order to only analyze C++ files. Failing to pass a file
filter leads to errors like this:
Ignoring base\win\embedded_i18n\create_string_rc.py; not a valid
file name (cc, cuh, cu, cpp, h)
Ignoring c:\src\chromium\src\components\arc\PRESUBMIT.py; not a
valid file name (cu, cc, cuh, cpp, h)
It doesn't make sense for the canned checks version of
CheckChangeLintsClean to default to passing in files that cpplint will
complain about so if source_file_filter is None then I configure an
appropriate filter.
This change also switches to vs7 output format by default on Windows,
and deletes a trailing space.
Bug: 1309977
Change-Id: I342ca9e56a41502755dfc09307b441dfa6a48ec0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3655592
Reviewed-by: Will Harris <wfh@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
