This updates _EnabledTurboCiCheckHandler.set_result so that it adds a
BotUpdateResults when finalizing the associated source check. This
result provides information about all of the unpatched revisions that
were checked out.
Bug: b:443496677
Change-Id: I5ea0f798ef720e02fa130ec108a44d273187d964
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7227736
Reviewed-by: Robbie Iannucci <iannucci@google.com>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
This setting ensures that bot_update/gclient will delete conflicting
directories instead of moving them to _bad_scm, which avoids wasting
disk space on bots.
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_internal
Recipe-Nontrivial-Roll: chrome_release
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Bug: 462013257
Change-Id: I247427179205bf22bd24d142fa851d6628846a48
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7208105
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
crrev.com/c/7152830 overhauled how the bot_update output json was
generated but neglected to handle the ref:revision syntax and
crrev.com/c/7147310 added a test data method to enable overriding the
URLs returned for the repos but neglected to actually pass it to the
output_json method. This change rectifies both issues.
Change-Id: I604b0d7ff3f7e68e4829624e5de66efa2f5a61b0
Recipe-Nontrivial-Roll: build
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7219689
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
This adds and fixes documentation for the TurboCI check handler types
and the turboci_check_id parameter to bot_update.ensure_checkout.
Additionally, the gclient_config to _TurboCICheckHandler.create and
_EnabledTurboCiCheckHandler.__init__ was removed because the gclient
config wasn't being used.
Bug: b:443496677
Change-Id: I4acc3170522735e63062bed47f555aabcd2a4304
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7172970
Auto-Submit: Garrett Beaty <gbeaty@google.com>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Reviewed-by: Robbie Iannucci <iannucci@google.com>
This adds an optional turboci_check_id parameter to
bot_update.ensure_checkout. If set, it will cause a SOURCE check to be
emitted based on the gclient config being used and any gitiles commit or
gerrit change that is used.
Bug: b:443496677
Change-Id: Ic7fc55511b9b698cffb85fe872221e534fee742d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7147310
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Reviewed-by: Robbie Iannucci <iannucci@google.com>
The json output for bot_update reports the value that are passed as
--revision flags to bot_update.py in the fixed_revisions field in the
json output. In the case of a revision of 'HEAD' for a project that has
a CL, the 'HEAD' gets resolved to the target ref of the CL. In
simulation tests, the --revision flag in the command correctly shows the
target ref, but the fixed_revisions in the json output shows 'HEAD'.
This updates the code so that the target ref resolution happens before
setting the value in the dict that is used as the fixed_revisions field
in the output json. This results in a change to the revision since the
input to produce the hash will incorporate the ref.
Change-Id: I4fc0594039faf0725ef9c293b00f31bc7b2692a2
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_internal
Recipe-Nontrivial-Roll: infra
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7148746
Auto-Submit: Garrett Beaty <gbeaty@google.com>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Scott Lee <ddoman@chromium.org>
Currently, in order to specify a revision for a project in the
bot_update output, one has to either set the fixed_revisions or the
revision_mapping values. Both of these have additional effects on the
output json (fixed_revisions or properties) and setting them requires
fully overriding the json output. This can easily lead to output data
that doesn't match the gclient config being used.
This change overhauls the output_json test API function by adding a
revisions parameter that can be used to override the revisions used for
a project without affecting other portions of the json output. A
mod_test_data-decorated revisions method was added that enables
overriding the revisions without having to specify the full output json,
so fixed_revisions and properties will be set automatically according to
the gclient config being used. The revision_mapping parameter was
renamed to revision_properties to better distinguish it from revisions
and more accurately indicate its contents.
The following changes in behavior occur for the default test data:
* The got_revision property will only be added if the gclient config being used doesn't specify a got_revision_reverse_mapping
* If a got_revision property is added because the gclient config doesn't specify a got_revision_reverse_mapping and commit_positions is True, a got_revision_cp property will also be added
Additionally a post_check_output_json method was added that simplifies
the process of making post checks on the contents of the bot_update json
output.
Change-Id: I012b440107fa3323564fa3e1e3024c038e9944a2
Recipe-Manual-Change: build
Recipe-Manual-Change: infra
Recipe-Nontrivial-Roll: build_internal
Recipe-Nontrivial-Roll: chromiumos
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7152830
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Currently, bot_update.py uses git clean -dff, which leaves ignored
files. This can cause build failures on bisect bots when switching
between different revisions, as stale ignored files from newer revisions
can break older ones.
Add an optional --clean-ignored flag to bot_update. When enabled, it
runs git clean -dffx.
Bug: 454315184
Change-Id: Ia7f0f6a94c78eb335bb1ed291c6d248db899b74d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7087912
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@chromium.org>
crrev.com/c/6049353 updated bot_update.ensure_checkout to comput the
output commit even when not setting the output commit so that callers
could do custom logic for setting output commits while still having
access to the one computed by bot_update. Unfortunately, this changed
behavior for some builds that have set_output_commit set false. When
computing the output commit, if none of the conditions that set the ref
on the output commit were true then an AssertionError would be raised
indicating that the caller should call
buildbucket.set_output_gitiles_commit. With crrev.com/c/6049353, the
assertion can be raised even if set_output_commit is false.
This changes it so that the constraint on the ref field is only enforced
when set_output_commit is true.
Bug: 366409421
Change-Id: Ifb3a39784d75a3c47a7d4a6e2ac9957c29a09938
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6064719
Auto-Submit: Garrett Beaty <gbeaty@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
This reverts commit 66b3972fc5.
Reason for revert: This won't work because chromium_checkout relies on the position being set in the output commit in order to set information for RDB. We'll have to come up with a different solution.
Original change's description:
> Add parse_commit_position parameter to BotUpdateApi.ensure_checkout.
>
> When true and got_revision_cp is present, output commit ref and position are set from got_revision_cp. Default value is true.
>
> For Chrome builds that build from tags we will set parse_commit_position to false to show tags instead of branches on Buildbucket UI, e.g. refs/tags/132.0.6824.0 instead of refs/branch-heads/6824@{#1}.
>
> Bug: 366409421
> Change-Id: I77d01615edb6b791445a06469f80c673c97ad8d6
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6000539
> Reviewed-by: Garrett Beaty <gbeaty@google.com>
> Reviewed-by: Scott Lee <ddoman@chromium.org>
> Commit-Queue: Alex Kravchuk <alexanderkr@google.com>
Bug: 366409421
Change-Id: I21a5fc6e011a64fffd1d2ef0689d5a3f3843482c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6037603
Reviewed-by: Garrett Beaty <gbeaty@google.com>
Commit-Queue: Alex Kravchuk <alexanderkr@google.com>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Rename parse_commit_position parameter to parse_commit_position_for_tags. If input commit ref is not a tag, or parse_commit_position_for_tags is True, execute the old code path. This makes the changes backward compatible with the existing code.
Bug: 366409421
Change-Id: I58405325e406a82c2e255fe5d3d4a2883c98a84a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6035030
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Alex Kravchuk <alexanderkr@google.com>
Reviewed-by: Garrett Beaty <gbeaty@google.com>
When true and got_revision_cp is present, output commit ref and position are set from got_revision_cp. Default value is true.
For Chrome builds that build from tags we will set parse_commit_position to false to show tags instead of branches on Buildbucket UI, e.g. refs/tags/132.0.6824.0 instead of refs/branch-heads/6824@{#1}.
Bug: 366409421
Change-Id: I77d01615edb6b791445a06469f80c673c97ad8d6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6000539
Reviewed-by: Garrett Beaty <gbeaty@google.com>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Alex Kravchuk <alexanderkr@google.com>
In order to facilitate removing uses of api.path.checkout_dir from
downstream repos, this change adds a custom return type for
bot_update.ensure_checkout. Now instead of a standard step result, an
object of Result will be returned. Result records the relevant paths
(directory where the checkout was performed, the repo that was checked
out and the repo that was patched, if any). This provides the caller
the ability to work in any of these directories without using
api.path.checkout_dir and without requiring boilerplate to construct
the paths. It also includes some attributes that provide details from
within the json output to abstract that out.
Bug: 329113288, 339472834
Change-Id: I2ec6db635c5b799bdb65d4e9364e7d99aae4159e
Recipe-Manual-Change: build
Recipe-Manual-Change: build_limited
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5523194
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Currently overriding test data for bot_update requires specifying the
json output and in some cases the corresponding retcode. It's easy to
set the json output to values that wouldn't match the build inputs. This
change adds module test data for bot_update to have the bot_update fail
to checkout or fail the patch without having to manually specify the
json output as well. It also adds module test data for controlling
whether there are commit positions in the output properties.
The fail_patch property of the bot_update module was removed since it
was only used by tests to force the patch failure codepath and now the
fail_patch module test data can be used instead.
Change-Id: I03f61c2ff95c2561b746efbaae8e35b520027a2b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5518044
Reviewed-by: Scott Lee <ddoman@chromium.org>
Auto-Submit: Garrett Beaty <gbeaty@google.com>
Commit-Queue: Scott Lee <ddoman@chromium.org>
In production, if there is no patch, then the bot_update output will
have the value for the patch_root key set to null and the patch_failure
key will only be set if there was a patch failure. This updates the
output_json test API method so that the json matches this behavior. The patch_root parameter was added that will be used to set the
patch_root key. The root parameter is preserved for now until a
downstream use passing the argument by keyword can be updated.
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chrome_release
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Change-Id: I0c34a41ff18bce2900a2f176ff992e6d147abbd4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5506838
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Use Path.joinpath() or the / operator instead of Path.join().
Eventually, config_types.Path will be replaced with pathlib.Path, which
doesn't have a join() method. In general, if the substitution was simple
the / operator was used. Otherwise, '.join(' was replaced with
'.joinpath('.
Change-Id: Iff2eb0abd9fdb0d3b3bca281c7c5436f932a5380
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5483206
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Auto-Submit: Rob Mohr <mohrr@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
The blamelist_pins attribute is being set on the json output, which is a
dictionary. This only works right now because json.output currently uses
OrderedDict as the dict type. With removal of support for python2, dict
could be used since its iteration order is insertion order in python3.
This change removes setting the blamelist_pins attribute since its not
used anywhere and it will allow json.output to use plain dict.
Change-Id: Ie19b87d61903774454df2d395acf695eaa39b201
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4617204
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Auto-Submit: Garrett Beaty <gbeaty@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
This is a reland of commit 053817260a
The change was not related to the errors that it was reverted for, so no
changes are necessary.
Original change's description:
> Set a default got_revision property in the bot_update json output.
>
> The bot_update.py script sets a got_revision property by default even if
> it is not present in the reverse revision mapping. This results in an
> uncaught exception when set_output_commit is set to True if got_revision
> isn't present in the reverse revision mapping, but it isn't caught until
> production because the test API doesn't match that behavior. This change
> updates the test API method to match the behavior of the script.
>
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited
> Recipe-Nontrivial-Roll: chromiumos
> Recipe-Nontrivial-Roll: infra
> Change-Id: Ideefa9d77d2a816ae66a2bb52737264ed3f5bcee
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3575361
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Garrett Beaty <gbeaty@google.com>
> Auto-Submit: Garrett Beaty <gbeaty@google.com>
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Change-Id: I80efeee8a2b951df43b00d89bb596dc2cf2fcec7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3582182
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
This reverts commit 053817260a.
Reason for revert: we reverted got_revision change, and this change
itself may be causing some other failues.
Original change's description:
> Set a default got_revision property in the bot_update json output.
>
> The bot_update.py script sets a got_revision property by default even if
> it is not present in the reverse revision mapping. This results in an
> uncaught exception when set_output_commit is set to True if got_revision
> isn't present in the reverse revision mapping, but it isn't caught until
> production because the test API doesn't match that behavior. This change
> updates the test API method to match the behavior of the script.
>
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited
> Recipe-Nontrivial-Roll: chromiumos
> Recipe-Nontrivial-Roll: infra
> Change-Id: Ideefa9d77d2a816ae66a2bb52737264ed3f5bcee
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3575361
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Garrett Beaty <gbeaty@google.com>
> Auto-Submit: Garrett Beaty <gbeaty@google.com>
Change-Id: Ie975dfe39e7b8997336761e12f53f3046359d825
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3578820
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
The bot_update.py script sets a got_revision property by default even if
it is not present in the reverse revision mapping. This results in an
uncaught exception when set_output_commit is set to True if got_revision
isn't present in the reverse revision mapping, but it isn't caught until
production because the test API doesn't match that behavior. This change
updates the test API method to match the behavior of the script.
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Change-Id: Ideefa9d77d2a816ae66a2bb52737264ed3f5bcee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3575361
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Auto-Submit: Garrett Beaty <gbeaty@google.com>
The blamelist pins are produced by iterating over a dictionary which
yields a different order between python2 and python3. The blamelist pins
will now be produced with a consistent ordering between python versions
by sorting on the revision name.
Change-Id: Ieb6c4dfe0aacea83c4a86a3a10380cc2f3839114
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3352250
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
These soft errors will become hard failures and block the proto module
to be marked PY2+3 in upstream recipe engine repo.
Bug: 1227140
Change-Id: I0e5fdebe7e7cf43552b525d63c12f82c96c49902
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chrome_release
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3152784
Commit-Queue: Yuanjun Huang <yuanjunh@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
