This CL unifies the opt-in/opt-out handling for build telemetry collections about Reclient and Ninjalog.
The user consent message will be displayed only once at the beginning of a build.
```
❯ autoninja -C out/deterministic-andorid-dbg base
*** NOTICE ***
Google-internal telemetry (including build logs, username, and hostname) is collected on corp machines to diagnose performance and fix build issues. This reminder will be shown 9 more times. See http://go/chrome-build-telemetry for details. Hide this notice or opt out by running: build_telemetry [opt-in] [opt-out]
*** END NOTICE ***
Proxy started successfully.
...
```
Bug: 345113094
Change-Id: Ie5886287c4bd20262be0ff247508ac3869441eb1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5669094
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Michael Savigny <msavigny@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
With this CL, build time of chrome with non-cache hit case improved a
bit on P620 Linux workstation.
```
Benchmark 1: RBE_local_resource_fraction=0.4 RBE_racing_bias=0.7 RBE_remote_accept_cache=false autoninja -C out/Default chrome
Time (mean ± σ): 337.697 s ± 11.439 s [User: 3086.301 s, System: 1362.952 s]
Range (min … max): 327.732 s … 351.680 s 5 runs
```
```
Benchmark 1: RBE_remote_accept_cache=false autoninja -C out/Default chrome
Time (mean ± σ): 347.795 s ± 15.906 s [User: 2764.181 s, System: 1199.244 s]
Range (min … max): 331.964 s … 367.898 s 5 runs
```
Bug: 352463976
Change-Id: Ic9732583857ce175888a7ec65cf9e323529f0a07
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5693250
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Prior to this change, autoninja output was:
```
src/buildtools/reclient/bootstrap --re_proxy=src/buildtools/reclient/reproxy --cfg=src/buildtools/reclient_cfgs/reproxy.cfg
Proxy started successfully.
0.705 s to start reproxy
2.85s loading fs state
2.80s load build.ninja
ninja: no work to do.
7.38s Build Succeeded: 0 steps - 0.00/s
src/buildtools/reclient/bootstrap --shutdown --cfg=src/buildtools/reclient_cfgs/reproxy.cfg
RBE Stats: down 0 B, up 0 B,
0.331 s to stop reproxy
```
Now it's:
```
src/buildtools/reclient/bootstrap --re_proxy=src/buildtools/reclient/reproxy --cfg=src/buildtools/reclient_cfgs/reproxy.cfg
Proxy started successfully.
0.705s to start reproxy
2.85s loading fs state
2.80s load build.ninja
ninja: no work to do.
7.38s Build Succeeded: 0 steps - 0.00/s
src/buildtools/reclient/bootstrap --shutdown --cfg=src/buildtools/reclient_cfgs/reproxy.cfg
RBE Stats: down 0 B, up 0 B,
0.331s to stop reproxy
```
Change-Id: If1c4cd211f29267ed01c14efbe2fe09a1d101287
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5509391
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Louis Romero <lpromero@google.com>
Windows devs use luci-auth as the auth mechanism (as credshelper). By
default, credshelper-based auth is cached, but that doesn't work quite
correctly with luci-auth (it does it's own caching). It can cause
intermittent auth failures during a build, which can slow down build
performance.
Bug: b/335525655
Change-Id: Ic4ef25cfa8e0f37929170e306e8bf57e737a4a72
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5473742
Reviewed-by: Ben Segall <bentekkie@google.com>
Auto-Submit: Michael Savigny <msavigny@google.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
Occasionally on windows there seems to be a racing problem where rmtree
tries to delete something that no longer exists. This seems to happen
when a build is cancelled/terminated while in-progress, so racing
actions in reproxy might have deleted the temporary file while autoninja
is cleaning up stuff. In any case, missing a file in this case isn't
something to fail over.
Bug: b/329738642
Change-Id: I86240e31093c6415707878277b01db5931b175ba
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5376532
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
I initially didn't want to make this code Google-specific, however I now
believe that it's better for everyone to simplify it like this:
- It's much less code to just check for the path prefix. We do this
already in gclient.py, too, so the approach is proven to work.
- The call to findmnt introduces a 15ms overhead that affects everyone,
even users who will never have their workspace on Cog.
- External folks who run their own infrastructure likely have their own
fork of depot_tools anyway and can patch this as needed.
Bug: b/324547324
Change-Id: Ic00d417c280ac71529d106cfb2bcc844dfd44bd3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5358773
Commit-Queue: Philipp Wollermann <philwo@chromium.org>
Auto-Submit: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Use the log directory for the hash used to determime the socket used, so
developers get a distinct socket per build instead of per build
directory. Should allow concurrent builds against the same output
directory as much as ninja allows it.
Bug:b/321554715
Change-Id: I714dd2d32e2c651075055134e1568423a43eabdb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5290983
Auto-Submit: Michael Savigny <msavigny@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Change is that racing directory is deleted atexit instead of being
cleaned up in a sweep at the start, whick wouldn't work anyway for the
actual use case the change was for (in-use racing directories would get
deleted, breaking concurrent builds).
Bug:b/324877315
Change-Id: I3baf25b07c13061e9b4c14a3a701ed9eceb9bbb4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5288415
Auto-Submit: Michael Savigny <msavigny@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Set cas_concurrency for mac builds to 100. It doesn't affect
performance on high-speed networks, but has shown an improvement on
networks prone to congestion.
Note that these adjustments have been checked out for builds on macs, and the cas_concurrency value being a default of 100 for mac doesn't mean it should be the default for other oses.
Bug: b/304616237
Change-Id: I9431fdb35737f9f2b236296625af2f2a31e52754
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5075857
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
The code rotating the old build logs expects to delete directories
(since it uses rmtree) but tries to delete all items returned by
os.listdir(...) which may includes symbolic links or regular files.
Change it to only consider the old build logs directories instead.
Based on crrev/5028323, with fix to unit test.
Fixed: b/310900283
Change-Id: I4c618e6618c0193331c063028ebf02d8c4e7baee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5038467
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Ramy Medhat <abdelaal@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.
Reformatted using:
files=( $(
git ls-tree -r --name-only HEAD | \
grep -Ev -e '^(third_party|recipes)/' | \
grep '\.py$';
git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"
The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py
These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py
Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
local_resource_fraction is set to 0.2 as goma limited the max local
racing subprocesses to num_cpu/5
racing_bias is set to 0.95 as experimentally that ensures reasonable
behaviour for cache hits.
This cl does not enable racing by default. These values are only used if
RBE_exec_strategy=racing is set
Test: Tested builds on my mac and linux cloudtop and got mostly remote exec
and cache hits with some local executions
Bug: b/288564971
Change-Id: Idb71c03f4addd23d0253dc5bf3564058cd212284
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4713186
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
