Currently, {autoninja, autoninja.bat} scripts generate AUTONINJA_BUILD_ID by calling `python3 -c "import uuid; print(uuid.uuid4())"`.
This CL moves the logic at the beginning `autoninja.py`.
This also reduces the overhead of starting Python interpreter, which
takes around 60ms on my Windows workstaion.
```
> hyperfine python3
Benchmark 1: python3
Time (mean ± σ): 63.4 ms ± 7.8 ms [User: 25.7 ms, System: 33.7 ms]
Range (min … max): 51.0 ms … 80.2 ms 41 runs
```
Change-Id: I93d5b65f3c5542c3a93a3de1f27a5aa5d06c09a6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5737673
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
This CL unifies the opt-in/opt-out handling for build telemetry collections about Reclient and Ninjalog.
The user consent message will be displayed only once at the beginning of a build.
```
❯ autoninja -C out/deterministic-andorid-dbg base
*** NOTICE ***
Google-internal telemetry (including build logs, username, and hostname) is collected on corp machines to diagnose performance and fix build issues. This reminder will be shown 9 more times. See http://go/chrome-build-telemetry for details. Hide this notice or opt out by running: build_telemetry [opt-in] [opt-out]
*** END NOTICE ***
Proxy started successfully.
...
```
Bug: 345113094
Change-Id: Ie5886287c4bd20262be0ff247508ac3869441eb1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5669094
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Michael Savigny <msavigny@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
This CL allows autoninja to use Siso's builtin RBE client instead of using Reclient.
Developers have to specify `use_remoteexec=true use_reclient=false` GN arg to use this mode.
This should be no-op for all developers unless `use_reclient=false` is set.
Bug: 342038479
Change-Id: I8a67f114c056be6ddb0cd26cccf0157e0b184c5a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5567811
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Include a luci-auth check of the account logged in.
This is to check that a user isn't using a non-google account on a corp machine. Also this check is only performed every 12 hours, so it doesn't impact every build. This check was already performed with other auth mechanisms, this is just making sure it is being done for luci-auth as well.
Bug: b/330339907
Change-Id: I8ea97c8de0f2d74c2d735fa959d8227cec35df46
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5379037
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Instead of printing a command-line, we just directly call into the respective main functions from Python. This saves spawning another interpreter and prevents things that can go wrong from having to quote, unquote, split and tunnel arguments through shells.
Part of my bigger auto{ninja,siso} refactoring.
Tested:
- Handling of the ^^ suffix on Windows still works correctly.
- Handling of error codes - i.e.; making sure
"autoninja base_unittests && base_unittests.exe" behaves properly
in the success/failure case.
- Make sure the command prompt title is reliably reset on exit.
I tested autoninja with all combinations of these:
- Host platform: Linux, macOS, Windows
- Remote GN args: <none>, use_goma=true, use_remoteexec=true
- Siso GN args: <none>, use_siso=true
- Targets: base, ../../base/types/expected_macros_unittest.cc^ (on Linux) and ../../base/types/expected_macros_unittest.cc^^ (on Windows)
R=brucedawson@chromium.org, jwata@google.com, tikuta@chromium.org
Bug: b/293657720
Change-Id: I275a775fdc5abb6555f79d4beab76cd0914d4bd6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4924185
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Philipp Wollermann <philwo@chromium.org>
Did you know that args.gn files can have import statements and
conditionals? I did not, but apparently some developers make use of both
of these.
Supporting import statements is not too hard, so this change adds this
support. Supporting conditionals is possible, but risks turning
autoninja into a turing complete language which is more than I think we
want to do.
This doesn't use the similar code in tools/mb/mb.py because that
code is complex, and relies on the script location to find the src
directory.
This change also updates two of the existing test conditionals that
were not quite sufficient - ninja/autoninja default to num-cores
plus 2 so > cpu_count() is actually not sufficient to prove
anything.
Bug: 1482404
Change-Id: I0539d8068af59d11927cbdad260278a24ab912e6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4864898
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
This CL makes autoninja raise the number of allowed open file
descriptors to its hard limit, instead of a random value.
Also, independently of whether this operation was successful or not, the
ninja j value is capped at 80% of the current value of allowed open file
descriptors. This approximately matches the previous value of 200 for
the default value of 256 in macOS, but it's also more future-proof.
Lastly, this changes are also applied on Linux.
Change-Id: Idf2cd08384fe9f2bc699293d7062122590284dba
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4852717
Commit-Queue: Henrique Ferreiro <hferreiro@igalia.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.
Reformatted using:
files=( $(
git ls-tree -r --name-only HEAD | \
grep -Ev -e '^(third_party|recipes)/' | \
grep '\.py$';
git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"
The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py
These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py
Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
autoninja tries to detect if a user switches an output directory from
ninja to siso without doing a "gn clean" between. Initiallly this
detection looked for .ninja_deps, but this file doesn't always exist
after a ninja build. This change switches to looking for .ninja_log.
However autosiso creates a .ninja_log file (currently zero length) so
the new check is:
a) If .siso_deps exists then a siso build has been done and only siso
builds are allowed.
b) If .siso_deps doesn't exist but .ninja_log does then a ninja build
has been done and only ninja builds are allowed.
c) If neither file exists then the directory was cleaned and any type of
build is allowed.
Bug: b/293657720
Change-Id: I2cdba74f0894e62aa0e68f91e225f497b2425f45
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4784103
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
autoninja is intended to mean "build this directory, don't bother me
with the details". As we transition to siso it is therefore appropriate
that it should take on the duty of deciding whether to invoke ninja or
siso.
By looking for a use_siso gn arg autoninja can easily do this.
This change relies on crrev.com/c/4753433 to add the use_siso gn arg.
This change also teaches autoninja to detect if a user switches
between siso and ninja without doing a gn clean inbetween.
Note that this change also teaches autoninja to invoke autosiso or
siso ninja based on whether use_remoteexec is true.
Bug: b/293657720
Change-Id: I3ad36a81857e75ffe6babc4f107b777e733a285b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4749722
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
On macOS the default limit on open file descriptors is really low
and causes ninja to fail with 'Too many open files' if run with a
value of -j greater than 200.
Running only 200 jobs however slow down the build considerably so
check whether the limit has been raised by the user or if it can
be raised programmatically. In the positive case, use a limit of
800 jobs, otherwise settle down on 200 jobs.
This should allow running ninja with a large number of jobs even
on macOS Ventura 13.5 which now requires to set the limit both in
/Library/LaunchDaemons/limit.maxfiles.plist and via ulimit -n.
Bug: 1467777
Change-Id: Ib8b7d0d1ee47d243c1872229c5340e7795c1b42e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4725183
Auto-Submit: Sylvain Defresne <sdefresne@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Sylvain Defresne <sdefresne@chromium.org>
The limit set in /Library/LaunchDaemons/limit.maxfiles.plist appears
to no longer be respected in macOS Ventura 13.5 and the OS forces a
limit of 256 file descriptors, causing build with autoninja to fail
on macOS running this most recent version of the OS.
Force the max number of process to 200 until a new working way to
increase the file descriptor limit is found. This will allow devs
working on macOS to be able to build (albeit slower than before).
Fixes the following build failure when goma is enabled and running
on macOS Ventura 13.5:
$ autoninja -C out/Debug-iphonesimulator chrome
ninja: Entering directory `out/Debug-iphonesimulator'
[0/3574] CXX ....oninja: fatal: pipe: Too many open files
Bug: 1467777
Change-Id: Ia7eaab552f7e6d26a2f48d72bb8235a70d6d442f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4720227
Reviewed-by: Dirk Pranke <dpranke@google.com>
Auto-Submit: Sylvain Defresne <sdefresne@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Chromecast builds use a buildflag called `use_rbe` instead of
`use_remoteexec` or `use_goma`.
Previous attempt at crrev.com/c/4144469 tried recycling the
use_remoteexec variable for the use_rbe case, but that caused
problems with Cast CI because it would hit the error case
where the reclient binary isn't found.
This new attempt introduces a new dedicated use_rbe variable
that skips that check intended only for use_remoteexec.
Bug: b/266099996
Test: run `autoninja` in the chromecast internal repo
Change-Id: Ieaf3af709589fe1b8611904afc2fd80284b333b8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4179133
Auto-Submit: Simeon Anfinrud <sanfin@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
