InTheForest/chromium_depot_tools
1
0
Fork 0
mirror of https://chromium.googlesource.com/chromium/tools/depot_tools.git synced 2026-01-11 18:51:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
11,947 Commits 6 Branches 0 Tags
ecbf89e1da46853986070273dd13cf067f698b0a
Commit Graph

151 Commits

Author SHA1 Message Date
Alex Ovsienko
f107245af8 siso: set GOOGLE_API_USE_CLIENT_CERTIFICATE for googlers that run siso.py without running autoninja.py 
apply_telemetry_flags runs only for googlers, so we don't check if it's the googler running the code again.

Additionally amend the comment.

Bug: b/348530235
Change-Id: I3b1ef1edc8c03a4aba11bdc474560de66a6a6964
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7143899
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Alex Ovsienko <ovsienko@google.com>
 2025-11-11 22:33:43 -08:00
Alex Ovsienko
c70073ccc7 Make system limits check run before Siso. 
If this is not done, the offline mode of Siso will fail on mac when soft limit is insufficient.

Change-Id: I0f6ff27917a508aefdd13c43c74fece26a6a6964
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7140025
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Alex Ovsienko <ovsienko@google.com>
 2025-11-11 21:22:57 -08:00
Alex Ovsienko
2b4898e327 Reland "Enable telemetry flags for Google corp machines only." 
This reverts commit 6f83fce865.

Reason for revert: Make sure that the flags are applied only for ninja runs.

Original change's description:
> Revert "Enable telemetry flags for Google corp machines only."
>
> This reverts commit 4e2be179a1.
>
> Reason for revert: 
> This caused exception
>
> ```
> + autoninja -strict_remote -d keepdepfile -d keeprsp -C out/Default chrome
> siso ninja -strict_remote -d keepdepfile -d keeprsp -C out/Default chrome
> ninja: Entering directory `out/Default'
>  4.51s init credentials by "credhelper"
> https://console.cloud.google.com/logs/viewer?project=rbe-chrome-untrusted&resource=generic_task/task_id/25d8cbb8-4fa1-486d-ba78-7865d0eea5ac
> use RBE instance "projects/rbe-chrome-untrusted/instances/default_instance"
>  2.27s loading/recompacting deps log
>  3.47s Regenerating ninja files
>  1.17s load siso config
> build finished
> local:3636 remote:33214 cache:3317 cache-write:0(err:0) fallback:0 retry:0 skip:40752
> reapi: ops: 102977(err:33221) / r:46503(err:8) 3.13GiB / w:72424(err:0) 152.35MiB
> fs: ops: 2790971(err:182693) / r:227138(err:0) 6.68GiB / w:13508(err:4) 1.87GiB
>     resource/capa used(err)  wait-avg |   s m |  serv-avg |   s m |
>   pool=action/128   2372(0)     0.00s |█      |     0.30s | █▅▂▂▂ |
>         rbe:sched  33214(0)     0.01s |▃█▂▂   |     6.43s |  ▂█▃▂ |
>        rbe:worker  33214(0)     0.87s |    █  |     5.56s |  ▂█▃▂ |
>  remoteexec/10240  33214(0)     0.10s |█▂▂▂   |    15.37s |   ██▂ |
>  1.00s shutdown cloud logging/monitoring
> 4m08.96s Build Succeeded: 40167 steps - 161.34/s
> flag provided but not defined: -enable_cloud_monitoring
> summarize siso_metrics.json
>
>  $ siso metrics summary -C <dir> \
>     [--step_types <types>] \
>     [--elapsed_time_sorting] \
>     [--elapsed_time=run|step] \
>     [--input siso_metrics.json]
>
> summarize <dir>/.siso_metrics.json (--input)
> as depot_tools/post_ninja_build_summary.py does.
>   -C string
>         ninja running directory, where siso_metrics.json exists (default ".")
>   -elapsed_time string
>         metrics to use for elapsed time. "run" or "step". "run": time to run local command or call remote execution.  "step": full duration for the step, including preproc, waiting resource to run command etc. (default "run")
>   -elapsed_time_sorting
>         Sort output by elapsed time instead of weighted time
>   -input string
>         filename of siso_metrics.json to summarize (default "siso_metrics.json")
>   -step_types string
>         semicolon separated glob patterns (go filepath.Match) for build-step grouping
> Traceback (most recent call last):
>   File "/usr/local/google/home/tikuta/depot_tools/post_build_ninja_summary.py", line 390, in <module>
>     sys.exit(main())
>              ^^^^^^
>   File "/usr/local/google/home/tikuta/depot_tools/post_build_ninja_summary.py", line 375, in main
>     subprocess.run(cmd, check=True)
>   File "/usr/local/google/home/tikuta/depot_tools/bootstrap-2@3.11.8.chromium.35_bin/python3/lib/python3.11/subprocess.py", line 571, in run
>     raise CalledProcessError(retcode, process.args,
> subprocess.CalledProcessError: Command '['/usr/local/google/home/tikuta/depot_tools/bootstrap-2@3.11.8.chromium.35_bin/python3/bin/python3', '/usr/local/google/home/tikuta/depot_tools/siso.py', 'metrics', 'summary', '-C', 'out/Default', '--step_types', 'trict_remote']' returned non-zero exit status 2.
> ```
>
> Original change's description:
> > Enable telemetry flags for Google corp machines only.
> >
> > Bug: b/348530235
> > Change-Id: I4ac32c60b806f44920bd06e31ec6c3b86a6a6964
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7104218
> > Commit-Queue: Alex Ovsienko <ovsienko@google.com>
> > Reviewed-by: Junji Watanabe <jwata@google.com>
>
> Bug: b/348530235
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Change-Id: Iff13fa73745e09d2dab4278e9571f7ed77bf3d0c
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7136522
> Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
> Reviewed-by: Alex Ovsienko <ovsienko@google.com>
> Commit-Queue: Alex Ovsienko <ovsienko@google.com>

Bug: b/348530235
Change-Id: I6307ea67de0045c705e087e1711b295c8cce4703
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7136644
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Alex Ovsienko <ovsienko@google.com>
 2025-11-09 20:17:56 -08:00
Takuto Ikuta
6f83fce865 Revert "Enable telemetry flags for Google corp machines only." 
This reverts commit 4e2be179a1.

Reason for revert: 
This caused exception

```
+ autoninja -strict_remote -d keepdepfile -d keeprsp -C out/Default chrome
siso ninja -strict_remote -d keepdepfile -d keeprsp -C out/Default chrome
ninja: Entering directory `out/Default'
 4.51s init credentials by "credhelper"
https://console.cloud.google.com/logs/viewer?project=rbe-chrome-untrusted&resource=generic_task/task_id/25d8cbb8-4fa1-486d-ba78-7865d0eea5ac
use RBE instance "projects/rbe-chrome-untrusted/instances/default_instance"
 2.27s loading/recompacting deps log
 3.47s Regenerating ninja files
 1.17s load siso config
build finished
local:3636 remote:33214 cache:3317 cache-write:0(err:0) fallback:0 retry:0 skip:40752
reapi: ops: 102977(err:33221) / r:46503(err:8) 3.13GiB / w:72424(err:0) 152.35MiB
fs: ops: 2790971(err:182693) / r:227138(err:0) 6.68GiB / w:13508(err:4) 1.87GiB
    resource/capa used(err)  wait-avg |   s m |  serv-avg |   s m |
  pool=action/128   2372(0)     0.00s |█      |     0.30s | █▅▂▂▂ |
        rbe:sched  33214(0)     0.01s |▃█▂▂   |     6.43s |  ▂█▃▂ |
       rbe:worker  33214(0)     0.87s |    █  |     5.56s |  ▂█▃▂ |
 remoteexec/10240  33214(0)     0.10s |█▂▂▂   |    15.37s |   ██▂ |
 1.00s shutdown cloud logging/monitoring
4m08.96s Build Succeeded: 40167 steps - 161.34/s
flag provided but not defined: -enable_cloud_monitoring
summarize siso_metrics.json

 $ siso metrics summary -C <dir> \
    [--step_types <types>] \
    [--elapsed_time_sorting] \
    [--elapsed_time=run|step] \
    [--input siso_metrics.json]

summarize <dir>/.siso_metrics.json (--input)
as depot_tools/post_ninja_build_summary.py does.
  -C string
        ninja running directory, where siso_metrics.json exists (default ".")
  -elapsed_time string
        metrics to use for elapsed time. "run" or "step". "run": time to run local command or call remote execution.  "step": full duration for the step, including preproc, waiting resource to run command etc. (default "run")
  -elapsed_time_sorting
        Sort output by elapsed time instead of weighted time
  -input string
        filename of siso_metrics.json to summarize (default "siso_metrics.json")
  -step_types string
        semicolon separated glob patterns (go filepath.Match) for build-step grouping
Traceback (most recent call last):
  File "/usr/local/google/home/tikuta/depot_tools/post_build_ninja_summary.py", line 390, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/google/home/tikuta/depot_tools/post_build_ninja_summary.py", line 375, in main
    subprocess.run(cmd, check=True)
  File "/usr/local/google/home/tikuta/depot_tools/bootstrap-2@3.11.8.chromium.35_bin/python3/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/google/home/tikuta/depot_tools/bootstrap-2@3.11.8.chromium.35_bin/python3/bin/python3', '/usr/local/google/home/tikuta/depot_tools/siso.py', 'metrics', 'summary', '-C', 'out/Default', '--step_types', 'trict_remote']' returned non-zero exit status 2.
```

Original change's description:
> Enable telemetry flags for Google corp machines only.
>
> Bug: b/348530235
> Change-Id: I4ac32c60b806f44920bd06e31ec6c3b86a6a6964
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7104218
> Commit-Queue: Alex Ovsienko <ovsienko@google.com>
> Reviewed-by: Junji Watanabe <jwata@google.com>

Bug: b/348530235
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Change-Id: Iff13fa73745e09d2dab4278e9571f7ed77bf3d0c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7136522
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Alex Ovsienko <ovsienko@google.com>
Commit-Queue: Alex Ovsienko <ovsienko@google.com>
 2025-11-09 18:39:02 -08:00
Alex Ovsienko
4e2be179a1 Enable telemetry flags for Google corp machines only. 
Bug: b/348530235
Change-Id: I4ac32c60b806f44920bd06e31ec6c3b86a6a6964
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7104218
Commit-Queue: Alex Ovsienko <ovsienko@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
 2025-11-09 17:13:31 -08:00
Junji Watanabe
45aaf4f713 autoninja: Stop reading use_reclient_on_{siso, ninja} default values 
Reclient is being removed from Chromium.
Siso/Reclient mode shouldn't be used anymore.

Bug: 379584977
Change-Id: I4cbf5f281e0a272b3ade72085f64c91969588bdc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7099239
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
 2025-10-29 18:11:02 -07:00
Alex Ovsienko
42ee589e26 Upload siso_metrics.json and siso_metadata.json file to GCS bucket from user machine for autoninja 
Make the uploader work only for googlers for now.

Bug: b/449906174

Change-Id: If3f9fec045812b8c9f3e04a3d7dd87d06a6a6964
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7019134
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Alex Ovsienko <ovsienko@google.com>
 2025-10-08 22:09:03 -07:00
Fumitoshi Ukai
a551472876 autoninja: only checks cr_build_revision when reclient_cfgs is deployed via cipd 
non-Googlers may use own reclient_cfgs, which may not have
cr_build_revision marker.
Don't check cr_build_revision if reclient_cfgs is not deployed
by cipd.

Change-Id: I1be1f88c47abb2ab192222e3c7bcebbb37ee3f79
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7019373
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
 2025-10-07 23:40:01 -07:00
Fumitoshi Ukai
de69ad92fa autoninja: check rewrapper cfg version matches with clang 
rewrapper cfg should match with clang.

Change-Id: I67d4ec61e5a1199b2c514b99ef7f1464a742549e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/7019592
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
 2025-10-07 21:29:03 -07:00
Fumitoshi Ukai
f80f5bc99e autoninja: support running from outside of workspace 
gclient_paths.GetPrimarySolutionPath() finds primary
solution path from current directory, so it would fail
if it is invoked from outside of workspace.

Pass directory to gclient_paths.GetPrimarySolutionPath
so it could find workspace correctly if it is invoked
ninja outside of workspace.

Bug: 441240584
Change-Id: I873f7883873e143ec8a64ee0e636042ac2336a2a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6898614
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
 2025-09-02 13:08:00 -07:00
Erik Staab
ea5d49a781 Set --quiet when run under gemini-cli 
Bug: 436540341
Change-Id: Ic04ba12e0bcb926b90dd5dfbd3df4685cf59535a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6832667
Reviewed-by: Andrew Grieve <agrieve@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Erik Staab <estaab@chromium.org>
 2025-08-12 11:33:52 -07:00
Junji Watanabe
dd8909f212 autoninja: Do not collect Reclient build metrics 
We don't need to collect metrics from Reclient, anymore since
most builds are done by Siso.

Change-Id: I38fcb42aca966acc1b45a0876634cb9949bd255d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6733778
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
 2025-07-13 20:11:12 -07:00
Fumitoshi Ukai
c2a6159a0d autoninja: fix siso suggestion message 
Bug: 430498033
Change-Id: I1b44916bd233a8261a714c358840ffde868f0538
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6719028
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Matt Stark <msta@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
 2025-07-10 20:16:50 -07:00
Alex Ovsienko
211126cbe1 autoninja: import resource only when its needed to prevent linter from complaning. 
Linter in question: https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance - Pylance.

Change-Id: I3d8c17c533a882d08b846364249795b32c51bd27
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6693923
Auto-Submit: Alex Ovsienko <ovsienko@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Commit-Queue: Alex Ovsienko <ovsienko@google.com>
 2025-07-01 00:02:53 -07:00
Alex Ovsienko
179c4a53fc autoninja: prevent null pointer exception on _get_remoteexec_defaults returning None. 
Change-Id: Ic89607b4e234b66253923383bc8b969c1434e9f1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6693922
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Alex Ovsienko <ovsienko@google.com>
 2025-06-30 23:58:36 -07:00
Fumitoshi Ukai
6a1451c0c0 autoninja: disable siso native by --reapi_address= 
siso v1.3.1 changed the way to enable reapi.
It allows empty project and empty reapi_instance.

Under reclient mode to disable siso native,
set reapi_address empty.

Bug: 427569563
Change-Id: Ib653186c5200605032de5c0fddab0b64412db85c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6668613
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
 2025-06-24 17:43:48 -07:00
Junji Watanabe
8f87843807 autoninja: Handle -j for Siso 
This CL adds -j support for Siso builds.

- With use_remoteexec=true
  - -j will be replaced with -remote_jobs
- Without use_remoteexec=true
  - -j will be replaced with -local_jobs unless j > num_cpus

Bug: 424659245
Change-Id: I0370cfdd6e2004094a599c3aa32f02106e1b3092
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6656547
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
 2025-06-23 00:23:32 -07:00
Junji Watanabe
909e2921a0 autoninja: remove no remote execution warning 
Siso is expected to use for local builds, too.
This warning doesn't make sense now. The message for -offline
flag should be sufficient for developers to notice about it.

Bug: 424659245
Change-Id: Iedab0c5de4886c70c7fe88640332d17f7d92889f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6656548
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
 2025-06-22 18:23:52 -07:00
Junji Watanabe
9a17210302 autoninja: switch between ninja and siso for help appropriately 
When running `autoninja -help` at Chromium project root, it prints
`ninja -help`. `autoninja` should switch between `ninja -help` and `siso
ninja -help` based on the project setting.

Bug: 426149609
Fixed: 426149609
Change-Id: I1a5d1b98e906130ad17061d49a1f25d2c0119c74
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6656541
Commit-Queue: Junji Watanabe <jwata@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
 2025-06-19 21:50:43 -07:00
Junji Watanabe
377f55f382 autoninja: Enable Siso by default for external Chromium developers 
This CL changes the default build system for Chromium builds by external
developers from Ninja to Siso.
See also https://crrev.com/c/6638830 for more info.

Bug: 424659245, 409223168
Change-Id: I382746cab2e8b40db5f555711e965f814d089987
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6636690
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
 2025-06-17 17:38:43 -07:00
Junji Watanabe
6d52c22ee3 autoninja: Do not show caffeinate warning 
Discussed in
https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6596109/comments/217e68b5_bcf66ede

The warning message doesn't seem useful. This CL removes the warning
message for caffeinate, but adds help message instead. This CL also
refactors the help message for `autoninja --offline`.

```
> autoninja --help
autoninja:
  -o/--offline  temporary disable remote execution

caffeinate:
  --no-caffeinate  do not prepend `caffeinate` to ninja command

usage: ninja [options] [targets...]
...
```

Change-Id: I152a144efbb6002e3bfb03b23d1e0eb2d3b2af1d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6620110
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Nico Weber <thakis@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Adam Norberg <norberg@google.com>
 2025-06-04 20:02:25 -07:00
Junji Watanabe
a8900cc0f0 autoninja: Read default use_siso variable from .gn 
This CL allows non-Chromium projects to enable Siso by default via
`.gn`.

How to determine use_siso default:
1) If //build/config/siso/.sisoenv doesn't exist, use Ninja.
2) If it's not on corp machine, use Ninja.
3) If .gn specifies use_siso=True, use Siso.
4) If //build/config/gclient_args.gni specifies
`build_with_chromium = true`, use Siso.
5) If the build dir has already used Siso, use Siso.
Otherwise, use Ninja.

Note that the condition 4) will be removed after Chromium specifies
use_siso=true in .gn and waits for a while.

Bug: 409223168
Change-Id: I05d36b0e4e0c50cca8a1587135724ec837f4ee36
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6585977
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Junji Watanabe <jwata@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
 2025-05-25 23:06:39 -07:00
Andrew Grieve
73a3fbf1fa autoninja.py: Make use_android_build_server enabled by default 
We're switching the GN arg to be default-enabled for local builds.

Bug: 404503040
Change-Id: I710b8fa38bf4ce97d592861b28a49b0bc867b702
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6454372
Auto-Submit: Andrew Grieve <agrieve@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Andrew Grieve <agrieve@chromium.org>
 2025-04-15 07:03:49 -07:00
Fumitoshi Ukai
079673b191 autoninja: check internal if not use rbe-chromium-untrusted 
non-Eng users don't have access to src-internal, so
need to use rbe-chromium-untrusted even on corp machine.

Bug: 408114984
Change-Id: I8956926a8723fbee70b2defb2f6f49bccc37e22e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6433491
Reviewed-by: Philipp Wollermann <philwo@google.com>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
 2025-04-04 01:01:49 -07:00
Fumitoshi Ukai
5e89ddacb7 autoninja: add comment on use_siso_default 
Change-Id: Ia22ae12f15b845664e3e88f27731506f24c9b5e9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6356123
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
 2025-03-13 22:28:57 -07:00
Fumitoshi Ukai
eae464cbfe autoninja: don't keep use_reclient=true with .reproxy_tmp 
//build/toolchain/rbe.gni side fix: https://chromium-review.googlesource.com/c/chromium/src/+/6355944

Bug: 379584977
Change-Id: I2c0947ed3a7355ce3fe332e3e8a56655c0cf8ff8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6355865
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
 2025-03-13 21:53:15 -07:00
Fumitoshi Ukai
f638e8a918 autoninja: use build/toolchain/remoteexec_defaults.gni for use_reclient 
Instead of using build/toolchain/use_reclient_value.py in
both autoninja and gn, use build/toolchain/remoteexec_defaults.gni
to set default value for use_reclient (on siso or on ninja).

Bug: 397994249
Change-Id: Ia4302fe8794218d485e52ca38f8523453814a6b9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6301378
Reviewed-by: Andrew Grieve <agrieve@chromium.org>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
 2025-02-26 22:01:53 -08:00
Mohamed Heikal
678f73bd0e [autoninja] Pass the output directory to the build server 
When registering the build with the build server, pass in the output
directory.

Change-Id: Ie6ed0bf5d73926fb163085067aafcd350063ee5a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6249305
Auto-Submit: Mohamed Heikal <mheikal@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Commit-Queue: Mohamed Heikal <mheikal@chromium.org>
 2025-02-26 09:05:49 -08:00
Fumitoshi Ukai
deb5575fbd autoninja: move use_siso default logic in autoninja.py 
chromium now doesn't use `use_siso` value for build graph,
so no need to use use_siso_default.py in gn context.
use_siso value is only used to determine autoninja use
siso or ninja, so put use_siso default logic in autoninja.py.

Will remove build/toolchain/use_siso_default.py in chromium

Bug: 397994249
Change-Id: Idb31c4e75f0a1c3bc4a97089494ec16be9879a49
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6301272
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
 2025-02-25 23:28:16 -08:00
Fumitoshi Ukai
6cf251b517 autoninja: fix UnboundLocalError 
Change-Id: Iff96b697b3c1f1d88c13d099ffec4c4159e002bd
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6276006
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Matt Stark <msta@google.com>
 2025-02-17 16:22:33 -08:00
Fumitoshi Ukai
7a0dbd2074 autoninja: set RBE_remote_disabled when offline even with siso 
Bug: 396724264
Change-Id: Id92e8d532e571420ff64be3da207ffd51fd282e9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6272278
Commit-Queue: Junji Watanabe <jwata@google.com>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
 2025-02-16 20:13:36 -08:00
Matt Stark
ce598256f2 Call use_siso_default_and_suggest_siso where possible. 
Said function is defined in https://crrev.com/c/6217339.

Bug: 393209062

Change-Id: I1bced33272a6dd19d4e71f96f80ac6de61b48d9e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6217100
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Matt Stark <msta@google.com>
 2025-01-29 17:46:21 -08:00
Matt Stark
c40e9fe596 Only call _get_use_siso_default if use_siso wasn't explicitly set. 
This allows us to configure the get_use_siso_default to give a notification suggesting people to switch.

Bug: 393209062

Change-Id: I1bced33272a6dd19d4e71f96f80ac6de61b48d9d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6217099
Commit-Queue: Matt Stark <msta@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
 2025-01-29 16:44:53 -08:00
Philipp Wollermann
49e10123de autoninja: Add workaround for slow reproxy starts 
Somewhere deep in the Google Cloud Go SDK is a call to user.Current() to
get the user's home directory, in order to find some configuration files
necessary for TLS Client Certificate authentication.

The user.Current() function in Go versions prior to the yet unreleased
Go 1.24 can be very slow on Windows, as it causes a network call to the
Active Directory server to be made if the machine is part of a Windows
domain.

As we don't use TLS Client Certificate at Google, we can safely disable
this code path by setting GOOGLE_API_USE_CLIENT_CERTIFICATE=false in the
environment prior to launching reproxy and other tools using the SDK.

As I'm not sure whether this would cause issues for people actually
using TLS Client Certificate auth with their REAPI backend, let's only
apply this workaround on Google corp machines. It's highly unlikely that
others are affected by this particular issue anyway, and when Go 1.24
is released in February, it will fix the underlying issue anyway.

Bug: 388195531
Change-Id: Ia9d4b74c251b303b66b19fb59957ff5a922d883b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6191410
Auto-Submit: Philipp Wollermann <philwo@google.com>
Commit-Queue: Philipp Wollermann <philwo@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
 2025-01-22 16:23:40 -08:00
Fumitoshi Ukai
d2c6b34294 autoninja: run ninjalog_uploader in new session 
Bug: 390332232
Change-Id: I5bbe95e1d26899ab7dfbba418d44e250d9277fc9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6171585
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
 2025-01-16 21:45:55 -08:00
Mohamed Heikal
9b4d1e485d Integrate autoninja.py with fast_local_dev_server.py 
- Starts the build server when the build starts.
- Writes tty filename to env variable.
- Tells the build server about the current build so it does not exit
  until autoninja does, even if idle.
- Cancels pending tasks on Ctrl+c.

Change-Id: I86bb9852bd0975f381b049b9ff21c38eef7cef9d
Bug: 370589852
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5917985
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Mohamed Heikal <mheikal@chromium.org>
Commit-Queue: Mohamed Heikal <mheikal@chromium.org>
 2024-12-12 10:28:46 -08:00
Fumitoshi Ukai
25fede45b3 autoninja: report if 'use_remoteexec=true' is missing. 
Change-Id: Id5c61b31dd46e77ff1308746e388b2347919a0a3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6082931
Reviewed-by: Peter Boström <pbos@chromium.org>
Commit-Queue: Peter Boström <pbos@chromium.org>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
 2024-12-10 00:29:29 +00:00
Philipp Wollermann
885ed47343 Remove assert statements in autoninja.py 
These were introduced in crrev.com/c/6024739 and currently break
autoninja for non-Googlers who manually enable Siso by setting the
use_siso=true GN arg.

$ gn gen --args='use_siso=true' out/Debug
$ autoninja -C out/Debug base
Traceback (most recent call last):
  File "/home/philwo/depot_tools/autoninja.py", line 504, in <module>
    sys.exit(main(sys.argv))
                ^^^^^^^^^^^^^^
  File "/home/philwo/depot_tools/autoninja.py", line 493, in main
    exit_code = _main_inner(input_args, build_id, should_collect_logs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/philwo/depot_tools/autoninja.py", line 249, in _main_inner
    assert use_siso != False
           ^^^^^^^^^^^^^^^^^
AssertionError

R=ukai@google.com

Change-Id: I6536576d39b4ef094daa4e7ec80f3d743ecefd85
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6058992
Auto-Submit: Philipp Wollermann <philwo@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
 2024-12-02 00:05:40 +00:00
Aleksey Khoroshilov
5786e1ee10 Fix reproxy start if use_reclient is not set or found 
If use_reclient_value.py is not found and args.gn only has
use_remotexec=true, the autoninja fails to start reproxy.

Change-Id: I6608ea48b66864bf2ab89176b6b01a79e07b7564
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6059221
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Aleksei Khoroshilov <akhoroshilov@brave.com>
 2024-11-29 08:24:17 +00:00
Fumitoshi Ukai
db017d1ffe autoninja: use build/toolchain/use_reclient_value.py 
https://crrev.com/c/6048393 provides build/toolchain/use_reclient_value.py
to get value for use_reclient.

Use it in autoninja if it exists.

Bug: 341167943
Change-Id: Ia6207ea044cc109fa156deff3c3357475368576f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6024739
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
 2024-11-29 03:05:40 +00:00
Fumitoshi Ukai
af449cb0bc autoninja: use build/toolchain/use_siso_default.py 
https://chromium-review.googlesource.com/c/chromium/src/+/5901413
provides build/toolchain/use_siso_default.py
to decide default value for use_siso.

Use it in autoninja if it exists.

Bug: 341167943
Change-Id: I379942b10063ea87008e044268fbd4155e6ef552
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6024737
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
 2024-11-20 05:13:52 +00:00
Fumitoshi Ukai
37c0cf9100 autoninja: skip RBE wrapper for -t tool invocations for siso 
ninja tools don't need an RBE environment active, so avoid an
unnecessary delay and noisy output when something like autoninja -t
commands is invoked.

Change-Id: Ieb58dee56b15968dcac9483ff41be25544f02470
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5991650
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
 2024-11-05 01:54:12 +00:00
Tomasz Śniatowski
d4d09a4536 autoninja: skip RBE wrapper for -t tool invocations 
ninja tools don't need an RBE environment active, so avoid an
unnecessary delay and noisy output when something like autoninja -t
commands is invoked.

Change-Id: I28fe5275a4a130251d5c0feac5841ba2c410527d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5987270
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
 2024-11-05 00:19:06 +00:00
Mark Mentovai
22df6f8e62 autoninja: fix strings with invalid escape sequences 
Since 61fad561d6 (https://chromium-review.googlesource.com/c/5848450,
2024-09-11), autoninja under Python 3.12 presents these warnings:

```
…/autoninja.py:73: SyntaxWarning: invalid escape sequence '\s'
  m = re.match('instance\s*=\s*projects/([^/]*)/instances/.*', line)
…/autoninja.py:92: SyntaxWarning: invalid escape sequence '\s'
  m = re.match('SISO_PROJECT=\s*(\S*)\s*', line)
```

This warning appears in Python 3.12 ([1], [2], [3]).

'\s' and '\S' are not valid escape sequences in strings. r'\s' and r'\S'
are valid in regular expressions, but outside of raw strings, they would
need to be written as '\\s' and '\\S'.

There is no reason to not use raw strings in this case, so the new
regular expression pattern strings introduced in 61fad561d6 are
changed to raw strings.

[1] https://docs.python.org/3/whatsnew/3.12.html#:~:text=A%20backslash%2Dcharacter%20pair%20that%20is%20not%20a%20valid%20escape%20sequence%20now%20generates%20a%20SyntaxWarning%2C%20instead%20of%20DeprecationWarning.
[2] https://github.com/python/cpython/issues/98401
[3] https://github.com/python/cpython/pull/99011

Bug: b/364318216
Change-Id: I0f237976fe9c39208541ae78205f5bdbf126fa82
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5859159
Commit-Queue: Mark Mentovai <mark@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Auto-Submit: Mark Mentovai <mark@chromium.org>
 2024-09-13 16:08:45 +00:00
Max Ihlenfeldt
62475a5ed8 autoninja.py: Fix "SyntaxWarning: invalid escape sequence '\s'" 
Bug: None
Change-Id: Id19557ff5f115f1cacee85f113174535112f7c9d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5854399
Commit-Queue: Philipp Wollermann <philwo@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Auto-Submit: Max Ihlenfeldt <max@igalia.com>
 2024-09-11 15:19:56 +00:00
Aleksey Khoroshilov
1ad5b6c0df Do not fail if RBE project is empty. 
External developers may use RBE without RBE_instance set.
Autoninja should not fail in this case.

Change-Id: I2dbafded690246132bb0ff1cd080caa0b758b4f6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5851434
Auto-Submit: Aleksei Khoroshilov <akhoroshilov@brave.com>
Commit-Queue: Aleksei Khoroshilov <akhoroshilov@brave.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
 2024-09-11 06:13:32 +00:00
Fumitoshi Ukai
61fad561d6 autoninja: check RBE project, not account 
Account check would become too slow.
We'll check RBE project to use instead.

On corp machine, our policy to use @google.com account
and rbe-chrome-untrusted to build chromium/chrome.
We don't allow rbe-chromium-untrusted with @chromium.org
on corp machine.

On non-corp machine, user couldn't use rbe-chrome-untrusted
because it's @google.com only, and corp security policy
doesn't allow @google.com account on non-corp machine.

Bug: b/364318216
Change-Id: I0f3a19e105b050aef6a62e1b25b45b1722382a34
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5848450
Reviewed-by: Scott Lee <ddoman@chromium.org>
Reviewed-by: Michael Savigny <msavigny@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
 2024-09-11 00:03:49 +00:00
Fumitoshi Ukai
1f6ef165b7 autoninja: show warning if siso builtin remote exec is not used on cog 
If user forgets to set `use_reclient=false`,

```
ukai@ukai /google/cog/cloud/ukai/siso-test/chromium
 % ~/depot_tools/autoninja -C out/Default base
WARNING: You're not using Siso's built-in remote execution. The build will be slow.
You should set the following in args.gn to get better performance:
  use_remoteexec=true
  use_reclient=false
  use_siso=true

Proxy started successfully.
```

Bug: b/362301140
Change-Id: I40972b96fd2576c32232f88e6248a1000708c3e2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5816108
Reviewed-by: Junji Watanabe <jwata@google.com>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Gary Tong <gatong@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@google.com>
 2024-08-27 22:36:43 +00:00
Junji Watanabe
8f617b4617 autoninja: Add exit code and build duration to ninjalog metadata 
- exit code will be useful to distinguish between successful builds and
others.
- build duration will cover the overhead outside of ninja invocation
e.g. Reclient and Siso's startup/shutdown.

Bug: 348527311
Change-Id: Ibebdf3d64597fb0e57914a84d33e46bca66671da
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5804201
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Junji Watanabe <jwata@google.com>
 2024-08-22 02:49:42 +00:00
Junji Watanabe
1e6454f8d2 autoninja: ignore ResourceWarning from subprocess for ninjalog_uploader.py 
The main process exits right after ninja_uploader.py subprocess starts,
which may print ResourceWarning from subprocess.
e319f774f9/Lib/subprocess.py (L946-L947)
This CL ignores the warning message.

Change-Id: I800387194763e60a16437b3bd41fa282b605dda7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5782795
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
 2024-08-13 05:01:02 +00:00