Retry switching to use CIPD ninja v1.8.2

This reverts commit 0a5bae7ebc + some modifications on recipe module.

The previous CL https://crrev.com/c/3697958 ensures depot_tools/.cipd_bin in recipe environments. This CL copies ninja.exe from .cipd_bin to depot_tools root as cipd_bin_setup.bat.
For developers, it's expected to run update_depot_tools which triggers cipd_bin_setup.
For CI/CQ, depot_tools._cipd_bin_setup() should be called before using ninja/autoninja recipe module APIs.

Builds with depot_tools recipe bundle are tested using led.

# Windows
> led get-build 8811491780369249585 | led edit-recipe-bundle -O depot_tools="$HOME/depot_tools" | led launch
https://ci.chromium.org/swarming/task/5b7512ee9d91fc10?

# Linux
> led get-build 8811504372914233697 | led edit-recipe-bundle -O depot_tools="$HOME/depot_tools" | led launch
https://ci.chromium.org/swarming/task/5b7515832d0f7c10

Original change's description:
> Revert "Switch to use CIPD ninja v1.8.2"
>
> This reverts commit c90a982106.
>
> Reason for revert:
> https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket/8811949917908303825/+/u/compile/stdout
>
> Original change's description:
> > Switch to use CIPD ninja v1.8.2
> >
> > https://crrev.com/c/3674981 installed CIPD ninja packages.
> > This CL switches from the old ninja binaries to them.
> >
> > Note that this CL doesn't change ninja version. the old/CIPD ninja binaries are v1.8.2.
> >
> > The differences from the old binaries are
> > - the CIPD packages don't use chromium's clang.
> > - the CIPD package for Linux doesn't use chromium's sysroot.
> > - the min macOS version is 10.13, which was 10.6.
> > Please also see https://crsrc.org/i/3pp/ninja/install_bootstrap.sh
> >
> >
> > On Windows, ninja.exe will be installed under .cipd_bin.
> > But there are many places that assume that ninja.exe exists on depot_tools root. So this CL also copies the ninja.exe from .cipd_bin to root in cipd_bin_setup.bat.
> >
> > Bug: 931218
> > Change-Id: Ib67eee5e9b6ad9b2937b789626970d9c85867dbd
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3674982
> > Commit-Queue: Junji Watanabe <jwata@google.com>
> > Reviewed-by: Philipp Wollermann <philwo@google.com>
> > Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> > Reviewed-by: Nico Weber <thakis@chromium.org>
> > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
>
> Bug: 931218
> Change-Id: I08d61dad119a0d23ae9ec8b1ed787dd3915de697
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3686854
> Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
> Owners-Override: Takuto Ikuta <tikuta@chromium.org>

Bug: 931218
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chrome_release
Recipe-Nontrivial-Roll: infra
Change-Id: Ic7a9395c0e858823375f1a6ec79034dbc54230bb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3696963
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>

.gitignore

@@ -6,6 +6,9 @@
 /.cipd_client*
 /.versions
 
+# ignore ninja.exe copied from .cipd_bin.
+/ninja.exe
+
 # Ignore "disable auto update" sentinel file.
 .disable_auto_update
 

cipd_bin_setup.bat

@@ -3,4 +3,6 @@
 :: Use of this source code is governed by a BSD-style license that can be
 :: found in the LICENSE file.
 
-"%~dp0\cipd.bat" ensure -log-level warning -ensure-file "%~dp0\cipd_manifest.txt" -root "%~dp0\.cipd_bin"
+call "%~dp0\cipd.bat" ensure -log-level warning -ensure-file "%~dp0\cipd_manifest.txt" -root "%~dp0\.cipd_bin"
+:: copy ninja.exe to the root since many places assume ninja.exe exists in depot_tools.
+copy /y "%~dp0\.cipd_bin\ninja.exe" "%~dp0\ninja.exe" > nul

ninja

@@ -1,40 +1,11 @@
 #!/usr/bin/env bash
 
-# Copyright (c) 2012 Google Inc. All rights reserved.
+# Copyright 2022 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
-OS="$(uname -s)"
+# See revert instructions in cipd_manifest.txt
-THIS_DIR="$(dirname "${0}")"
 
-function print_help() {
+MYPATH="$(dirname "${BASH_SOURCE[0]}")"
-cat <<-EOF
-No prebuilt ninja binary was found for this system.
-Try building your own binary by doing:
-  cd ~
-  git clone https://github.com/ninja-build/ninja.git -b v1.8.2
-  cd ninja && ./configure.py --bootstrap
-Then add ~/ninja/ to your PATH.
-EOF
-}
 
-case "$OS" in
+exec "$MYPATH/.cipd_bin/ninja" "$@"
-  Linux)
-    MACHINE=$(uname -m)
-    case "$MACHINE" in
-      x86_64)
-        exec "${THIS_DIR}/ninja-linux64" "$@";;
-      *)
-        echo Unsupported architecture \($MACHINE\) -- unable to run ninja.
-        print_help
-        exit 1;;
-    esac
-    ;;
-  Darwin)    exec "${THIS_DIR}/ninja-mac" "$@";;
-  CYGWIN*)   exec cmd.exe /c $(cygpath -t windows $0).exe "$@";;
-  MINGW*)    cmd.exe //c $0.exe "$@";;
-  MSYS_NT*)  cmd.exe //c $0.exe "$@";;
-  *)         echo "Unsupported OS ${OS}"
-             print_help
-             exit 1;;
-esac

recipes/README.recipes.md

@@ -119,7 +119,7 @@ Sets a fixed revision for a single dependency using project revision
 properties.
 ### *recipe_modules* / [depot\_tools](/recipes/recipe_modules/depot_tools)
 
-[DEPS](/recipes/recipe_modules/depot_tools/__init__.py#7): [recipe\_engine/cipd][recipe_engine/recipe_modules/cipd], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime]
+[DEPS](/recipes/recipe_modules/depot_tools/__init__.py#7): [recipe\_engine/cipd][recipe_engine/recipe_modules/cipd], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime]
 
 PYTHON_VERSION_COMPATIBILITY: PY2+3
 

recipes/recipe_modules/depot_tools/__init__.py

@@ -7,6 +7,7 @@ PYTHON_VERSION_COMPATIBILITY = 'PY2+3'
 DEPS = [
   'recipe_engine/cipd',
   'recipe_engine/context',
+  'recipe_engine/file',
   'recipe_engine/platform',
   'recipe_engine/runtime',
 ]

recipes/recipe_modules/depot_tools/api.py

@@ -85,4 +85,11 @@ class DepotToolsApi(recipe_api.RecipeApi):
       self.repo_resource('.cipd_bin'),
       self.repo_resource('cipd_manifest.txt'),
       'ensure depot_tools/.cipd_bin')
+    if self.m.platform.is_win:
+      # Copy ninja.exe from .cipd_bin to depot_tools root because there are
+      # many places that assume depot_tools/ninja.exe exists.
+      self.m.file.copy(
+        'copy depot_tools/.cipd_bin/ninja.exe to depot_tools/',
+        self.repo_resource('.cipd_bin', 'ninja.exe'),
+        self.repo_resource('ninja.exe'))
     self._cipd_bin_setup_called = True

recipes/recipe_modules/depot_tools/examples/full.expected/win.json

@@ -62,6 +62,20 @@
       "@@@STEP_LOG_END@json.output@@@"
     ]
   },
+  {
+    "cmd": [
+      "vpython3",
+      "-u",
+      "RECIPE_MODULE[recipe_engine::file]\\resources\\fileutil.py",
+      "--json-output",
+      "/path/to/tmp/json",
+      "copy",
+      "RECIPE_REPO[depot_tools]\\.cipd_bin\\ninja.exe",
+      "RECIPE_REPO[depot_tools]\\ninja.exe"
+    ],
+    "infra_step": true,
+    "name": "copy depot_tools/.cipd_bin/ninja.exe to depot_tools/"
+  },
   {
     "cmd": [
       "ls",