Always displaying the commit message/git log makes it easier to
upload a manual roll with private details. Omit the git log if rolling
from a non-public host. The `--always-log` flag can be used to include
it anyway.
Bug: 332331835
Change-Id: Ic9201d5323fa04a55dcf4a451a9f76c48eebfd5e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5444190
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Allows multiple objects to use the same directory path.
This requires that each object has its own unique hash file and
unique migration file name. All object names must be unique.
Also update download_from_google_storage to check for the unique
migration file name.
Remove ConvertToGcs smoke tests since we're not converting
any git <-> gcs deps.
Example:
```
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'not llvm_force_head_revision',
'bucket': 'chromium-browser-clang',
'objects': [
{
'object_name': 'Linux_x64/clang-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz',
'sha256sum': '7b33138d8592199f97d132242d7b3e10f460c5c9655d49a3ad3767218fba7a77',
'size_bytes': 50212876,
},
{
'object_name': 'Linux_x64/llvmobjdump-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz',
'sha256sum': '14d669650cd212eb0ccb8c34a9e655338dfdee66fe2ecdaa517f6bd607c09a97',
'size_bytes': 5302312,
},
]
},
```
TODO: update .gitignore to search for *_is_first_class and *_hash
Bug: b/324418194
Change-Id: I89d34b06ee24f4c1aa316cd51530ad078e823143
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5420793
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Include a luci-auth check of the account logged in.
This is to check that a user isn't using a non-google account on a corp machine. Also this check is only performed every 12 hours, so it doesn't impact every build. This check was already performed with other auth mechanisms, this is just making sure it is being done for luci-auth as well.
Bug: b/330339907
Change-Id: I8ea97c8de0f2d74c2d735fa959d8227cec35df46
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5379037
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Paths containing ':' characters are illegal only on Windows platforms.
So, ssh_url like git@github.com:chromium/chromium.git will generate
runtime exception on Windows. We replace ':' with '__' to avoid
exceptions in the ssh form of git_url case.
R=ddoman@chromium.org
Bug: 329781388
Change-Id: Ia4db6a0927b40b521477e5d90addb4056c2c8ef8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5369296
Auto-Submit: 吴金立 <wujinli.cn@gmail.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
If is_first_class_gcs file exists with contents '1'
- GCS hooks will download content and then overwrite the file with '0'
- First class GCS deps will download as necessary by checking the hash
file.
If there is no is_first_class_gcs file or its content is '0' for a GCS hook:
- Hook will run normally and, if content is downloaded, an is_first_class_gcs
file will be written with '0'
If there is no is_first_class_gcs file or its content is '0' for a first
class gcs dep:
- Content will be downloaded and is_first_class_gcs will be set to '1'
Verified locally with migrating and unmigrating a GCS hook.
TODO: update other GCS hook scripts in chromium/src to check migration
file.
Bug: b/324418194
Change-Id: Ida8541cb70839b86e99115bcabfc38bc87e40136
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5368250
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
When an 'output_file' arg is included, the downloaded object from GCS
will be downloaded in that path.
Example:
```
'src/buildtools/linux': {
'bucket': 'bucket123',
'object_name': 'clang-format-version123',
'dep_type': 'gcs',
'sha256sum': 'abcd123',
'output_file': 'clang-format',
}
```
The GCS object will be downloaded at src/buildtools/linux/clang-format
Bug: b/324418194
Change-Id: I1049abeb09a1027c5477d955e50611d43015d0a4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5353387
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Also take out GCS calling logic from download_google_storage and
into call_google_storage.
GCS deps look like:
'src/third_party/node/linux': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-nodejs/20.11.0',
'object_name': '46795170ff5df9831955f163f6966abde581c8af',
'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
},
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-browser-clang',
'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
},
Example directory for src/third_party/node/linux after gclient sync:
- tar_file.gz is the downloaded file from GCS.
- node_linux_x64/ is extracted in its path.
- `hash` contains the sha of GCS filename.
```
chromium/src/ ->
third_party/node/linux/ ->
hash, tar_file.gz, node_linux_x64/
```
Bug: b/324418194
Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Use the log directory for the hash used to determime the socket used, so
developers get a distinct socket per build instead of per build
directory. Should allow concurrent builds against the same output
directory as much as ninja allows it.
Bug:b/321554715
Change-Id: I714dd2d32e2c651075055134e1568423a43eabdb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5290983
Auto-Submit: Michael Savigny <msavigny@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
git cl upload prompts a user if they upload a particularly large change,
determined by the number of CCs added on the change. With the watchlists
analyzer and eventual removal of CCing watchers from git cl, this check
needs to be updated to work without knowing the number of CCs.
This change does two things:
1. Moves the check from git cl upload to a pan-project presubmit check.
2. Update the heuristic of a LSC from > 100 CCs to > 100 files modified.
Bug: 324562917
Change-Id: I6bd0db9fde942273acebc320c8b52e81b02bd42f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5296199
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Had some debugging logging in autoninjas test. Removed it. Lines were erroneously added in https://crrev.com/c/5272474 to address b/277197166
Lines initially added to debug a problem in during development. In the initial review I received a +1 CR with a comment requesting they be removed. I made the change, started the patchset upload, went to a meeting, forgot to enter the patchset comment (thinking I had done it before the meeting) and submitted the change without the patchset. Basically context switching got me 😅
Bug: b/277197166
Change-Id: I504569c713dbca7302988c39230caf9e9e6fa2c0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5292094
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Auto-Submit: Michael Savigny <msavigny@google.com>
This reverts commit 3569608028.
Reason for revert: This includes a fix for crbug.com/324358728.
The rebase-update command has logic which tries to specifically set a key to an empty string and this has been intentionally set this way[1]. The new SetConfig implementation does treats empty string as None and hence tries to unset the config, resulting in error code 5. The patchset 2 fixes this bug and adds a test to ensure SetConfig can set an empty string to be backward compatible.
[1] https://codereview.chromium.org/228353003
Original change's description:
> Revert "Update gclient to use git config caching"
>
> This reverts commit 3edda8d185.
>
> Reason for revert: Breaks rebase-update; crbug.com/324358728
>
> Original change's description:
> > Update gclient to use git config caching
> >
> > This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
> >
> > This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
> >
> > A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
> >
> > Benchmarking
> > ============
> > With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
> >
> > Windows Benchmark
> > =================
> > Baseline (gpaste/6360045736951808): ~1min 12 sec.
> > With Caching (gpaste/6480065209040896): ~1min 3sec.
> > ~12.5% decrease in gclient sync noop runtime.
> >
> > Linux Benchmark
> > ===============
> > Baseline (gpaste/4730436763254784): ~3.739 sec.
> > With Caching (gpaste/4849870978940928): ~3.534 sec.
> > ~5.5% decrease in gclient sync noop runtime.
> >
> > Bug: 1501984
> > Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
> > Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
>
> Bug: 1501984
> Change-Id: I4a603238d9ed43edafc8e574493800670520a1d9
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5279198
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bug: 1501984
Change-Id: I405abc16c2ef6f0689031c82c61af71aad302122
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280779
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This reverts commit 3edda8d185.
Reason for revert: Breaks rebase-update; crbug.com/324358728
Original change's description:
> Update gclient to use git config caching
>
> This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
>
> This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
>
> A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
>
> Benchmarking
> ============
> With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
>
> Windows Benchmark
> =================
> Baseline (gpaste/6360045736951808): ~1min 12 sec.
> With Caching (gpaste/6480065209040896): ~1min 3sec.
> ~12.5% decrease in gclient sync noop runtime.
>
> Linux Benchmark
> ===============
> Baseline (gpaste/4730436763254784): ~3.739 sec.
> With Caching (gpaste/4849870978940928): ~3.534 sec.
> ~5.5% decrease in gclient sync noop runtime.
>
> Bug: 1501984
> Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Bug: 1501984
Change-Id: I4a603238d9ed43edafc8e574493800670520a1d9
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5279198
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
Benchmarking
============
With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
Windows Benchmark
=================
Baseline (gpaste/6360045736951808): ~1min 12 sec.
With Caching (gpaste/6480065209040896): ~1min 3sec.
~12.5% decrease in gclient sync noop runtime.
Linux Benchmark
===============
Baseline (gpaste/4730436763254784): ~3.739 sec.
With Caching (gpaste/4849870978940928): ~3.534 sec.
~5.5% decrease in gclient sync noop runtime.
Bug: 1501984
Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
git_cache populate can get interrupted midway (e.g. by LUCI CV sending a
signal that build is no longer needed). When that happens, a git mirror
may be in a state where some commits are available, but cloning such
repositry results in an empty repsitory.
This change ensures that the initial fetch operation finished
successfully.
R=ddoman@google.com
Bug: 1517944
Change-Id: I1ee860860877dbfff7a444b45fe4515fe26b248c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5219032
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
