Previously little bits of the scm.GIT were mocked (like SetConfig,
and GetConfig, but not the other config related methods).
This changes things so that the git config state is a class whose
logic is shared between prod and test, with a 'real' and 'test'
implementation which know how to load and save configuration at
a low level.
R=yiwzhang
Change-Id: I1dc11b550908862607ea539de7fa60fbce30e700
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5660891
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Allen Li <ayatane@chromium.org>
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
In preparation for some Windows optimizations to how git_common calls
git it is important to use git_common more widely, specifically from
scm.py and gclient_scm.py. This change updates scm.py and gclient_scm.py
and updates the associated tests:
Test command lines used when updating the tests include:
vpython3 tests/gclient_scm_test.py ManagedGitWrapperTestCaseMock.testUpdateConflict
vpython3 tests/gclient_scm_test.py GerritChangesTest.testRecoversAfterPatchFailure
vpython3 tests/gerrit_util_test.py CookiesAuthenticatorTest.testGetGitcookiesPath
Bug: 332982922
Change-Id: I7aacb110b2888c164259815385cd77e26942adc7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5478509
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
If symbolic-ref remote HEAD is not available, we skip setting it and
query remote Git using ls-remote. Such information is not stored, and
gclient will need to repeat it on the next invocation.
Instead, we can call set-head on symbolic-ref failure. While that's
slower operation than ls-remote, it saved in internal Git database and
can be reused on next gclient invocation.
We may be okay with hardcoding 'main' as default remote branch today,
but it's possible that some projects still use old default.
R=gavinmak@google.com
Change-Id: Ic4c826b888d96e367039bfc4b9bd2ba0d8b58b52
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5492789
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Also take out GCS calling logic from download_google_storage and
into call_google_storage.
GCS deps look like:
'src/third_party/node/linux': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-nodejs/20.11.0',
'object_name': '46795170ff5df9831955f163f6966abde581c8af',
'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
},
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-browser-clang',
'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
},
Example directory for src/third_party/node/linux after gclient sync:
- tar_file.gz is the downloaded file from GCS.
- node_linux_x64/ is extracted in its path.
- `hash` contains the sha of GCS filename.
```
chromium/src/ ->
third_party/node/linux/ ->
hash, tar_file.gz, node_linux_x64/
```
Bug: b/324418194
Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
This reverts commit 3569608028.
Reason for revert: This includes a fix for crbug.com/324358728.
The rebase-update command has logic which tries to specifically set a key to an empty string and this has been intentionally set this way[1]. The new SetConfig implementation does treats empty string as None and hence tries to unset the config, resulting in error code 5. The patchset 2 fixes this bug and adds a test to ensure SetConfig can set an empty string to be backward compatible.
[1] https://codereview.chromium.org/228353003
Original change's description:
> Revert "Update gclient to use git config caching"
>
> This reverts commit 3edda8d185.
>
> Reason for revert: Breaks rebase-update; crbug.com/324358728
>
> Original change's description:
> > Update gclient to use git config caching
> >
> > This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
> >
> > This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
> >
> > A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
> >
> > Benchmarking
> > ============
> > With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
> >
> > Windows Benchmark
> > =================
> > Baseline (gpaste/6360045736951808): ~1min 12 sec.
> > With Caching (gpaste/6480065209040896): ~1min 3sec.
> > ~12.5% decrease in gclient sync noop runtime.
> >
> > Linux Benchmark
> > ===============
> > Baseline (gpaste/4730436763254784): ~3.739 sec.
> > With Caching (gpaste/4849870978940928): ~3.534 sec.
> > ~5.5% decrease in gclient sync noop runtime.
> >
> > Bug: 1501984
> > Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
> > Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
>
> Bug: 1501984
> Change-Id: I4a603238d9ed43edafc8e574493800670520a1d9
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5279198
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bug: 1501984
Change-Id: I405abc16c2ef6f0689031c82c61af71aad302122
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280779
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This reverts commit 3edda8d185.
Reason for revert: Breaks rebase-update; crbug.com/324358728
Original change's description:
> Update gclient to use git config caching
>
> This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
>
> This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
>
> A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
>
> Benchmarking
> ============
> With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
>
> Windows Benchmark
> =================
> Baseline (gpaste/6360045736951808): ~1min 12 sec.
> With Caching (gpaste/6480065209040896): ~1min 3sec.
> ~12.5% decrease in gclient sync noop runtime.
>
> Linux Benchmark
> ===============
> Baseline (gpaste/4730436763254784): ~3.739 sec.
> With Caching (gpaste/4849870978940928): ~3.534 sec.
> ~5.5% decrease in gclient sync noop runtime.
>
> Bug: 1501984
> Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Bug: 1501984
Change-Id: I4a603238d9ed43edafc8e574493800670520a1d9
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5279198
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
Benchmarking
============
With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
Windows Benchmark
=================
Baseline (gpaste/6360045736951808): ~1min 12 sec.
With Caching (gpaste/6480065209040896): ~1min 3sec.
~12.5% decrease in gclient sync noop runtime.
Linux Benchmark
===============
Baseline (gpaste/4730436763254784): ~3.739 sec.
With Caching (gpaste/4849870978940928): ~3.534 sec.
~5.5% decrease in gclient sync noop runtime.
Bug: 1501984
Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reformat this dir by itself to help merging with conflicts with other CLs.
Reformatted using:
parallel ./yapf -i -- tests/*.py
~/chromiumos/chromite/contrib/reflow_overlong_comments tests/*.py
These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
tests/bot_update_coverage_test.py
tests/cipd_bootstrap_test.py
tests/gclient_eval_unittest.py
tests/gclient_git_smoketest.py
tests/gclient_scm_test.py
tests/gclient_smoketest.py
tests/gclient_test.py
tests/gclient_transitions_smoketest.py
tests/gclient_utils_test.py
tests/git_cl_test.py
tests/git_hyper_blame_test.py
tests/git_rebase_update_test.py
tests/lockfile_test.py
tests/metrics_test.py
tests/presubmit_canned_checks_test.py
tests/presubmit_unittest.py
tests/roll_dep_test.py
Change-Id: I8fed04b4ba81d54b8f45da612213aad27a9e1a2c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4842592
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
At the moment on repo's with many CIPD packages the number
of CLI calls to 'cipd describe' can make it very slow.
This mechanism will call cipd ensure-file-resolve with the entire
list in one call then reference items from the list before trying
individual cipd describe commands.
Bug:b/279097790
Change-Id: Ieb4bc77853357cfadb3ea6311fa0013aa7999ac6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4551125
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Rachael Newitt <renewitt@google.com>
Commit-Queue: Dan Le Febvre <dlf@google.com>
This is a reland of fc9a40e3c6
Hopefully the cause of the 2nd revert was fixed in https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3489327
Original change's description:
> Reland "Add support for Gerrit topics in gclient syncs"
>
> This is a reland of 0f13273f1f
>
> Hopefully the cause of the original revert was fixed in https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3480835
>
> Original change's description:
> > Add support for Gerrit topics in gclient syncs
> >
> >
> > If the new flag "--download-topics" is specified with a "--patch-ref" then:
> > * Finds the topic of the Gerrit change.
> > * Finds all open changes in the same repo as the Gerrit change.
> > * Cherrypicks all changes locally.
> >
> > This functionality can be used by developers and bots to apply all changes with the same topic in the checkout to be tested at the same time (similar to how Android's TreeHugger handles topics).
> >
> >
> > Tested by:
> >
> > * Running the new unit test with `python gclient_scm_test.py GerritChangesTest.testDownloadsTopics` from the `tests/` directory.
> >
> > * Running an end-to-end test with `DEPOT_TOOLS_UPDATE=0 gclient sync --patch-ref "skia@d831da5b8ac2d257c5b0cf2ec6645a148f05e662:refs/changes/17/505217/2" --download-topics` in a skia checkout.
> >
> >
> > Bug: chromium:1298922
> > Change-Id: Ieace5e27fbc9c5d0ea90a037bf80a95062c1b164
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3444003
> > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > Commit-Queue: Ravi Mistry <rmistry@chromium.org>
>
> Bug: chromium:1298922
> Change-Id: I80747d797234bba06c17ef5c5e85b310281922c4
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3484976
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Ravi Mistry <rmistry@chromium.org>
Bug: chromium:1298922
Change-Id: I21d7251bafff808b1144d6e522fa9f384f4541bf
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3490488
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Ravi Mistry <rmistry@chromium.org>
This reverts commit fc9a40e3c6.
Reason for revert: at least one tryjob if failing:
Gerrit Plugins Tester
Original change's description:
> Reland "Add support for Gerrit topics in gclient syncs"
>
> This is a reland of 0f13273f1f
>
> Hopefully the cause of the original revert was fixed in https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3480835
>
> Original change's description:
> > Add support for Gerrit topics in gclient syncs
> >
> >
> > If the new flag "--download-topics" is specified with a "--patch-ref" then:
> > * Finds the topic of the Gerrit change.
> > * Finds all open changes in the same repo as the Gerrit change.
> > * Cherrypicks all changes locally.
> >
> > This functionality can be used by developers and bots to apply all changes with the same topic in the checkout to be tested at the same time (similar to how Android's TreeHugger handles topics).
> >
> >
> > Tested by:
> >
> > * Running the new unit test with `python gclient_scm_test.py GerritChangesTest.testDownloadsTopics` from the `tests/` directory.
> >
> > * Running an end-to-end test with `DEPOT_TOOLS_UPDATE=0 gclient sync --patch-ref "skia@d831da5b8ac2d257c5b0cf2ec6645a148f05e662:refs/changes/17/505217/2" --download-topics` in a skia checkout.
> >
> >
> > Bug: chromium:1298922
> > Change-Id: Ieace5e27fbc9c5d0ea90a037bf80a95062c1b164
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3444003
> > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > Commit-Queue: Ravi Mistry <rmistry@chromium.org>
>
> Bug: chromium:1298922
> Change-Id: I80747d797234bba06c17ef5c5e85b310281922c4
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3484976
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Ravi Mistry <rmistry@chromium.org>
Bug: chromium:1298922
Change-Id: I845321941157ab55d026488b7ce59787ba5e57f2
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3488245
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This is a reland of 0f13273f1f
Hopefully the cause of the original revert was fixed in https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3480835
Original change's description:
> Add support for Gerrit topics in gclient syncs
>
>
> If the new flag "--download-topics" is specified with a "--patch-ref" then:
> * Finds the topic of the Gerrit change.
> * Finds all open changes in the same repo as the Gerrit change.
> * Cherrypicks all changes locally.
>
> This functionality can be used by developers and bots to apply all changes with the same topic in the checkout to be tested at the same time (similar to how Android's TreeHugger handles topics).
>
>
> Tested by:
>
> * Running the new unit test with `python gclient_scm_test.py GerritChangesTest.testDownloadsTopics` from the `tests/` directory.
>
> * Running an end-to-end test with `DEPOT_TOOLS_UPDATE=0 gclient sync --patch-ref "skia@d831da5b8ac2d257c5b0cf2ec6645a148f05e662:refs/changes/17/505217/2" --download-topics` in a skia checkout.
>
>
> Bug: chromium:1298922
> Change-Id: Ieace5e27fbc9c5d0ea90a037bf80a95062c1b164
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3444003
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Ravi Mistry <rmistry@chromium.org>
Bug: chromium:1298922
Change-Id: I80747d797234bba06c17ef5c5e85b310281922c4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3484976
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Ravi Mistry <rmistry@chromium.org>
This reverts commit 0f13273f1f.
Reason for revert: breaks codesearch recipes and autorollers
crbug.com/1298961
Original change's description:
> Add support for Gerrit topics in gclient syncs
>
>
> If the new flag "--download-topics" is specified with a "--patch-ref" then:
> * Finds the topic of the Gerrit change.
> * Finds all open changes in the same repo as the Gerrit change.
> * Cherrypicks all changes locally.
>
> This functionality can be used by developers and bots to apply all changes with the same topic in the checkout to be tested at the same time (similar to how Android's TreeHugger handles topics).
>
>
> Tested by:
>
> * Running the new unit test with `python gclient_scm_test.py GerritChangesTest.testDownloadsTopics` from the `tests/` directory.
>
> * Running an end-to-end test with `DEPOT_TOOLS_UPDATE=0 gclient sync --patch-ref "skia@d831da5b8ac2d257c5b0cf2ec6645a148f05e662:refs/changes/17/505217/2" --download-topics` in a skia checkout.
>
>
> Bug: chromium:1298922
> Change-Id: Ieace5e27fbc9c5d0ea90a037bf80a95062c1b164
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3444003
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Ravi Mistry <rmistry@chromium.org>
Bug: chromium:1298922
Bug: chromium:1298961
Change-Id: I88c56cd68372bad09b612de7de1a45f9a0c6c681
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3474793
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
If the new flag "--download-topics" is specified with a "--patch-ref" then:
* Finds the topic of the Gerrit change.
* Finds all open changes in the same repo as the Gerrit change.
* Cherrypicks all changes locally.
This functionality can be used by developers and bots to apply all changes with the same topic in the checkout to be tested at the same time (similar to how Android's TreeHugger handles topics).
Tested by:
* Running the new unit test with `python gclient_scm_test.py GerritChangesTest.testDownloadsTopics` from the `tests/` directory.
* Running an end-to-end test with `DEPOT_TOOLS_UPDATE=0 gclient sync --patch-ref "skia@d831da5b8ac2d257c5b0cf2ec6645a148f05e662:refs/changes/17/505217/2" --download-topics` in a skia checkout.
Bug: chromium:1298922
Change-Id: Ieace5e27fbc9c5d0ea90a037bf80a95062c1b164
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3444003
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Ravi Mistry <rmistry@chromium.org>
