InTheForest/chromium_depot_tools
1
0
Fork 0
mirror of https://chromium.googlesource.com/chromium/tools/depot_tools.git synced 2026-01-11 18:51:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
9,819 Commits 6 Branches 0 Tags
d9011c559b51f3edf9878196b595222def18f309
Commit Graph

147 Commits

Author SHA1 Message Date
Joanna Wang
d9011c559b Set --git-dir for git commands that may be executed in bare gits. 
Bug:b/294415576
Change-Id: I18ca8ebebf95e1c31e30aa1f5d62da3467df940f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4778199
Auto-Submit: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
 2023-08-15 12:30:06 +00:00
Aravind Vasudevan
6eccb0e24c Reset git_cache mirror when HEAD points to master 
master->main branch migration left the cache in some builders to have its HEAD still pointing to refs/heads/master. This causes bot_update to fail. If in this state, delete the cache and force bootstrap.

Bug: 1418866
Change-Id: Id716a013833d459ee37306d27b369099faf846ab
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4310522
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
 2023-03-06 17:28:15 +00:00
Joanna Wang
5175d18f8a Remove branch option from git cache update-bootstrap. 
in update_bootstrap, it's only used for `git number`.
without it git number defaults to `HEAD`
and
nothing seems to actually set the `branch` option
https://source.chromium.org/search?q=%22update-bootstrap%22

Should fix the following failure: http://shortn/_nLLz2nJAGS

Change-Id: I5f3eb8497133c1e5d2f964af969ca61ab6256330
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4080542
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
 2022-12-07 17:27:57 +00:00
Alexander Thomas
ca3ebf119e Revert "Reset origin/HEAD in git_cache if it points to master" 
This reverts commit 63fea808b0.

Reason for revert: broke bot_update for dart, b/259380822

Original change's description:
> Reset origin/HEAD in git_cache if it points to master
>
> This change resets refs/remotes/origin/HEAD if it points to master. With the master to main migration going on with chromium repositories, some of the cached git repositories end up pointing to master instead of main. This change aids in fixing this issue.
>
> Bug: 1364030
> Change-Id: Ic181b0179daf7b2b842cff7f7187d94d9fb20007
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4029356
> Auto-Submit: Aravind Vasudevan <aravindvasudev@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>

Bug: 1364030
Change-Id: I96842c15eba886f5b95cdd10f323b907d5ea165d
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4030304
Reviewed-by: Gavin Mak <gavinmak@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
 2022-11-16 19:33:42 +00:00
Aravind Vasudevan
63fea808b0 Reset origin/HEAD in git_cache if it points to master 
This change resets refs/remotes/origin/HEAD if it points to master. With the master to main migration going on with chromium repositories, some of the cached git repositories end up pointing to master instead of main. This change aids in fixing this issue.

Bug: 1364030
Change-Id: Ic181b0179daf7b2b842cff7f7187d94d9fb20007
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4029356
Auto-Submit: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
 2022-11-16 00:29:09 +00:00
Josip Sokcevic
d540d8b0cd [git-cache] Fix remove files recursively 
os.remove can't be used to remove non empty directory. Use rmtree from
gclient_utils.

R=jojwang@google.com

Bug: 1372658
Change-Id: I5c08bad9778396cb9b1ec0bebe6287a0f3823fab
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3950469
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
 2022-10-12 18:43:49 +00:00
Joanna Wang
5b5ee2de1b [git-cache] Fix print log. 
Bug: 1372658
Change-Id: I12ab46c8f32b64cf575bff087ed2f82072ebbf8f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3949430
Auto-Submit: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
 2022-10-12 17:18:22 +00:00
Joanna Wang
17cf81d90a [git-cache] Fix path splitting. 
Bug:1372658
Change-Id: I83627ecc8cee754a1830a9c4453c59bd9c01d728
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3946647
Auto-Submit: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
 2022-10-12 03:41:24 +00:00
Joanna Wang
38d1673525 Prevent unexpected directories from being uploaded to google storage. 
Bug: 1372658
Change-Id: Ifc19420ee703fcdd0ff8c3d53ddcfeabc499d605
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3938927
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
 2022-10-10 17:12:47 +00:00
Joanna Wang
b9d987ecdb Add print gsutil ls output for git_cache 
Bug: 1370443
Change-Id: I113169a0fdfb3ea87f5f27ba066cce93b3ee6c91
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3938922
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
 2022-10-07 00:20:21 +00:00
Joanna Wang
2c54a19bd0 git_cache: print gsutil command. 
Bug: 1370443
Change-Id: If2cf31982b13a1f6ec9bb9fd6dccb9095e209d63
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3933833
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
 2022-10-05 01:01:40 +00:00
Jack Neus
1601b3eba0 git_cache: wrap "git remote show" call in retries 
BUG=b:236709670
TEST=none

Change-Id: I575dd82260acdbeef3c4c7607e2c2ac8020a44c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3749247
Commit-Queue: Jack Neus <jackneus@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
 2022-07-08 21:18:26 +00:00
Josip Sokcevic
a4b3602457 Use remote information about default branch 
When initializing git_cache repository, we can't assume the default
branch is main. This results in HEAD pointing to invalid ref for repos
that don't have main branch.

Instead, fetch information from remote git repository.

R=gavinmak@google.com

Fixed: 1334889
Change-Id: I84e6d8e09c81bdae7e5e31af88973534431824e3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3696906
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
 2022-06-09 19:59:33 +00:00
Josip Sokcevic
4de5deacd4 Explicitly run everything with python3 
R=aravindvasudev@google.com, gavinmak@google.com

Change-Id: Iaa5e960c6226dea3a0814c7cb778d952eafb1502
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3525960
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
 2022-03-23 21:15:14 +00:00
Aravind Vasudevan
c5f0cbb865 Use pylint 2.7 for depot_tools 
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.

Bug:1262286
Change-Id: I4b8f8fc521386419a3121bbb07edc8ac83170a94
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413679
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
 2022-01-24 23:56:57 +00:00
Josip Sokcevic
42c5bbbc96 Revert "Use pylint 2.7 for depot_tools" 
This reverts commit 22bf605bb6.

Reason for revert: breaks gclient sync

Original change's description:
> Use pylint 2.7 for depot_tools
>
> This includes a few fixes for specific errors, and disables several new
> warnings introduced in this version, in order to allow for an incremental migration.
>
> Bug:1262286
> Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>

Bug: 1262286
Change-Id: Ieb946073c7886c7bf056ce843a5a48e82becf7a5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413672
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
 2022-01-24 21:42:28 +00:00
Aravind Vasudevan
22bf605bb6 Use pylint 2.7 for depot_tools 
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.

Bug:1262286
Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
 2022-01-24 21:11:19 +00:00
Josip Sokcevic
35061445ba Improve error message when a commit is not found 
Improve error message when gclient sync errors due to a bad commit.

R=aravindvasudev@google.com

Fixed: 1078864
Change-Id: I574c7f7d2b5419cf96a4b3a3e3c57e16baeeecf0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3362175
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
 2022-01-12 00:32:54 +00:00
Josip Sokcevic
650f853ced Disable print for git for-each-ref in git_cache 
The command is used for verification purposes when git index is fetched
from GS. The output is verbose for chromium/src and not useful at all.

R=gavinmak@google.com

Bug: 1255228
Change-Id: Ie95963a23ef397e9fd6ce7bcf959ec9964d21ba7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3227145
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
 2021-10-15 18:35:31 +00:00
Josip Sokcevic
604f160239 Reland "Limit number of concurrent gsutil cp calls" 
This is a reland of d3cfea0c78

Original change's description:
> Limit number of concurrent gsutil cp calls
>
> gsutil utilizes as many cores as possible, and will create many threads
> too. In combination with gclient doing the same thing (utilize as many
> cores as possible), this results in creating way too many processes and
> some builders are running out of resources (e.g. can't create threads).
>
> This limits number of gsutil cp commands in git_cache to two.
>
> R=apolito@google.com, dpranke@google.com
>
> Bug: 1255228, 1249003
> Change-Id: Ifcccaacde6c9e1c60c1da0a0bc5a3e512e158dcc
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3219900
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>

Bug: 1255228, 1249003
Change-Id: I712bf5f625fd42992e7b5310ae005fad2633e1a4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3223155
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
 2021-10-15 15:45:10 +00:00
Josip Sokcevic
13225b70bc Force set HEAD to refs/heads/main for git_cache 
We only build git_cache if main branch exists, and newly boostrap caches
have main as default. This sets HEAD for already boostrapped caches, in
case they were boostrapped before the default branch change.

R=gavinmak@google.com

Bug: 1251783
Change-Id: I715329ee06cf523f9fe2e6a324381dc5703bac34
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3218281
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
 2021-10-14 22:24:00 +00:00
Josip Sokcevic
9fe4fb87a6 Revert "Limit number of concurrent gsutil cp calls" 
This reverts commit d3cfea0c78.

Reason for revert: using self.GSUTIL_CP_SEMAPHORE instead of
global GSUTIL_CP_SEMAPHORE

Original change's description:
> Limit number of concurrent gsutil cp calls
>
> gsutil utilizes as many cores as possible, and will create many threads
> too. In combination with gclient doing the same thing (utilize as many
> cores as possible), this results in creating way too many processes and
> some builders are running out of resources (e.g. can't create threads).
>
> This limits number of gsutil cp commands in git_cache to two.
>
> R=​apolito@google.com, dpranke@google.com
>
> Bug: 1255228, 1249003
> Change-Id: Ifcccaacde6c9e1c60c1da0a0bc5a3e512e158dcc
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3219900
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>

Bug: 1255228, 1249003
Change-Id: I0fde2477ea620c683b49efaa2a70262bd8e615e2
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3221900
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
 2021-10-13 23:54:26 +00:00
Josip Sokcevic
d3cfea0c78 Limit number of concurrent gsutil cp calls 
gsutil utilizes as many cores as possible, and will create many threads
too. In combination with gclient doing the same thing (utilize as many
cores as possible), this results in creating way too many processes and
some builders are running out of resources (e.g. can't create threads).

This limits number of gsutil cp commands in git_cache to two.

R=apolito@google.com, dpranke@google.com

Bug: 1255228, 1249003
Change-Id: Ifcccaacde6c9e1c60c1da0a0bc5a3e512e158dcc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3219900
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
 2021-10-13 20:28:05 +00:00
Josip Sokcevic
a40c1e1666 Make git_cache compatible with git <2.28 
R=apolito@google.com

Bug: 1240982
Change-Id: Ibc13c58f35e4d08c5bd402137f08ac1b2e7ae106
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3104288
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
 2021-08-18 20:38:32 +00:00
Josip Sokcevic
6f896d0bfc Set main as default branch for git_cache 
Currently, git cache depends on the global default branch. Set it to
main explicitly instead.

Bug: 3082645, b/195975120
Change-Id: Ia9331d479dbe9547b6913c74bbf86852fff3a596
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3083251
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Anthony Polito <apolito@google.com>
 2021-08-09 23:29:37 +00:00
Josip Sokcevic
ee1a2c72b7 Bootstrap git_cache using main as default branch 
Bug: 1238084
Change-Id: I805f1e7b294c5317adeb7206a021d173e045e8f7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3082645
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Anthony Polito <apolito@google.com>
 2021-08-09 21:09:08 +00:00
Edward Lesmes
cb04744f32 Reland "git-cache: Start collecting metrics." 
This is a reland of a84eaf515f

Original change's description:
> git-cache: Start collecting metrics.
>
> Change-Id: I36f4b37a78ad23d615e3f49f158e8de785d5214a
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2875855
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

Change-Id: I80091d78deb0b1aaba280572a1384848990d9d93
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2877262
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
 2021-05-06 20:18:49 +00:00
Edward Lesmes
5009fd68ac Revert "git-cache: Start collecting metrics." 
This reverts commit a84eaf515f.

Reason for revert:
Should not print notice.
https://logs.chromium.org/logs/infra-internal/buildbucket/cr-buildbucket.appspot.com/8848034039500209104/+/u/bot_update/stdout

Original change's description:
> git-cache: Start collecting metrics.
>
> Change-Id: I36f4b37a78ad23d615e3f49f158e8de785d5214a
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2875855
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

Change-Id: I5d73cfccbcfc75ec26c7ba7127c1320002224e47
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2876069
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
 2021-05-05 22:55:09 +00:00
Edward Lesmes
a84eaf515f git-cache: Start collecting metrics. 
Change-Id: I36f4b37a78ad23d615e3f49f158e8de785d5214a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2875855
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
 2021-05-05 21:54:36 +00:00
Edward Lesmes
07a683430c Reland "git-cache: Add option to fetch commits." 
This is a reland of 4c67f856f0
Issues should have been fixed by crrev.com/c/2838026

Original change's description:
> git-cache: Add option to fetch commits.
>
> Add option to git cache to fetch commits.
> And use it in bot_update and gclient sync to make sure
> the needed commits are present on the checkout.
>
> Change-Id: I9e90da9e3be6e7bacf714b22bf0b735463e655b6
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2829942
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

Change-Id: Ie5a29737f5a75d28bc7c5c2f6cb99ec7f87cd9e8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2841046
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
 2021-04-20 23:39:30 +00:00
Edward Lesmes
93d80d8686 Revert "git-cache: Add option to fetch commits." 
This reverts commit 4c67f856f0.

Reason for revert: crbug.com/1200657

Original change's description:
> git-cache: Add option to fetch commits.
>
> Add option to git cache to fetch commits.
> And use it in bot_update and gclient sync to make sure
> the needed commits are present on the checkout.
>
> Change-Id: I9e90da9e3be6e7bacf714b22bf0b735463e655b6
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2829942
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

Change-Id: I31a1defb26119bcb46894ab4084d626b65c45fed
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2837413
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
 2021-04-20 02:55:11 +00:00
Edward Lesmes
4c67f856f0 git-cache: Add option to fetch commits. 
Add option to git cache to fetch commits.
And use it in bot_update and gclient sync to make sure
the needed commits are present on the checkout.

Change-Id: I9e90da9e3be6e7bacf714b22bf0b735463e655b6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2829942
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
 2021-04-19 22:09:10 +00:00
Josip Sokcevic
67e12286df Add validation after downloading cache from GS 
R=ehmaldonado@chromium.org, gavinmak@google.com

Bug: 1159123
Change-Id: I21aedd9e0c557f75d8c11e055013cc4a540ea20c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2595787
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
 2020-12-16 17:12:45 +00:00
Gavin Mak
e6a623392d [depot_tools] Use logging.warning 
logging.warn is deprecated and should be replaced with logging.warning.

Bug:1097402
Change-Id: I55ce007bec47608eeafcf3cd9ac2b90f60073115
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2575621
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
 2020-12-04 21:57:10 +00:00
Anthony Polito
90b4c0f76b add support for main in git_cache 
Change-Id: I32dcd812b41518bd8e2ca50eae1862611ff577de
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2528797
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Anthony Polito <apolito@google.com>
 2020-11-17 18:09:21 +00:00
Andrii Shyshkalov
46b91c04d1 git-cache: pack refs to reduce # of files downloaded. 
I've observed today that V8 builder spends ~1 minute to download
a bunch of lose files representing tags[1]. This CL should
shrink ~2K files to just 72.

[1]
https://logs.chromium.org/logs/v8/buildbucket/cr-buildbucket.appspot.com/8865281064556718960/+/steps/initialization/0/steps/bot_update/0/stdout

R=jochen@chromium.org, machenbach@chromium.org

Change-Id: Ib8f245e6ed67ea91b53b22a8e4f729ab899a1cbc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2502476
Auto-Submit: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
 2020-10-27 17:25:47 +00:00
Michael Moss
7748094578 Try to be more efficient with git-cache gargage collecting. 
BUG=1092640,1027990
R=iannucci@google.com, tandrii@google.com

Change-Id: I5d14925dc2b6d7a6fd93b6db19cfb98e1ea8df25
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2257740
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Michael Moss <mmoss@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
 2020-06-22 18:32:37 +00:00
Andrii Shyshkalov
03a0d5b08c git-cache: delete support for internal repos. 
It worked only because of an all-powerful service account on each bot,
which isn't the case any more.

R=sokcevic@google.com

Change-Id: Id66f9cc44cd416e5b61745c0f690d5a91b3b4c67
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2225976
Auto-Submit: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
 2020-06-02 04:44:51 +00:00
Josip Sokcevic
14a83aec56 Reland "Use OS level locking in git_cache.py" 
This is a reland of d3affaa624

Original change's description:
> Use OS level locking in git_cache.py
> 
> Without OS level locking it's possible to leave "lock" files on disk
> which will prevent next run to acquire those locks. This can easily
> happen if SIGKIL is issued.
> 
> R=apolito@google.com, ehmaldonado@chromium.org
> 
> Bug: 1049610
> Change-Id: Id87aa1376b9ea5ff0c2d14f3603636493ed1dd5b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2189333
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>

Bug: 1049610
Change-Id: I58e65a10f7c779e0de1121ba7167c694996e390c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2211189
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
 2020-05-21 01:36:34 +00:00
Josip Sokcevic
8f6bfe301f Revert "Use OS level locking in git_cache.py" 
This reverts commit d3affaa624.

Reason for revert: no attribute ignore_lock

Original change's description:
> Use OS level locking in git_cache.py
> 
> Without OS level locking it's possible to leave "lock" files on disk
> which will prevent next run to acquire those locks. This can easily
> happen if SIGKIL is issued.
> 
> R=​apolito@google.com, ehmaldonado@chromium.org
> 
> Bug: 1049610
> Change-Id: Id87aa1376b9ea5ff0c2d14f3603636493ed1dd5b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2189333
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>

TBR=iannucci@chromium.org,ehmaldonado@chromium.org,apolito@google.com,infra-scoped@luci-project-accounts.iam.gserviceaccount.com,sokcevic@google.com

Change-Id: Iecc963e0a99d7f59f3f8801e529839346f9fbaf3
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 1049610
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2211186
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
 2020-05-20 23:23:16 +00:00
Josip Sokcevic
d3affaa624 Use OS level locking in git_cache.py 
Without OS level locking it's possible to leave "lock" files on disk
which will prevent next run to acquire those locks. This can easily
happen if SIGKIL is issued.

R=apolito@google.com, ehmaldonado@chromium.org

Bug: 1049610
Change-Id: Id87aa1376b9ea5ff0c2d14f3603636493ed1dd5b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2189333
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
 2020-05-20 20:42:50 +00:00
Josip Sokcevic
6afaa6ca97 Prune branches that no longer exist on remote 
If a local branch tracks removed remote tracking branch (e.g. foo) and
remote has a branch that starts with such name (e.g. foo/bar), git fetch
will fail. --prune flag removes any remote-tracking branches that no
longer exist.

R=apolito@google.com, ehmaldonado@chromium.org

Bug: 1079483
Change-Id: I9bc31bf961d52a86b6fa2342249971b99a003666
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2190341
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
 2020-05-08 18:20:17 +00:00
Edward Lesmes
34f71abca1 git-cache: Also bootstrap in the case that there are 0 pack files 
This can happen if the cache repo was init'd, but has no pack files;
previously it would try to fetch from scratch.

Change-Id: I71689e30bdede392588c69e118e9297d86a134a3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2120281
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
 2020-03-25 21:24:00 +00:00
Edward Lesmes
4c3eb70358 git-cache: Decode subprocess.check_output 
Always decode subprocess.check_output.

Bug: 1064547
Change-Id: I459a236235a2355f9dd9813dafc12dff44866529
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2120966
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
 2020-03-25 21:09:30 +00:00
Edward Lesmes
cc11b065c5 Revert "[bot_update] Also bootstrap in the case that there are 0 pack files" 
This reverts commit 32a801b5da.

Reason for revert:
It's causing depot tools presubmit to time out
(Also, introduces a Python 3 error for git_cache).

Original change's description:
> [bot_update] Also bootstrap in the case that there are 0 pack files
> 
> This can happen if the cache repo was init'd, but has no pack files;
> previously it would try to fetch from scratch.
> 
> R=​ehmaldonado@chromium.org, tandrii@chromium.org
> 
> No-Presubmit: true
> Change-Id: I97e863dd6c2ecfd00fdb4238dda48e0d3929c4f2
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1941337
> Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@google.com>

TBR=iannucci@chromium.org,tandrii@google.com,ehmaldonado@chromium.org

Change-Id: I93366bcc6ff1df579a67d51c2cda812c84a06215
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1951985
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
 2019-12-04 22:37:14 +00:00
Robert Iannucci
32a801b5da [bot_update] Also bootstrap in the case that there are 0 pack files 
This can happen if the cache repo was init'd, but has no pack files;
previously it would try to fetch from scratch.

R=ehmaldonado@chromium.org, tandrii@chromium.org

No-Presubmit: true
Change-Id: I97e863dd6c2ecfd00fdb4238dda48e0d3929c4f2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1941337
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
 2019-12-04 22:05:12 +00:00
Andrii Shyshkalov
c50b0963e3 git_cache: don't lose options in upload_bootstrap. 
I noticed that chromium/src archive grew 2x from 7.3GiB in August
to 15+ GiB now. Turns out --aggressive-gc had no effect.

R=iannucci

Change-Id: I6a27f67f5a4dd2101ac1622221106e094af4688a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1928583
Auto-Submit: Andrii Shyshkalov <tandrii@google.com>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
 2019-11-21 23:03:18 +00:00
Edward Lemur
e9024d00d2 gclient: Make smoke tests use local dir instead of git daemon. 
Running gclient_smoketests times out on windows when using git daemon,
so use a local directory as remote instead.

Bug: 1024683
Change-Id: I6ca506d74de463d914317f176eefbe74996298c5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1879723
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
 2019-11-19 18:47:46 +00:00
John Budorick
3da78c4bb7 Revert git cache diagnostic tracing. 
This reverts commit ebba8d75bf.
This reverts commit fcde3ba0a6.
This reverts commit fd843fa456.

Bug: 1019824
Change-Id: I14ad2d4f620ac316421f1c3d168b66176af2d26e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1917496
Auto-Submit: John Budorick <jbudorick@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
 2019-11-14 20:06:30 +00:00
danakj
c41f72cf44 Allow the bot_update api to not fetch tags from the git server. 
There are 3 layers modified, starting from the bottom up:

1. git_cache.py populate

Now takes a --no-fetch-tags option. If specified, the cache will not
fetch updated tags from the server by passing --no-tags to git fetch.
This prevents the server from sending all the tag refs. In chromium
this prevents significant time bottlenecks dealing with 10k+ tags.

2. bot_update.py

bot_update has to deal with multiple git repos, it has the root repo
that is checked out through git-cache, as well as repos synched via
DEPS and gclient.

The script now takes a --no_fetch_tags option. If specified,
the git-cache populate call will include --no-fetch-tags. Otherwise, it
won't. So this controls (for chromium) if fetches to the src.git server
are passed --no-tags.

3. bot_update/api.py

This is the entry point for recipes and also has to deal with multiple
git repos. The behaviour at this point is not changed from the default.
A |no_fetch_tags| parameter is added to ensure_checkout() that defaults
to False.


This CL is a refactor with no intended behavior change.

The next step will be to change the chromium build repo to pass True
for ensure_checkout(no_fetch_tags) on chromium trybots.

This represents solution #2 in
https://docs.google.com/document/d/1hDIunJjjfpmr50y3YnZ4o3aq1NZF4gJa1TS0p7AIL64/edit#

Bug: 1019824
Change-Id: I935430603299daa9e301a95a5184c0ce486fd912
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1894352
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Erik Chen <erikchen@chromium.org>
Commit-Queue: danakj <danakj@chromium.org>
Auto-Submit: danakj <danakj@chromium.org>
 2019-11-05 17:12:01 +00:00