[Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR

[Yann E. MORIN]

Thomas, All,

On 2022-11-03 23:45 +0100, Thomas Petazzoni via buildroot spake thusly:
> CARGO_HOME is where Cargo stores its downloaded artefacts. See
> https://doc.rust-lang.org/cargo/reference/environment-variables.html:
> 
>   CARGO_HOME — Cargo maintains a local cache of the registry index and
>   of git checkouts of crates. By default these are stored under
>   $HOME/.cargo (%USERPROFILE%\.cargo on Windows), but this variable
>   overrides the location of this directory. Once a crate is cached it
>   is not removed by the clean command. For more details refer to the
>   guide.
> 
> We currently make it point to $(HOST_DIR)/share/cargo, but this has a
> number of drawbacks:
> 
>  (1) It is not shared between Buildroot builds. Each Buildroot build
>      will re-download the crates index, and the crates themselves,
>      unless of course the final vendored tarball is already there.
> 
>  (2) With BR2_PER_PACKAGE_DIRECTORIES=y, it is even worse: CARGO_HOME
>      is not even shared between packages, as $(HOST_DIR)/share/cargo
>      is per package. So each package in the build that needs vendoring
>      of Cargo crates will download the crates index and the crates in
>      its own CARGO_HOME location.
> 
> To solve this, this commit moves CARGO_HOME into $(DL_DIR), so that it
> is shared between builds and packages.

Unfortunately, this causes breakage when there are parallel downloads
(see below for except; full log as attachment).

So, there are two solutions:

 1. revert to a per-package cargo home, losing the benefit of a shared
    cache, but trivial to implement

 2. add locking ourselves in the post-process scripts, so that there are
    never two "cargo vendor" running in parallel, which is slightly more
    involved.

 3. tell upstream that they borked their locking?

Toughts?

Note: doing sequential downloads works, of course...

>>> dust 0.8.1 Downloading
wget --passive-ftp -nd -t 3 -O '/home/ymorin/dev/buildroot/O/master/build/.dust-0.8.1.tar.gz.ybDevZ/output' 'https://github.com/bootandy/dust/archive/v0.8.1/dust-0.8.1.tar.gz'
[--SNIP--]
2023-01-14 14:30:15 (1.42 MB/s) - ‘/home/ymorin/dev/buildroot/O/master/build/.dust-0.8.1.tar.gz.ybDevZ/output’ saved [89023/89023]

    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
 Downloading crates ...
error: failed to sync

Caused by:
  failed to download packages

Caused by:
  failed to download `autocfg v1.1.0`

Caused by:
  unable to get packages from source

Caused by:
  failed to unpack package `autocfg v1.1.0`

Caused by:
  failed to unpack entry at `autocfg-1.1.0/src/tests.rs`

Caused by:
  No such file or directory (os error 2) while canonicalizing /home/ymorin/dev/buildroot/O/master/yem-dl/br-cargo-home/registry/src/github.com-1ecc6299db9ec823/autocfg-1.1.0/src
make[1]: *** [package/pkg-generic.mk:189: /home/ymorin/dev/buildroot/O/master/build/dust-0.8.1/.stamp_downloaded] Error 101
make[1]: *** Waiting for unfinished jobs....

>>> ripgrep af6b6c543b224d348a8876f0c06245d9ea7929c5 Downloading
wget --passive-ftp -nd -t 3 -O '/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/output' 'https://github.com/burntsushi/ripgrep/archive/af6b6c543b224d348a8876f0c06245d9ea7929c5/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz'
[--SNIP--]
2023-01-14 14:30:16 (1.69 MB/s) - ‘/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/output’ saved [506803/506803]

warning: Both `/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5/.cargo/config` and `/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5/.cargo/config.toml` exist. Using `/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5/.cargo/config`
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
error: failed to sync

Caused by:
  failed to download packages

Caused by:
  failed to download `aho-corasick v0.7.18`

Caused by:
  unable to get packages from source

Caused by:
  failed to parse manifest at `/home/ymorin/dev/buildroot/O/master/yem-dl/br-cargo-home/registry/src/github.com-1ecc6299db9ec823/aho-corasick-0.7.18/Cargo.toml`

Caused by:
  can't find library `aho_corasick`, rename file to `src/lib.rs` or specify lib.path
make[1]: *** [package/pkg-generic.mk:189: /home/ymorin/dev/buildroot/O/master/build/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5/.stamp_downloaded] Error 101

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'