The reason `test_vfio_user` fails is as @likebreath pointed: our ARM host does not support SVE, while the nvme_tgt binary built from the container image requires it. As a result, we encountered a SIGILL when running the nvme_tgt binary. This also explains why this is not happening when the container is built on the same host itself. And quote from @rbradford: When a job is run on one of the workers it looks to see if there is a container locally matching the name as specified in the dev_cli.sh script - if there is then it uses it. Otherwise it will try and download it from the container registry - if that fails then it will built locally. For the x86-64 workers started dynamically it will never have a local version as they are a fresh VM. But on the ARM64 builder is a local container image cache. This can lead to an issue where if the image is build with one version (a handcrafted datestamp) and then the Dockerfile is changed without changing the timestamp then an old version may be fetched from the cache or server. It is there for essential to always bump the datestamp (there is a number after the - that can be used for this.) However there is also the added complexity that image that is build and uploaded to the container registry is not the same as the built locally and thus used for the initial testing of the Dockerfile change. This leads to the issue we have seen where different CPU compiler flags (from -march=native) from the QEMU cross build in the hosted GHA action and the local ARM64 build. Resulting in a binary in the remotely built container not working locally. We end up specifying TARGET_ARCHITECTURE="armv8.2-a" for building spdk, and put built `python/spdk/` folder into `/usr/local/bin/spdk-nvme`. Signed-off-by: Ruoqing He <heruoqing@iscas.ac.cn>
155 lines
5.7 KiB
Docker
155 lines
5.7 KiB
Docker
# Copyright © 2025 Intel Corporation
|
|
#
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
#
|
|
# When changing this file don't forget to update the tag name in the
|
|
# .github/workflows/docker-image.yaml file if doing multiple per day
|
|
|
|
FROM ubuntu:24.04 AS dev
|
|
|
|
ARG TARGETARCH
|
|
ARG RUST_TOOLCHAIN="1.77.0"
|
|
ARG CLH_SRC_DIR="/cloud-hypervisor"
|
|
ARG CLH_BUILD_DIR="$CLH_SRC_DIR/build"
|
|
ARG CARGO_REGISTRY_DIR="$CLH_BUILD_DIR/cargo_registry"
|
|
ARG CARGO_GIT_REGISTRY_DIR="$CLH_BUILD_DIR/cargo_git_registry"
|
|
|
|
ENV CARGO_HOME=/usr/local/rust
|
|
ENV RUSTUP_HOME=$CARGO_HOME
|
|
ENV PATH="$PATH:$CARGO_HOME/bin"
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
# Install all CI dependencies
|
|
# DL3015 ignored cause not installing openvswitch-switch-dpdk recommended packages breaks ovs_dpdk test
|
|
# hadolint ignore=DL3008,DL3015
|
|
RUN apt-get update \
|
|
&& apt-get -yq upgrade \
|
|
&& apt-get install --no-install-recommends -yq \
|
|
build-essential \
|
|
bc \
|
|
curl \
|
|
wget \
|
|
sudo \
|
|
mtools \
|
|
musl-tools \
|
|
libssl-dev \
|
|
pkg-config \
|
|
flex \
|
|
bison \
|
|
libelf-dev \
|
|
qemu-utils \
|
|
libglib2.0-dev \
|
|
libpixman-1-dev \
|
|
libseccomp-dev \
|
|
libcap-ng-dev \
|
|
socat \
|
|
dosfstools \
|
|
cpio \
|
|
python3 \
|
|
python3-setuptools \
|
|
ntfs-3g \
|
|
uuid-dev \
|
|
iperf3 \
|
|
zip \
|
|
git-core \
|
|
dnsmasq \
|
|
dmsetup \
|
|
ca-certificates \
|
|
unzip \
|
|
iproute2 \
|
|
dbus \
|
|
&& apt-get install openvswitch-switch-dpdk -yq \
|
|
&& apt-get clean \
|
|
&& rm -rf /var/lib/apt/lists/* /var/log/*log /var/log/apt/* /var/lib/dpkg/*-old /var/cache/debconf/*-old
|
|
|
|
RUN update-alternatives --set ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk
|
|
|
|
# hadolint ignore=DL3008
|
|
RUN if [ "$TARGETARCH" = "amd64" ]; then \
|
|
apt-get update \
|
|
&& apt-get -yq upgrade \
|
|
&& apt-get install --no-install-recommends -yq swtpm \
|
|
&& apt-get clean \
|
|
&& rm -rf /var/lib/apt/lists/* /var/log/*log /var/log/apt/* /var/lib/dpkg/*-old /var/cache/debconf/*-old; fi
|
|
|
|
# hadolint ignore=DL3008
|
|
RUN if [ "$TARGETARCH" = "arm64" ]; then \
|
|
# On AArch64, `setcap` binary should be installed via `libcap2-bin`.
|
|
# The `setcap` binary is used in integration tests.
|
|
# `libguestfs-tools` is used for modifying cloud image kernel, and it requires
|
|
# kernel (any version) image in `/boot` and modules in `/lib/modules`.
|
|
apt-get update \
|
|
&& apt-get -yq upgrade \
|
|
&& apt-get install --no-install-recommends -yq \
|
|
libcap2-bin \
|
|
libguestfs-tools \
|
|
linux-image-generic \
|
|
autotools-dev \
|
|
autoconf \
|
|
automake \
|
|
perl \
|
|
texinfo \
|
|
&& apt-get clean \
|
|
&& rm -rf /var/lib/apt/lists/* /var/log/*log /var/log/apt/* /var/lib/dpkg/*-old /var/cache/debconf/*-old; fi
|
|
|
|
# Fix the libssl-dev install
|
|
# hadolint ignore=SC2155
|
|
RUN export ARCH="$(uname -m)" \
|
|
&& cp /usr/include/"$ARCH"-linux-gnu/openssl/opensslconf.h /usr/include/openssl/
|
|
ENV X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR=/usr/lib/x86_64-linux-gnu/
|
|
ENV X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_LIB_DIR=/usr/lib/x86_64-linux-gnu/
|
|
ENV AARCH64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR=/usr/lib/aarch64-linux-gnu/
|
|
ENV AARCH64_UNKNOWN_LINUX_MUSL_OPENSSL_LIB_DIR=/usr/lib/aarch64-linux-gnu/
|
|
ENV OPENSSL_INCLUDE_DIR=/usr/include/
|
|
|
|
# Install the rust toolchain
|
|
# hadolint ignore=DL4006,SC2155
|
|
RUN export ARCH="$(uname -m)" \
|
|
&& nohup curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain "$RUST_TOOLCHAIN" \
|
|
&& rustup target add "$ARCH"-unknown-linux-musl --toolchain "$RUST_TOOLCHAIN" \
|
|
&& if [ "$TARGETARCH" = "amd64" ]; then rustup toolchain add --profile minimal $RUST_TOOLCHAIN-x86_64-unknown-linux-musl; fi \
|
|
&& if [ "$TARGETARCH" = "amd64" ]; then rustup component add rustfmt; fi \
|
|
&& if [ "$TARGETARCH" = "amd64" ]; then rustup component add clippy; fi \
|
|
&& rm -rf "$CARGO_HOME/registry" \
|
|
&& ln -s "$CARGO_REGISTRY_DIR" "$CARGO_HOME/registry" \
|
|
&& rm -rf "$CARGO_HOME/git" \
|
|
&& ln -s "$CARGO_GIT_REGISTRY_DIR" "$CARGO_HOME/git"
|
|
|
|
# Set the rust environment
|
|
# hadolint ignore=SC2016
|
|
RUN echo 'source $CARGO_HOME/env' >> "$HOME"/.bashrc \
|
|
&& mkdir "$HOME"/.cargo \
|
|
&& ln -s $CARGO_HOME/env "$HOME"/.cargo/env
|
|
|
|
# Allow pip to install packages system wide
|
|
# hadolint ignore=DL3003,SC2046
|
|
RUN rm /usr/lib/python3.12/EXTERNALLY-MANAGED \
|
|
&& git clone https://github.com/spdk/spdk \
|
|
&& cd spdk \
|
|
&& git checkout ef8bcce58f3f02b79c0619a297e4f17e81e62b24 \
|
|
&& git submodule update --init \
|
|
&& apt-get update \
|
|
&& ./scripts/pkgdep.sh \
|
|
&& apt-get clean \
|
|
&& ./configure --with-vfio-user \
|
|
&& if [ "$TARGETARCH" = "amd64" ]; then \
|
|
make -j $(nproc) TARGET_ARCHITECTURE=skylake; \
|
|
elif [ "$TARGETARCH" = "arm64" ]; then \
|
|
make -j $(nproc) TARGET_ARCHITECTURE="armv8.2-a" DPDKBUILD_FLAGS="-Dplatform=generic"; \
|
|
else \
|
|
echo "Unsupported architecture: ${TARGETARCH}" && exit 1; \
|
|
fi \
|
|
&& rm -rf /usr/local/bin/spdk-nvme \
|
|
&& mkdir /usr/local/bin/spdk-nvme \
|
|
&& cp -f ./build/bin/nvmf_tgt /usr/local/bin/spdk-nvme \
|
|
&& cp -f ./scripts/rpc.py /usr/local/bin/spdk-nvme \
|
|
&& cp -rf ./python/spdk/ /usr/local/bin/spdk-nvme \
|
|
&& cp -rf ./python /usr/local/bin \
|
|
&& cd .. && rm -rf spdk
|
|
|
|
# install ethr tool for performance tests
|
|
RUN if [ "$TARGETARCH" = "amd64" ]; then \
|
|
wget -nv https://github.com/microsoft/ethr/releases/latest/download/ethr_linux.zip \
|
|
&& unzip ethr_linux.zip \
|
|
&& cp ethr /usr/local/bin \
|
|
&& rm ethr_linux.zip; fi
|