Deploying Hortonworks Sandbox on Docker on MAC M1 - Installation error arm64

Question

I am trying to setup "Deploying Hortonworks Sandbox on Docker" on MAC M1

Official Cloudera Installation directory

https://www.cloudera.com/tutorials/sandbox-deployment-and-install-guide/3.html

Reference installation

https://javamana.com/2021/02/20210206191356735v.html

below is the configuration

Ram - 16gb

Rosetta2 Installed

Docker installed Mac M1

Docker hub image link

https://hub.docker.com/r/hortonworks/sandbox-hdp/tags

command used

docker pull hortonworks/sandbox-hdp:3.0.1 --platform linux/amd64

SCRIPT

#!/usr/bin/env sh
#This script downloads HDP sandbox along with their proxy docker container
set -x

# CAN EDIT THESE VALUES
registry="hortonworks"
name="sandbox-hdp"
version="3.0.1"
proxyName="sandbox-proxy"
proxyVersion="1.0"
flavor="hdp"

# NO EDITS BEYOND THIS LINE
# housekeeping
echo $flavor > sandbox-flavor


# create necessary folders for nginx and copy over our rule generation script there
mkdir -p sandbox/proxy/conf.d
mkdir -p sandbox/proxy/conf.stream.d

# pull and tag the sandbox and the proxy container
docker pull "$registry/$name:$version"
docker pull "$registry/$proxyName:$proxyVersion"


# start the docker container and proxy
if [ "$flavor" == "hdf" ]; then
 hostname="sandbox-hdf.hortonworks.com"
elif [ "$flavor" == "hdp" ]; then
 hostname="sandbox-hdp.hortonworks.com"
fi

version=$(docker images | grep $registry/$name  | awk '{print $2}');

# Create cda docker network
docker network create cda 2>/dev/null

# Deploy the sandbox into the cda docker network
docker run --privileged --name $name -h $hostname --network=cda --network-alias=$hostname -d "$registry/$name:$version"

echo " Remove existing postgres run files. Please wait"
sleep 2
docker exec -t "$name" sh -c "rm -rf /var/run/postgresql/*; systemctl restart postgresql-9.6.service;"


#Deploy the proxy container.
sed 's/sandbox-hdp-security/sandbox-hdp/g' assets/generate-proxy-deploy-script.sh > assets/generate-proxy-deploy-script.sh.new
mv -f assets/generate-proxy-deploy-script.sh.new assets/generate-proxy-deploy-script.sh
chmod +x assets/generate-proxy-deploy-script.sh
assets/generate-proxy-deploy-script.sh 2>/dev/null

#check to see if it's windows
if uname | grep MINGW; then 
 sed -i -e 's/\( \/[a-z]\)/\U\1:/g' sandbox/proxy/proxy-deploy.sh
fi
chmod +x sandbox/proxy/proxy-deploy.sh 2>/dev/null
sandbox/proxy/proxy-deploy.sh 

LOG

➜  Documents sh docker-deploy-hdp30.sh
+ registry=hortonworks
+ name=sandbox-hdp
+ version=3.0.1
+ proxyName=sandbox-proxy
+ proxyVersion=1.0
+ flavor=hdp
+ echo hdp
+ mkdir -p sandbox/proxy/conf.d
+ mkdir -p sandbox/proxy/conf.stream.d
+ docker pull hortonworks/sandbox-hdp:3.0.1
3.0.1: Pulling from hortonworks/sandbox-hdp
70799bbf2226: Pull complete
40963917cdad: Pull complete
3fe9adbb8d7e: Pull complete
ee3ec4e8cb3d: Pull complete
7ea5917732c0: Pull complete
2d951411620c: Pull complete
f4c5e354e7fd: Pull complete
22ffa6ef360f: Pull complete
2060aa0f3751: Pull complete
ca01ba34744d: Pull complete
83326dded077: Pull complete
eb3d71b90b73: Pull complete
bdd1cab41c81: Pull complete
500cc770c4bd: Pull complete
0cb1decd5474: Pull complete
b9591f4b6855: Pull complete
f28e56086127: Pull complete
e7de4e7d0bca: Pull complete
ec77967d2166: Pull complete
4fdcae170114: Pull complete
6347f5df8ffc: Pull complete
6a6ecc232709: Pull complete
ea845898ff50: Pull complete
02135573b1bf: Pull complete
cb0176867cd8: Pull complete
3c08321268fd: Pull complete
82e82a97c465: Pull complete
8aaaa48ed101: Pull complete
74b321ac2ac5: Pull complete
569da02c0a66: Pull complete
af40820407ef: Pull complete
Digest: sha256:7b767af7b42030fb1dd0f672b801199241e6bef1258e3ce57361edb779d95921
Status: Downloaded newer image for hortonworks/sandbox-hdp:3.0.1
docker.io/hortonworks/sandbox-hdp:3.0.1
+ docker pull hortonworks/sandbox-proxy:1.0
1.0: Pulling from hortonworks/sandbox-proxy
951bdea65c93: Pull complete
4b9047c5fbbb: Pull complete
773156407aae: Pull complete
d8524176841d: Pull complete
Digest: sha256:42e4cfbcbb76af07e5d8f47a183a0d4105e65a1e7ef39fe37ab746e8b2523e9e
Status: Downloaded newer image for hortonworks/sandbox-proxy:1.0
docker.io/hortonworks/sandbox-proxy:1.0
+ '[' hdp == hdf ']'
+ '[' hdp == hdp ']'
+ hostname=sandbox-hdp.hortonworks.com
++ docker images
++ grep hortonworks/sandbox-hdp
++ awk '{print $2}'
+ version=3.0.1
+ docker network create cda
ce52e1d04aa49c2da70dfd7405b95fc2e2184fed9625feaea0ddafb9ab5c361d
+ docker run --privileged --name sandbox-hdp -h sandbox-hdp.hortonworks.com --network=cda --network-alias=sandbox-hdp.hortonworks.com -d hortonworks/sandbox-hdp:3.0.1
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
3ab0f0281918fc10d008c93aa7a844302e974b696e00788df76cde779fdec46d
+ echo ' Remove existing postgres run files. Please wait'
 Remove existing postgres run files. Please wait
+ sleep 2
+ docker exec -t sandbox-hdp sh -c 'rm -rf /var/run/postgresql/*; systemctl restart postgresql-9.6.service;'
Failed to get D-Bus connection: No such file or directory
+ sed s/sandbox-hdp-security/sandbox-hdp/g assets/generate-proxy-deploy-script.sh
+ mv -f assets/generate-proxy-deploy-script.sh.new assets/generate-proxy-deploy-script.sh
+ chmod +x assets/generate-proxy-deploy-script.sh
+ assets/generate-proxy-deploy-script.sh
+ uname
+ grep MINGW
+ chmod +x sandbox/proxy/proxy-deploy.sh
+ sandbox/proxy/proxy-deploy.sh
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
fbe0c31a680f51db9c91367df65493ea1cf874cc900466dd943824dc19bd9e74
➜  Documents

how to resolve the error

I have tried performing some tweaks

adding the below line --platform linux/amd64

still i encounter the same error

what is the work around to fix this issue ?

Tweaks i have tried

before running the script i have tried adjusting the script by adding

# CAN EDIT THESE VALUES
registry="hortonworks"
name="sandbox-hdp"
version="3.0.1"
proxyName="sandbox-proxy"
proxyVersion="1.0"
flavor="hdp"
arch = arm64
platform = linux/arm64

Deploy the sandbox into the cda docker network

docker run --privileged --name $name -h $hostname --network=cda --network-alias=$hostname -d "$registry/$name:$version" $platform

option 1

arch = linux/arm64

option 2

platform = linux/arm64

Answer 1

First validate if Rosetta2 is configured correctly. You can test with alpine - very small linux docker container. Run:

docker run --rm -ti --platform linux/amd64 alpine:latest uname -a

The output should be something like this:

Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
Digest: sha256:d6d0a0eb4d40ef96f2310ead734848b9c819bb97c9d846385c4aca1767186cd4
Status: Downloaded newer image for alpine:latest
Linux 35d1145cff16 5.11.0-49-generic #55-Ubuntu SMP Wed Jan 12 17:36:34 UTC 2022 x86_64 Linux

Now if that works you can modify scripts to run HDP. There are actually 2 scripts to modify.

First

Open docker-deploy-hdp30.sh in a editor and modify following line:

docker run --privileged --name $name -h $hostname --network=cda --network-alias=$hostname -d "$registry/$name:$version"

To:

docker run --platform linux/amd64 --privileged --name $name -h $hostname --network=cda --network-alias=$hostname -d "$registry/$name:$version"

Second

Open assets/generate-proxy-deploy-script.sh in an editor & scroll to the bottom. There's docker run in line 204, add platform switch after it like this (the backslashes at the end of line are crucial!):

docker run --name sandbox-proxy --network=cda \\
--platform linux/amd64 \\
-v $absPath/assets/nginx.conf:/etc/nginx/nginx.conf \\

Now you are ready to run the first script (it triggers the other one). After it executes docker ps should return 2 containers: sandbox-proxy and sandbox-hdp.