Skip to content

Setup Cloud9 for EKS

快速设置 cloud9 用于日常测试环境搭建,包含从 cloudshell 中创建 cloud9 instance,然后登录 cloud9 instance 进行基础软件安装、磁盘大小调整和容器环境相关软件安装。为了更方便配置,在 quick-setup-cloud9 中,直接可以仅通过 cloudshell 即完成所有初始化动作,登录 cloud9 instance 后就可以开始使用。

spin-up-a-cloud9-instance-in-your-region

  • 点击这里 运行 cloudshell,执行代码块创建 cloud9 测试环境 (open cloudshell, and then execute following code to create cloud9 environment)

    # name=<give your cloud9 a name>
    datestring=$(date +%Y%m%d-%H%M)
    echo ${name:=cloud9-$datestring}
    
    # VPC_ID=<your vpc id> 
    # ensure you have public subnet in it
    DEFAULT_VPC_ID=$(aws ec2 describe-vpcs \
      --filter Name=is-default,Values=true \
      --query 'Vpcs[0].VpcId' --output text \
      --region ${AWS_DEFAULT_REGION})
    VPC_ID=${VPC_ID:=$DEFAULT_VPC_ID}
    
    if [[ ! -z ${VPC_ID} ]]; then
      FIRST_SUBNET=$(aws ec2 describe-subnets \
        --filters "Name=vpc-id,Values=${VPC_ID}" \
        --query 'Subnets[?(AvailabilityZone==`'"${AWS_DEFAULT_REGION}a"'` && MapPublicIpOnLaunch==`true`)].SubnetId' \
        --output text \
        --region ${AWS_DEFAULT_REGION})
      aws cloud9 create-environment-ec2 \
        --name ${name} \
        --image-id amazonlinux-2-x86_64 \
        --instance-type m5.large \
        --subnet-id ${FIRST_SUBNET%% *} \
        --automatic-stop-time-minutes 10080 \
        --region ${AWS_DEFAULT_REGION} |tee /tmp/$$
      echo "Open URL to access your Cloud9 Environment:"
      C9_ID=$(cat /tmp/$$ |jq -r '.environmentId')
      echo "https://${AWS_DEFAULT_REGION}.console.aws.amazon.com/cloud9/ide/${C9_ID}"
    else
      echo "you have no default vpc in $AWS_DEFAULT_REGION"
    fi
    

  • 点击输出的 URL 链接,打开 cloud9 测试环境 (click the URL at the bottom to open cloud9 environment)
    IMG-setup-cloud9-for-eks.png

using internal proxy or not

  • 如果你不需要使用代理服务器下载软件包,跳过执行下面代码 (skip this code block if you do not need proxy in your environment)
    cat >> ~/.bash_profile <<-EOF
    export http_proxy=http://10.101.1.55:998
    export https_proxy=http://10.101.1.55:998
    export NO_PROXY=169.254.169.254,10.0.0.0/8,172.16.0.0/16,192.168.0.0/16
    EOF
    source ~/.bash_profile
    

install-in-cloud9-

  • 下面代码块包含一些基本设置,包括:(execute this code block to install tools for your lab, and resize ebs of cloud9)
    • 安装更新常用的软件
    • 修改 cloud9 磁盘大小 (link)
script-prep-eks-env-part-one.sh
#!/bin/bash
###-SCRIPT-PART-ONE-BEGIN-###
echo "###"
echo "SCRIPT-PART-ONE-BEGIN"
echo "###"
# set size as your expectation, otherwize 100g as default volume size
# size=200

# default execute this script in EC2, not Cloud9
echo ${EXECUTE_IN_CLOUD9:=false}

# install others
sudo yum -y install jq gettext bash-completion wget argon2 # moreutils

# install terraform 
sudo yum install -y yum-utils shadow-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
sudo yum -y install terraform

# install code-server
IDE_PASSWORD=$(echo -n $(aws sts get-caller-identity --query "Account" --output text) | argon2 $(uuidgen) -e)
mkdir -p ~/.config/code-server
tee ~/.config/code-server/config.yaml <<-EOF
cert: false
auth: password
hashed-password: "$IDE_PASSWORD"
bind-addr: 0.0.0.0:8088
EOF
mkdir -p ~/.local/share/code-server/User
tee ~/.local/share/code-server/User/settings.json <<EOF
{
"extensions.autoUpdate": false,
"extensions.autoCheckUpdates": false,
"terminal.integrated.cwd": "/home/$USER",
"telemetry.telemetryLevel": "off",
"security.workspace.trust.startupPrompt": "never",
"security.workspace.trust.enabled": false,
"security.workspace.trust.banner": "never",
"security.workspace.trust.emptyWindow": false,
"editor.indentSize": "tabSize",
"editor.tabSize": 2,
"python.testing.pytestEnabled": true,
"auto-run-command.rules": [
    {
    "command": "workbench.action.terminal.new"
    }
]
}
EOF

CODE_SERVER_VER=4.93.1
wget -qO /tmp/code-server.rpm https://github.com/coder/code-server/releases/download/v${CODE_SERVER_VER}/code-server-${CODE_SERVER_VER}-amd64.rpm
sudo yum install -y /tmp/code-server.rpm
sudo systemctl enable --now code-server@ec2-user
sudo systemctl restart code-server@ec2-user

# install awscli v2
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o /tmp/awscliv2.zip
echo A |unzip /tmp/awscliv2.zip -d /tmp
sudo /tmp/aws/install --update 2>&1 >/tmp/awscli-install.log
echo "complete -C '/usr/local/bin/aws_completer' aws" >> ~/.bash_profile

# remove existed aws
if [[ $? -eq 0 ]]; then
  sudo yum remove -y awscli
  source ~/.bash_profile
  aws --version
fi

# install awscli v1
# curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
# unzip awscli-bundle.zip
# sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

# install ssm session plugin
curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/linux_64bit/session-manager-plugin.rpm" -o "/tmp/session-manager-plugin.rpm"
sudo yum install -y /tmp/session-manager-plugin.rpm

# your default region 
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
export AWS_DEFAULT_REGION=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.region')

# change root volume size for cloud9
if [[ ${EXECUTE_IN_CLOUD9} == "true" ]]; then
if [[ -c /dev/nvme0 ]]; then
  wget -qO- https://github.com/amazonlinux/amazon-ec2-utils/raw/main/ebsnvme-id >/tmp/ebsnvme-id
  VOLUME_ID=$(sudo python3 /tmp/ebsnvme-id -v /dev/nvme0 |awk '{print $NF}')
  DEVICE_NAME=/dev/nvme0n1
else
  C9_INST_ID=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id)
  VOLUME_ID=$(aws ec2 describe-volumes --filters Name=attachment.instance-id,Values=${C9_INST_ID} --query "Volumes[0].VolumeId" --output text)
  DEVICE_NAME=/dev/xvda
fi

aws ec2 modify-volume --volume-id ${VOLUME_ID} --size ${size:-100}
sleep 10
sudo growpart ${DEVICE_NAME} 1
sudo xfs_growfs -d /

if [[ $? -eq 1 ]]; then
  ROOT_PART=$(df |grep -w / |awk '{print $1}')
  sudo resize2fs ${ROOT_PART}
fi
fi

echo "###"
echo "SCRIPT-PART-ONE-END"
echo "###"
###-SCRIPT-PART-ONE-END-###
script-ubuntu-prep-eks-env-part-one.sh
#!/bin/bash
(###-SCRIPT-PART-ONE-BEGIN-###
echo "###"
echo "SCRIPT-PART-ONE-BEGIN"
echo "###"
# set size as your expectation, otherwize 100g as default volume size
# size=200

# default execute this script in EC2, not Cloud9
echo ${EXECUTE_IN_CLOUD9:=false}

# install others
export DEBIAN_FRONTEND=noninteractive
sudo -E apt update
sudo -E apt-get -yq install jq gettext bash-completion wget argon2 moreutils awscli

# install terraform 
sudo rm -f /usr/share/keyrings/hashicorp-archive-keyring.gpg
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo -E apt update
sudo -E apt-get -yq install terraform=1.5.7-1
sudo apt-mark hold terraform

# install code-server
IDE_PASSWORD=$(echo -n $(aws sts get-caller-identity --query "Account" --output text) | argon2 $(uuidgen) -e)
mkdir -p ~/.config/code-server
tee ~/.config/code-server/config.yaml <<-EOF
cert: false
auth: password
hashed-password: "${IDE_PASSWORD}"
bind-addr: 0.0.0.0:8088
EOF
mkdir -p ~/.local/share/code-server/User
tee ~/.local/share/code-server/User/settings.json <<EOF
{
"extensions.autoUpdate": false,
"extensions.autoCheckUpdates": false,
"terminal.integrated.cwd": "/home/$USER",
"telemetry.telemetryLevel": "off",
"security.workspace.trust.startupPrompt": "never",
"security.workspace.trust.enabled": false,
"security.workspace.trust.banner": "never",
"security.workspace.trust.emptyWindow": false,
"editor.indentSize": "tabSize",
"editor.tabSize": 2,
"python.testing.pytestEnabled": true,
"auto-run-command.rules": [
    {
    "command": "workbench.action.terminal.new"
    }
]
}
EOF

CODE_SERVER_VER=4.93.1
wget -qO /tmp/code-server.deb https://github.com/coder/code-server/releases/download/v${CODE_SERVER_VER}/code-server_${CODE_SERVER_VER}_amd64.deb
sudo dpkg -i /tmp/code-server.deb
sudo systemctl enable --now code-server@ubuntu
sudo systemctl restart code-server@ubuntu

# install awscli v2
# sudo -E apt-get -yq install awscli
echo "complete -C '/usr/bin/aws_completer' aws" >> ~/.bash_profile

# install ssm session plugin
curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/ubuntu_64bit/session-manager-plugin.deb" -o "/tmp/session-manager-plugin.deb"
sudo dpkg -i /tmp/session-manager-plugin.deb

# your default region 
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
export AWS_DEFAULT_REGION=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.region')

# change root volume size in cloud9
if [[ ${EXECUTE_IN_CLOUD9} == "true" ]]; then
if [[ -c /dev/nvme0 ]]; then
  wget -qO- https://github.com/amazonlinux/amazon-ec2-utils/raw/main/ebsnvme-id >/tmp/ebsnvme-id
  VOLUME_ID=$(sudo python3 /tmp/ebsnvme-id -v /dev/nvme0 |awk '{print $NF}')
  DEVICE_NAME=/dev/nvme0n1
else
  C9_INST_ID=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id)
  VOLUME_ID=$(aws ec2 describe-volumes --filters Name=attachment.instance-id,Values=${C9_INST_ID} --query "Volumes[0].VolumeId" --output text)
  DEVICE_NAME=/dev/xvda
fi

aws ec2 modify-volume --volume-id ${VOLUME_ID} --size ${size:-100}
sleep 10
sudo growpart ${DEVICE_NAME} 1
sudo xfs_growfs -d /

if [[ $? -eq 1 ]]; then
  ROOT_PART=$(df |grep -w / |awk '{print $1}')
  sudo resize2fs ${ROOT_PART}
fi
fi

echo "###"
echo "SCRIPT-PART-ONE-END"
echo "###"
###-SCRIPT-PART-ONE-END-###
)
  • 安装 eks 相关的常用软件 (install some eks related tools)
  • for AL2 & ubuntu: script-prep-eks-env-part-two.sh

    script-prep-eks-env-part-two.sh
    #!/bin/bash
    ###-SCRIPT-PART-TWO-BEGIN-###
    echo "###"
    echo "SCRIPT-PART-TWO-BEGIN"
    echo "###"
    
    mv -f ~/.bash_completion ~/.bash_completion.$(date +%N)
    # install kubectl with +/- 1 cluster version 1.28.14 / 1.29.9 / 1.30.4 / 1.31.1
    # refer: https://kubernetes.io/releases/
    # sudo curl --location -o /usr/local/bin/kubectl "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
    sudo curl --silent --location -o /usr/local/bin/kubectl "https://storage.googleapis.com/kubernetes-release/release/v1.30.4/bin/linux/amd64/kubectl"
    sudo chmod +x /usr/local/bin/kubectl
    
    /usr/local/bin/kubectl completion bash >>  ~/.bash_completion
    # source /etc/profile.d/bash_completion.sh
    # source ~/.bash_completion
    alias k=kubectl 
    complete -F __start_kubectl k
    echo "alias k=kubectl" >> ~/.bashrc
    echo "complete -F __start_kubectl k" >> ~/.bashrc
    
    # install eksctl
    # consider install eksctl version 0.89.0
    # if you have older version yaml 
    # https://eksctl.io/announcements/nodegroup-override-announcement/
    curl -L "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp/
    sudo mv -v /tmp/eksctl /usr/local/bin
    /usr/local/bin/eksctl completion bash >> ~/.bash_completion
    # source /etc/profile.d/bash_completion.sh
    # source ~/.bash_completion
    
    # install kubectx
    curl -L "https://github.com/ahmetb/kubectx/releases/download/v0.9.5/kubectx_v0.9.5_linux_x86_64.tar.gz" |tar xz -C /tmp/
    sudo mv -f /tmp/kubectx /usr/local/bin/
    # install kubens
    curl -L "https://github.com/ahmetb/kubectx/releases/download/v0.9.5/kubens_v0.9.5_linux_x86_64.tar.gz" |tar xz -C /tmp/
    sudo mv -f /tmp/kubens /usr/local/bin/
    
    # install k9s
    curl -L "https://github.com/derailed/k9s/releases/latest/download/k9s_Linux_amd64.tar.gz" |tar xz -C /tmp/
    sudo mv -f /tmp/k9s /usr/local/bin/
    
    # install eksdemo
    curl -L "https://github.com/awslabs/eksdemo/releases/latest/download/eksdemo_$(uname -s)_$(uname -p).tar.gz" |tar xz -C /tmp/
    sudo mv -v /tmp/eksdemo /usr/local/bin
    /usr/local/bin/eksdemo completion bash >> ~/.bash_completion
    # source /etc/profile.d/bash_completion.sh
    # source ~/.bash_completion
    
    # helm newest version (3.15.4)
    curl -sSL https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
    /usr/local/bin/helm version --short
    # helm 3.8.2 (helm 3.9.0 will have issue #10975)
    # wget https://get.helm.sh/helm-v3.8.2-linux-amd64.tar.gz
    # tar xf helm-v3.8.2-linux-amd64.tar.gz
    # sudo mv linux-amd64/helm /usr/local/bin/helm
    
    # install aws-iam-authenticator 0.6.11 (2023/10) 
    wget -O /tmp/aws-iam-authenticator https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v0.6.14/aws-iam-authenticator_0.6.14_linux_amd64
    chmod +x /tmp/aws-iam-authenticator
    sudo mv /tmp/aws-iam-authenticator /usr/local/bin/
    
    # install kube-no-trouble
    sh -c "$(curl -sSL https://git.io/install-kubent)"
    
    # install kubectl convert plugin
    curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl-convert" --output-dir /tmp
    curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl-convert.sha256" --output-dir /tmp
    echo "$(cat /tmp/kubectl-convert.sha256) /tmp/kubectl-convert" | sha256sum --check
    sudo install -o root -g root -m 0755 /tmp/kubectl-convert /usr/local/bin/kubectl-convert
    rm /tmp/kubectl-convert /tmp/kubectl-convert.sha256
    
    # option install jwt-cli
    # https://github.com/mike-engel/jwt-cli/blob/main/README.md
    # sudo yum -y install cargo
    # cargo install jwt-cli
    # sudo ln -sf ~/.cargo/bin/jwt /usr/local/bin/jwt
    
    # install flux & fluxctl
    curl -s https://fluxcd.io/install.sh | sudo -E bash
    /usr/local/bin/flux -v
    source <(/usr/local/bin/flux completion bash)
    
    # sudo wget -O /usr/local/bin/fluxctl $(curl https://api.github.com/repos/fluxcd/flux/releases/latest | jq -r ".assets[] | select(.name | test(\"linux_amd64\")) | .browser_download_url")
    # sudo chmod 755 /usr/local/bin/fluxctl
    # fluxctl version
    # fluxctl identity --k8s-fwd-ns flux
    
    echo "###"
    echo "SCRIPT-PART-TWO-END"
    echo "###"
    ###-SCRIPT-PART-TWO-END-###
    

  • 直接执行下面代码块可能遇到权限不够的告警,需要:

    • 如果你有 workshop 的 Credentials ,直接先复制粘贴到命令行,再执行下列步骤;(copy and paste your workshop’s credential to CLI and then execute this code block)
    • 或者,如果自己账号的 cloud9,先用环境变量方式(AWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEY)保证有足够权限执行 (or using environment variables to export credential yourself)
    • 下面代码块包括:
      • 禁用 cloud9 中的 credential 管理,从 ~/.aws/credentials 中删除 aws_session_token=
      • 分配管理员权限 role 到 cloud9 instance
  • for AL2: script-prep-eks-env-part-three.sh

    script-prep-eks-env-part-three.sh
    #!/bin/bash
    ###-SCRIPT-PART-THREE-BEGIN-###
    echo "###"
    echo "SCRIPT-PART-THREE-BEGIN"
    echo "###"
    
    aws cloud9 update-environment  --environment-id $C9_PID --managed-credentials-action DISABLE
    rm -vf ${HOME}/.aws/credentials
    
    # ---
    export AWS_PAGER=""
    export AWS_DEFAULT_REGION=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.region')
    C9_INST_ID=$(curl 169.254.169.254/latest/meta-data/instance-id)
    ROLE_NAME=adminrole-$(TZ=CST-8 date +%Y%m%d-%H%M%S)
    MY_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
    
    cat > ec2.json <<-EOF
    {
        "Effect": "Allow",
        "Principal": {
            "Service": "ec2.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }
    EOF
    STATEMENT_LIST=ec2.json
    
    for i in WSParticipantRole WSOpsRole TeamRole OpsRole ; do
      aws iam get-role --role-name $i >/dev/null 2>&1
      if [[ $? -eq 0 ]]; then
        envsubst >$i.json <<-EOF
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::${MY_ACCOUNT_ID}:role/$i"
      },
      "Action": "sts:AssumeRole"
    }
    EOF
        STATEMENT_LIST=$(echo ${STATEMENT_LIST} "$i.json")
      fi
    done
    
    jq -n '{Version: "2012-10-17", Statement: [inputs]}' ${STATEMENT_LIST} > trust.json
    echo ${STATEMENT_LIST}
    rm -f ${STATEMENT_LIST}
    
    # create role
    aws iam create-role --role-name ${ROLE_NAME} \
      --assume-role-policy-document file://trust.json
    aws iam attach-role-policy --role-name ${ROLE_NAME} \
      --policy-arn "arn:aws:iam::aws:policy/AdministratorAccess"
    
    instance_profile_arn=$(aws ec2 describe-iam-instance-profile-associations \
      --filter Name=instance-id,Values=$C9_INST_ID \
      --query IamInstanceProfileAssociations[0].IamInstanceProfile.Arn \
      --output text)
    if [[ ${instance_profile_arn} == "None" ]]; then
      # create one
      aws iam create-instance-profile \
        --instance-profile-name ${ROLE_NAME}
      sleep 10
      # attach role to it
      aws iam add-role-to-instance-profile \
        --instance-profile-name ${ROLE_NAME} \
        --role-name ${ROLE_NAME}
      sleep 10
      # attach instance profile to ec2
      aws ec2 associate-iam-instance-profile \
        --iam-instance-profile Name=${ROLE_NAME} \
        --instance-id ${C9_INST_ID}
    else
      existed_role_name=$(aws iam get-instance-profile \
        --instance-profile-name ${instance_profile_arn##*/} \
        --query 'InstanceProfile.Roles[0].RoleName' \
        --output text)
      aws iam attach-role-policy --role-name ${existed_role_name} \
        --policy-arn "arn:aws:iam::aws:policy/AdministratorAccess"
    fi
    
    echo "###"
    echo "SCRIPT-PART-THREE-END"
    echo "###"
    ###-SCRIPT-PART-THREE-END-###
    

  • 在 cloud9 中,重新打开一个 terminal 窗口,并验证权限符合预期。上面代码块将创建一个 instance profile ,并将关联名为 adminrole-xxx 的 role,或者在 cloud9 现有的 role 上关联 AdministratorAccess role policy。(open new tab to verify you have new role, adminrole-xxx, on your cloud9)

    aws sts get-caller-identity
    

reference