主題: Spark 本機安裝 with Ansible

[sakana]

圖文並茂版本在 http://sakananote2.blogspot.tw/2016/06/spark-with-ansible.html

Spark 本機安裝 with Ansible
現在的實驗環境會透過 Ansible 來管理

 所以整理了一下 spark local 套件安裝的 ansible playbook
使用 group 方式控管
安裝 spark 相關套件, 透過變數詢問下載不同版本, 預設是 1.4.0
使用 when: 方式針對不同的 OS 來安裝, 目前有 openSUSE Leap, CentOS, 還有 ubuntu
目前已測試 openSUSE 42.1 / CentOS 7.2 / Ubuntu 14.04

 檔案 spark_local.yml

代碼: [選擇]

- name: Install spark with local
  hosts: sparkSingle
#  透過 vars_prompt 動態詢問要裝的版本, 預設是 1.4.0
  vars_prompt:
    - name: "spark_version"
      prompt: "Please enter spark version, example 1.4.0"
      private: no
      default: "1.4.0"

   tasks:
    - name: Install java, wget  with openSUSE Leap
# Use sudo before ansible 1.9
#      sudo: True
      become: True
# 使用 pkg 加上 {{ item }} 配合後面的 with_item 來安裝多個套件
      zypper: name={{ item }}
      with_items:
        - java-1.7.0-openjdk-devel
        - wget
      when: ansible_distribution == "openSUSE Leap"

     - name: Install java, wget with CentOS
      become: True
      yum: name={{ item }}
      with_items:
        - java-1.7.0-openjdk-devel
        - wget
      when: ansible_distribution == "CentOS"

     - name: Install java, wget and scala with Ubuntu
      become: True
      apt: name={{ item }} update_cache=yes
      with_items:
        - openjdk-7-jdk
        - wget
        - scala
      when: ansible_distribution == "Ubuntu"

 # ----------------------------------------------------------

     - name: Get scala
      shell: wget http://www.scala-lang.org/files/archive/scala-2.10.1.tgz
      when: ansible_distribution == "CentOS" or ansible_distribution == "openSUSE Leap"

     - name: Exact source code of scala
      shell: tar xvf scala-2.10.1.tgz
      when: ansible_distribution == "CentOS" or ansible_distribution == "openSUSE Leap"

     - name: Move scala to /usr/lib
      become: True
      shell: mv scala-2.10.1 /usr/lib
      when: ansible_distribution == "CentOS" or ansible_distribution == "openSUSE Leap"

     - name: create link for scala
      become: True
      shell: ln -s /usr/lib/scala-2.10.1/  /usr/lib/scala
      when: ansible_distribution == "CentOS" or ansible_distribution == "openSUSE Leap"

     - name: touch .bashrc
      shell: touch ~/.bashrc
      when: ansible_distribution == "CentOS" or ansible_distribution == "openSUSE Leap"

     - name: export $PATH to .bashrc
      lineinfile: dest=~/.bashrc line="export PATH=$PATH:/usr/lib/scala/bin"
      when: ansible_distribution == "CentOS" or ansible_distribution == "openSUSE Leap"

 # ----------------------------------------------------------


# 由於沒有在 general option 使用 sudo, 這邊在使用 shell module 抓下來就是使用者自己的身份額非 root
# 這邊用 spark 1.4.0 版本
    - name: Download spark with pre-build hadoop
        shell: wget    http://archive.apache.org/dist/spark/spark-{{ spark_version }}/spark-{{ spark_version }}-bin-hadoop2.6.tgz

 # 解開 source code
    - name: Exact pre-build source
      shell: tar zxvf spark-*.tgz

     - name: Rename spark folder
      shell: mv ~/spark*/ ~/spark


# 測試 jps 指令
    - name: Testing jps command
      command: jps
      register: jps
    - debug: var=jps.stdout_lines
 執行方式

 將要安裝 spark 的主機加入 sparkSingle 群組
然後執行 spark_local.yml 即可

 先記下來

 ~ enjoy it