搜索到
211
篇与
cby
的结果
-
搭建Hadoop2.7.2和Hive2.3.3以及Spark3.1.2 Hadoop 简介Hadoop是一个用Java编写的Apache开源框架，允许使用简单的编程模型跨计算机集群分布式处理大型数据集。Hadoop框架工作的应用程序在跨计算机集群提供分布式存储和计算的环境中工作。Hadoop旨在从单个服务器扩展到数千个机器，每个都提供本地计算和存储。Hive简介Apache Hive是一个构建于Hadoop顶层的数据仓库，可以将结构化的数据文件映射为一张数据库表，并提供简单的SQL查询功能，可以将SQL语句转换为MapReduce任务进行运行。需要注意的是，Hive它并不是数据库。Hive依赖于HDFS和MapReduce，其对HDFS的操作类似于SQL，我们称之为HQL，它提供了丰富的SQL查询方式来分析存储在HDFS中的数据。HQL可以编译转为MapReduce作业，完成查询、汇总、分析数据等工作。这样一来，即使不熟悉MapReduce 的用户也可以很方便地利用SQL 语言查询、汇总、分析数据。而MapReduce开发人员可以把己写的mapper 和reducer 作为插件来支持Hive 做更复杂的数据分析。Apache Spark 简介用于大数据工作负载的分布式开源处理系统Apache Spark 是一种用于大数据工作负载的分布式开源处理系统。它使用内存中缓存和优化的查询执行方式，可针对任何规模的数据进行快速分析查询。它提供使用 Java、Scala、Python 和 R 语言的开发 API，支持跨多个工作负载重用代码—批处理、交互式查询、实时分析、机器学习和图形处理等。本文将先搭建 jdk1.8 + MySQL5.7基础环境之后搭建Hadoop2.7.2和Hive2.3.3以及Spark3.1.2此文章搭建为单机版 1.创建目录并解压jdk安装包[root@localhost ~]# mkdir jdk [root@localhost ~]# cd jdk/ [root@localhost jdk]# ls jdk-8u202-linux-x64.tar.gz [root@localhost jdk]# ll total 189496 -rw-r--r--. 1 root root 194042837 Oct 18 12:05 jdk-8u202-linux-x64.tar.gz [root@localhost jdk]# [root@localhost jdk]# tar xvf jdk-8u202-linux-x64.tar.gz2.配置环境变量[root@localhost ~]# vim /etc/profile [root@localhost ~]# tail -n 3 /etc/profile export JAVA_HOME=/root/jdk/jdk1.8.0_202/ export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar [root@localhost ~]# [root@localhost ~]# source /etc/profile3.下载安装MySQL并设置为开机自启[root@localhost ~]# mkdir mysql [root@localhost ~]# cd mysql [root@localhost mysql]# wget https://downloads.mysql.com/archives/get/p/23/file/mysql-5.7.35-1.el7.x86_64.rpm-bundle.tar [root@localhost mysql]# tar xvf mysql-5.7.35-1.el7.x86_64.rpm-bundle.tar [root@localhost mysql]# yum install ./*.rpm [root@localhost mysql]# systemctl start mysqld.service [root@localhost mysql]# [root@localhost mysql]# systemctl enable mysqld.service [root@localhost mysql]# [root@localhost mysql]#4.查看MySQL默认密码，并修改默认密码，同时创建新的用户，将其设置为可以远程登录[root@localhost mysql]# sudo grep 'temporary password' /var/log/mysqld.log 2021-10-18T06:12:35.519726Z 6 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: eNHu<sXHt3rq [root@localhost mysql]# [root@localhost mysql]# [root@localhost mysql]# mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 9 Server version: 8.0.25 Copyright (c) 2000, 2021, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'Cby123..'; Query OK, 0 rows affected (0.02 sec) mysql> mysql> mysql> use mysql; Database changed mysql> mysql> update user set host='%' where user ='root'; Query OK, 1 row affected (0.01 sec) Rows matched: 1 Changed: 1 Warnings: 0 mysql> set global validate_password_policy=0; Query OK, 0 rows affected (0.01 sec) mysql> set global validate_password_mixed_case_count=0; Query OK, 0 rows affected (0.00 sec) mysql> set global validate_password_number_count=3; Query OK, 0 rows affected (0.00 sec) mysql> set global validate_password_special_char_count=0; Query OK, 0 rows affected (0.00 sec) mysql> set global validate_password_length=3; Query OK, 0 rows affected (0.00 sec) mysql> SHOW VARIABLES LIKE 'validate_password%'; +--------------------------------------+-------+ | Variable_name | Value | +--------------------------------------+-------+ | validate_password_check_user_name | OFF | | validate_password_dictionary_file | | | validate_password_length | 3 | | validate_password_mixed_case_count | 0 | | validate_password_number_count | 3 | | validate_password_policy | LOW | | validate_password_special_char_count | 0 | +--------------------------------------+-------+ 7 rows in set (0.00 sec) mysql> create user 'cby'@'%' identified by 'cby'; Query OK, 0 rows affected (0.00 sec) mysql> grant all on *.* to 'cby'@'%'; Query OK, 0 rows affected (0.00 sec) mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%'WITH GRANT OPTION; Query OK, 0 rows affected (0.00 sec) mysql> CREATE DATABASE dss_dev; Query OK, 1 row affected (0.00 sec) mysql> mysql> select host,user,plugin from user; +-----------+---------------+-----------------------+ | host | user | plugin | +-----------+---------------+-----------------------+ | % | root | mysql_native_password | | localhost | mysql.session | mysql_native_password | | localhost | mysql.sys | mysql_native_password | +-----------+---------------+-----------------------+ 3 rows in set (0.01 sec) mysql>注：若上面root不是mysql_native_password使用以下命令将其改掉update user set plugin='mysql_native_password' where user='root'; 5.添加hosts解析，同时设置免密登录[root@localhost ~]# mkdir Hadoop [root@localhost ~]# [root@localhost ~]# vim /etc/hosts [root@localhost ~]# [root@localhost ~]# [root@localhost ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 namenode [root@localhost ~]# ssh-keygen [root@localhost ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@127.0.0.1 [root@localhost ~]#6.下载Hadoop，解压后创建所需目录[root@localhost ~]# cd Hadoop/ [root@localhost Hadoop]# ls [root@localhost Hadoop]# wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.2/hadoop-2.7.2.tar.gz [root@localhost Hadoop]# tar xvf hadoop-2.7.2.tar.gz [root@localhost Hadoop]# [root@localhost Hadoop]# mkdir -p /root/Hadoop/hadoop-2.7.2/hadoopinfra/hdfs/namenode [root@localhost Hadoop]# [root@localhost Hadoop]# [root@localhost Hadoop]# mkdir -p /root/Hadoop/hadoop-2.7.2/hadoopinfra/hdfs/datanode7.添加Hadoop环境变量[root@localhost ~]# vim /etc/profile [root@localhost ~]# tail -n 8 /etc/profile export HADOOP_HOME=/root/Hadoop/hadoop-2.7.2/ export HADOOP_INSTALL=$HADOOP_HOME export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin [root@localhost ~]# [root@localhost ~]# source /etc/profile8.修改Hadoop配置[root@localhost ~]# cd Hadoop/hadoop-2.7.2/ [root@localhost hadoop]# vim /root/Hadoop/hadoop-2.7.2/etc/hadoop/core-site.xml [root@localhost hadoop]# [root@localhost hadoop]# [root@localhost hadoop]# tail /root/Hadoop/hadoop-2.7.2/etc/hadoop/core-site.xml <!-- Put site-specific property overrides in this file. --> <configuration> <!-- 指定HDFS中NameNode的地址 --> <property> <name>fs.defaultFS</name> <value>hdfs://127.0.0.1:9000</value> </property> <!-- 指定Hadoop运行时产生文件的存储目录 --> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/hadoop-2.7.2/data/tmp</value> </property> </configuration>9.修改Hadoop的hdfs目录配置[root@localhost hadoop]# vim /root/Hadoop/hadoop-2.7.2/etc/hadoop/hdfs-site.xml [root@localhost hadoop]# [root@localhost hadoop]# [root@localhost hadoop]# [root@localhost hadoop]# tail -n 15 /root/Hadoop/hadoop-2.7.2/etc/hadoop/hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.name.dir</name> <value>/root/Hadoop/hadoop-2.7.2/hadoopinfra/hdfs/namenode</value> </property> <property> <name>dfs.data.dir</name> <value>/root/Hadoop/hadoop-2.7.2/hadoopinfra/hdfs/datanode</value> </property> </configuration>10.修改Hadoop的yarn配置[root@localhost hadoop]# [root@localhost hadoop]# vim /root/Hadoop/hadoop-2.7.2/etc/hadoop/yarn-site.xml [root@localhost hadoop]# [root@localhost hadoop]# [root@localhost hadoop]# tail -n 6 /root/Hadoop/hadoop-2.7.2/etc/hadoop/yarn-site.xml <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>[root@localhost hadoop]# [root@localhost hadoop]# cp /root/Hadoop/hadoop-2.7.2/etc/hadoop/mapred-site.xml.template /root/Hadoop/hadoop-2.7.2/etc/hadoop/mapred-site.xml [root@localhost hadoop]# vim /root/Hadoop/hadoop-2.7.2/etc/hadoop/mapred-site.xml [root@localhost hadoop]# [root@localhost hadoop]# [root@localhost hadoop]# tail /root/Hadoop/hadoop-2.7.2/etc/hadoop/mapred-site.xml <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> [root@localhost hadoop]#11.修改Hadoop环境配置文件[root@localhost hadoop]# vim /root/Hadoop/hadoop-2.7.2/etc/hadoop/hadoop-env.sh 修改JAVA_HOME export JAVA_HOME=/root/jdk/jdk1.8.0_202/ [root@localhost ~]# hdfs namenode -format [root@localhost ~]# start-dfs.sh [root@localhost ~]# start-yarn.sh若重置太多次会导致clusterID不匹配，datanode起不来，删除版本后在初始化启动[root@localhost ~]# rm -rf /root/Hadoop/hadoop-2.7.2/hadoopinfra/hdfs/datanode/current/VERSION [root@localhost ~]# hadoop namenode -format [root@localhost ~]# hdfs namenode -format [root@localhost ~]# start-dfs.sh [root@localhost ~]# start-yarn.sh在浏览器访问Hadoop访问Hadoop的默认端口号为50070.使用以下网址，以获取浏览器Hadoop服务。http://localhost:50070/验证集群的所有应用程序访问集群中的所有应用程序的默认端口号为8088。使用以下URL访问该服务。http://localhost:8088/12.创建hive目录并解压[root@localhost ~]# mkdir hive [root@localhost ~]# cd hive [root@localhost hive]# wget https://archive.apache.org/dist/hive/hive-2.3.3/apache-hive-2.3.3-bin.tar.gz [root@localhost hive]# tar xvf apache-hive-2.3.3-bin.tar.gz [root@localhost hive]#13.备份hive配置文件[root@localhost hive]# cd /root/hive/apache-hive-2.3.3-bin/conf/ [root@localhost conf]# cp hive-env.sh.template hive-env.sh [root@localhost conf]# cp hive-default.xml.template hive-site.xml [root@localhost conf]# cp hive-log4j2.properties.template hive-log4j2.properties [root@localhost conf]# cp hive-exec-log4j2.properties.template hive-exec-log4j2.properties14.在Hadoop中创建文件夹并设置权限[root@localhost conf]# hadoop fs -mkdir -p /data/hive/warehouse 21/10/18 14:27:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [root@localhost conf]# [root@localhost conf]# hadoop fs -mkdir /data/hive/tmp 21/10/18 14:27:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [root@localhost conf]# [root@localhost conf]# hadoop fs -mkdir /data/hive/log 21/10/18 14:27:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [root@localhost conf]# [root@localhost conf]# hadoop fs -chmod -R 777 /data/hive/warehouse 21/10/18 14:27:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [root@localhost conf]# hadoop fs -chmod -R 777 /data/hive/tmp 21/10/18 14:27:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [root@localhost conf]# hadoop fs -chmod -R 777 /data/hive/log 21/10/18 14:27:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [root@localhost conf]#15.修改hive配置文件[root@localhost conf]# vim hive-site.xml hive 配置入下： <property> <name>hive.exec.scratchdir</name> <value>hdfs://127.0.0.1:9000/data/hive/tmp</value> </property> <property> <name>hive.metastore.warehouse.dir</name> <value>hdfs://127.0.0.1:9000/data/hive/warehouse</value> </property> <property> <name>hive.querylog.location</name> <value>hdfs://127.0.0.1:9000/data/hive/log</value> </property> <!—该配置是关闭hive元数据版本认证，否则会在启动spark程序时报错--> <property> <name>hive.metastore.schema.verification</name> <value>false</value> </property> 配置mysql IP 端口以及放元数据的库名称 <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://127.0.0.1:3306/hive?createDatabaseIfNotExist=true</value> </property> <!—配置mysql启动器名称 --> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <!—配置连接mysql用户名 --> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> </property> <!—配置连接mysql用户名登录密码--> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>Cby123..</value> </property>修改配置文件中 system:java.io.tmpdir 和 system:user.name 相关信息，改为实际目录和用户名，或者加入如下配置 <property> <name>system:java.io.tmpdir</name> <value>/tmp/hive/java</value> </property> <property> <name>system:user.name</name> <value>${user.name}</value> </property>并修改临时路径 ： <property> <name>hive.exec.local.scratchdir</name> <value>/root/hive/apache-hive-2.3.3-bin/tmp/${system:user.name}</value> <description>Local scratch space for Hive jobs</description> </property> <property> <name>hive.downloaded.resources.dir</name> <value>/root/hive/apache-hive-2.3.3-bin/tmp/${hive.session.id}_resources</value> <description>Temporary local directory for added resources in the remote file system.</description> </property> <property> <name>hive.server2.logging.operation.log.location</name> <value>/root/hive/apache-hive-2.3.3-bin/tmp/root/operation_logs</value> <description>Top level directory where operation logs are stored if logging functionality is enabled</description> </property>16.配置hive中jdbc的MySQL驱动[root@localhost lib]# cd /root/hive/apache-hive-2.3.3-bin/lib/ [root@localhost lib]# wget https://downloads.mysql.com/archives/get/p/3/file/mysql-connector-java-5.1.49.tar.gz [root@localhost lib]# tar xvf mysql-connector-java-5.1.49.tar.gz [root@localhost lib]# cp mysql-connector-java-5.1.49/mysql-connector-java-5.1.49.jar . [root@localhost bin]# [root@localhost bin]# vim /root/hive/apache-hive-2.3.3-bin/conf/hive-env.sh [root@localhost bin]# tail -n 3 /root/hive/apache-hive-2.3.3-bin/conf/hive-env.sh export HADOOP_HOME=/root/Hadoop/hadoop-2.7.2/ export HIVE_CONF_DIR=/root/hive/apache-hive-2.3.3-bin/conf export HIVE_AUX_JARS_PATH=/root/hive/apache-hive-2.3.3-bin/lib [root@localhost bin]#17.配置hive环境变量[root@localhost ~]# vim /etc/profile [root@localhost ~]# tail -n 6 /etc/profile export HADOOP_HOME=/root/Hadoop/hadoop-2.7.2/ export HIVE_CONF_DIR=/root/hive/apache-hive-2.3.3-bin/conf export HIVE_AUX_JARS_PATH=/root/hive/apache-hive-2.3.3-bin/lib export HIVE_PATH=/root/hive/apache-hive-2.3.3-bin/ export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$HIVE_PATH/bin [root@localhost bin]# ./schematool -dbType mysql -initSchema初始化完成后修改MySQL链接信息，之后配置mysql IP 端口以及放元数据的库名称[root@localhost conf]# vim hive-site.xml <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://127.0.0.1:3306/hive?characterEncoding=utf8&useSSL=false</value> </property> [root@localhost bin]# nohup hive --service metastore & [root@localhost bin]# nohup hive --service hiveserver2 &**18.创建spark目录并下载所需文件 **[root@localhost ~]# mkdir spark [root@localhost ~]# [root@localhost ~]# cd spark [root@localhost spark]# [root@localhost spark]# wget https://dlcdn.apache.org/spark/spark-3.1.2/spark-3.1.2-bin-without-hadoop.tgz --no-check-certificate [root@localhost spark]# tar xvf spark-3.1.2-bin-without-hadoop.tgz19.配置spark环境变量以及备份配置文件[root@localhost ~]# vim /etc/profile [root@localhost ~]# [root@localhost ~]# tail -n 3 /etc/profile export SPARK_HOME=/root/spark/spark-3.1.2-bin-without-hadoop/ export PATH=$PATH:$SPARK_HOME/bin [root@localhost spark]# cd /root/spark/spark-3.1.2-bin-without-hadoop/conf/ [root@localhost conf]# cp spark-env.sh.template spark-env.sh [root@localhost conf]# cp spark-defaults.conf.template spark-defaults.conf [root@localhost conf]# cp metrics.properties.template metrics.properties20.配置程序的环境变量[root@localhost conf]# cp workers.template workers [root@localhost conf]# vim spark-env.sh export JAVA_HOME=/root/jdk/jdk1.8.0_202 export HADOOP_HOME=/root/Hadoop/hadoop-2.7.2 export HADOOP_CONF_DIR=/root/Hadoop/hadoop-2.7.2/etc/hadoop export SPARK_DIST_CLASSPATH=$(/root/Hadoop/hadoop-2.7.2/bin/hadoop classpath) export SPARK_MASTER_HOST=127.0.0.1 export SPARK_MASTER_PORT=7077 export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 - Dspark.history.retainedApplications=50 - Dspark.history.fs.logDirectory=hdfs://127.0.0.1:9000/spark-eventlog"21.修改默认的配置文件[root@localhost conf]# vim spark-defaults.conf spark.master spark://127.0.0.1:7077 spark.eventLog.enabled true spark.eventLog.dir hdfs://127.0.0.1:9000/spark-eventlog spark.serializer org.apache.spark.serializer.KryoSerializer spark.driver.memory 3g spark.eventLog.enabled true spark.eventLog.dir hdfs://127.0.0.1:9000/spark-eventlog spark.eventLog.compress true22.配置工作节点[root@localhost conf]# vim workers [root@localhost conf]# cat workers 127.0.0.1 [root@localhost conf]# [root@localhost sbin]# /root/spark/spark-3.1.2-bin-without-hadoop/sbin/start-all.sh验证应用程序访问集群中的所有应用程序的默认端口号为8080。使用以下URL访问该服务。http://localhost:8080/Linux运维交流社区Linux运维交流社区，互联网新闻以及技术交流。41篇原创内容公众号本文使用 文章同步助手 同步 -
elk7.15.1安装部署搭建 ELK简介ELK是Elasticsearch、Logstash、Kibana三大开源框架首字母大写简称（但是后期出现的Filebeat（beats中的一种）可以用来替代Logstash的数据收集功能，比较轻量级）。市面上也被成为Elastic Stack。Filebeat是用于转发和集中日志数据的轻量级传送工具。Filebeat监视您指定的日志文件或位置，收集日志事件，并将它们转发到Elasticsearch或 Logstash进行索引。Filebeat的工作方式如下：启动Filebeat时，它将启动一个或多个输入，这些输入将在为日志数据指定的位置中查找。对于Filebeat所找到的每个日志，Filebeat都会启动收集器。每个收集器都读取单个日志以获取新内容，并将新日志数据发送到libbeat，libbeat将聚集事件，并将聚集的数据发送到为Filebeat配置的输出。Logstash是免费且开放的服务器端数据处理管道，能够从多个来源采集数据，转换数据，然后将数据发送到您最喜欢的“存储库”中。Logstash能够动态地采集、转换和传输数据，不受格式或复杂度的影响。利用Grok从非结构化数据中派生出结构，从IP地址解码出地理坐标，匿名化或排除敏感字段，并简化整体处理过程。Elasticsearch是Elastic Stack核心的分布式搜索和分析引擎，是一个基于Lucene、分布式、通过Restful方式进行交互的近实时搜索平台框架。Elasticsearch为所有类型的数据提供近乎实时的搜索和分析。无论您是结构化文本还是非结构化文本，数字数据或地理空间数据，Elasticsearch都能以支持快速搜索的方式有效地对其进行存储和索引。Kibana是一个针对Elasticsearch的开源分析及可视化平台，用来搜索、查看交互存储在Elasticsearch索引中的数据。使用Kibana，可以通过各种图表进行高级数据分析及展示。并且可以为Logstash和ElasticSearch提供的日志分析友好的 Web 界面，可以汇总、分析和搜索重要数据日志。还可以让海量数据更容易理解。它操作简单，基于浏览器的用户界面可以快速创建仪表板（Dashboard）实时显示Elasticsearch查询动态完整日志系统基本特征收集：能够采集多种来源的日志数据传输：能够稳定的把日志数据解析过滤并传输到存储系统存储：存储日志数据分析：支持UI分析警告：能够提供错误报告，监控机制安装jdk17环境root@elk:~# mkdir jdk root@elk:~# cd jdk root@elk:~/jdk# wget https://download.oracle.com/java/17/latest/jdk-17_linux-x64_bin.tar.gz root@elk:~/jdk# tar xf jdk-17_linux-x64_bin.tar.gz root@elk:~/jdk# cd .. root@elk:~# root@elk:~# mv jdk/ / root@elk:~# vim /etc/profile root@elk:~# root@elk:~# root@elk:~# tail -n 4 /etc/profile export JAVA_HOME=/jdk/jdk-17.0.1/ export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar root@elk:~# root@elk:~# source /etc/profile root@elk:~# chmod -R 777 /jdk/创建elk文件夹，并下载所需包root@elk:~# mkdir elk root@elk:~# cd elk root@elk:~/elk# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.15.1-linux-x86_64.tar.gz root@elk:~/elk# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.15.1-linux-x86_64.tar.gz root@elk:~/elk# wget https://artifacts.elastic.co/downloads/logstash/logstash-7.15.1-linux-x86_64.tar.gz解压安装包root@elk:~/elk# tar xf elasticsearch-7.15.1-linux-x86_64.tar.gz root@elk:~/elk# tar xf kibana-7.15.1-linux-x86_64.tar.gz root@elk:~/elk# tar xf logstash-7.15.1-linux-x86_64.tar.gz root@elk:~/elk# ll total 970288 drwxr-xr-x 5 root root 4096 Oct 20 06:09 ./ drwx------ 7 root root 4096 Oct 20 06:04 ../ drwxr-xr-x 9 root root 4096 Oct 7 22:00 elasticsearch-7.15.1/ -rw-r--r-- 1 root root 340849929 Oct 14 13:28 elasticsearch-7.15.1-linux-x86_64.tar.gz drwxr-xr-x 10 root root 4096 Oct 20 06:09 kibana-7.15.1-linux-x86_64/ -rw-r--r-- 1 root root 283752241 Oct 14 13:34 kibana-7.15.1-linux-x86_64.tar.gz drwxr-xr-x 13 root root 4096 Oct 20 06:09 logstash-7.15.1/ -rw-r--r-- 1 root root 368944379 Oct 14 13:38 logstash-7.15.1-linux-x86_64.tar.gz创建用户并设置权限root@elk:~/elk# cd root@elk:~# useradd elk root@elk:~# mkdir /home/elk root@elk:~# cp -r elk/ /home/elk/ root@elk:~# chown -R elk:elk /home/elk/修改系统配置文件root@elk:~# vim /etc/security/limits.conf root@elk:~# root@elk:~# root@elk:~# tail -n 3 /etc/security/limits.conf * soft nofile 65536 * hard nofile 65536 root@elk:~# root@elk:~# vim /etc/sysctl.conf root@elk:~# root@elk:~# tail -n 2 /etc/sysctl.conf vm.max_map_count=262144 root@elk:~# root@elk:~# sysctl -p vm.max_map_count = 262144 root@elk:~#修改elk配置文件root@elk:~# su - elk $ bash elk@elk:~$ cd /elk/elasticsearch-7.15.1/config elk@elk:~/elk/elasticsearch-7.15.1/config$ vim elasticsearch.yml elk@elk:~/elk/elasticsearch-7.15.1/config$ elk@elk:~/elk/elasticsearch-7.15.1/config$ tail -n 20 elasticsearch.yml #设置data存放的路径为/data/es-data path.data: /home/elk/data/ #设置logs日志的路径为/log/es-log path.logs: /home/elk/data/ #设置内存不使用交换分区 bootstrap.memory_lock: false #配置了bootstrap.memory_lock为true时反而会引发9200不会被监听，原因不明 #设置允许所有ip可以连接该elasticsearch network.host: 0.0.0.0 #开启监听的端口为9200 http.port: 9500 #增加新的参数，为了让elasticsearch-head插件可以访问es (5.x版本，如果没有可以自己手动加) http.cors.enabled: true http.cors.allow-origin: "*" cluster.initial_master_nodes: ["elk"] node.name: elk root@elk:~/elk/elasticsearch-7.15.1/config#使用elk用户去启动elasticsearchroot@elk:~# su - elk $ bash elk@elk:~$ elk@elk:~$ mkdir data elk@elk:~/elk/elasticsearch-7.15.1/bin$ cd elk@elk:~$ cd /home/elk/elk/elasticsearch-7.15.1/bin elk@elk:~/elk/elasticsearch-7.15.1/bin$ ./elasticsearch启动之后访问测试：root@elk:~# curl -I http://192.168.1.19:9500/ HTTP/1.1 200 OK X-elastic-product: Elasticsearch Warning: 299 Elasticsearch-7.15.1-83c34f456ae29d60e94d886e455e6a3409bba9ed "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security." content-type: application/json; charset=UTF-8 content-length: 532 root@elk:~#放到后台运行elk@elk:~/elk/elasticsearch-7.15.1/bin$ nohup /home/elk/elk/elasticsearch-7.15.1/bin/elasticsearch >> /home/elk/elk/elasticsearch-7.15.1/output.log 2>&1 & [1] 8811 elk@elk:~/elk/elasticsearch-7.15.1/bin$elk@elk:~$ cd elk/kibana-7.15.1-linux-x86_64/config/ elk@elk:~/elk/kibana-7.15.1-linux-x86_64/config$ vim kibana.yml elk@elk:~/elk/kibana-7.15.1-linux-x86_64/config$ tail -n 18 kibana.yml #设置监听端口为5601 server.port: 5601 #设置可访问的主机地址 server.host: "0.0.0.0" #设置elasticsearch主机地址 elasticsearch.hosts: ["http://localhost:9500"] #如果elasticsearch设置了用户名密码,那么需要配置该两项,如果没配置,那就不用管 #elasticsearch.username: "user" #elasticsearch.password: "pass" elk@elk:~/elk/kibana-7.15.1-linux-x86_64/config$ elk@elk:~$ cd /home/elk/elk/kibana-7.15.1-linux-x86_64/bin elk@elk:~/elk/kibana-7.15.1-linux-x86_64/bin$ ./kibana测试访问root@elk:~# curl -I http://192.168.1.19:5601/app/home#/tutorial_directory HTTP/1.1 200 OK content-security-policy: script-src 'unsafe-eval' 'self'; worker-src blob: 'self'; style-src 'unsafe-inline' 'self' x-content-type-options: nosniff referrer-policy: no-referrer-when-downgrade kbn-name: elk kbn-license-sig: aaa69ea6a0792153cde61e88d0cd9bbad7ddcdaec87b613f281dd275e9dbad47 content-type: text/html; charset=utf-8 cache-control: private, no-cache, no-store, must-revalidate content-length: 144351 vary: accept-encoding Date: Wed, 20 Oct 2021 07:11:10 GMT Connection: keep-alive Keep-Alive: timeout=120 root@elk:~#放到后台运行elk@elk:~/elk/kibana-7.15.1-linux-x86_64/bin$ nohup /home/elk/elk/kibana-7.15.1-linux-x86_64/bin/kibana >> /home/elk/elk/kibana-7.15.1-linux-x86_64/output.log 2>&1 & [2] 9378 elk@elk:~/elk/kibana-7.15.1-linux-x86_64/bin$将日志信息输出到屏幕上elk@elk:~$ cd elk/logstash-7.15.1/bin/ elk@elk:~/elk/logstash-7.15.1/bin$ ./logstash -e 'input {stdin{}} output{stdout{}}' 输入个123然后回车,会把结果输出到屏幕上 { "host" => "elk", "@timestamp" => 2021-10-20T07:15:54.230Z, "@version" => "1", "message" => "" } 123 { "host" => "elk", "@timestamp" => 2021-10-20T07:15:56.453Z, "@version" => "1", "message" => "123" } elk@elk:~/elk/logstash-7.15.1/bin$ cd ../config/ elk@elk:~/elk/logstash-7.15.1/config$ vim logstash elk@elk:~/elk/logstash-7.15.1/config$ cat logstash input { # 从文件读取日志信息 file { path => "/var/log/messages" type => "system" start_position => "beginning" } } filter { } output { # 标准输出 stdout {} } elk@elk:~/elk/logstash-7.15.1/config$ mv logstash logstash.conf elk@elk:~/elk/logstash-7.15.1/config$启动测试elk@elk:~/elk/logstash-7.15.1/config$ cd ../bin/ elk@elk:~/elk/logstash-7.15.1/bin$ ./logstash -f ../config/logstash.conf后台启动elk@elk:~$ nohup /home/elk/elk/logstash-7.15.1/bin/logstash -f /home/elk/elk/logstash-7.15.1/config/logstash.conf >> /home/elk/elk/logstash-7.15.1/output.log 2>&1 & [3] 10177 elk@elk:~$设置开机自启elk@elk:~$ vim startup.sh elk@elk:~$ elk@elk:~$ cat startup.sh #!/bin/bash nohup /home/elk/elk/elasticsearch-7.15.1/bin/elasticsearch >> /home/elk/elk/elasticsearch-7.15.1/output.log 2>&1 & nohup /home/elk/elk/kibana-7.15.1-linux-x86_64/bin/kibana >> /home/elk/elk/kibana-7.15.1-linux-x86_64/output.log 2>&1 & nohup /home/elk/elk/logstash-7.15.1/bin/logstash -f /home/elk/elk/logstash-7.15.1/config/logstash.conf >> /home/elk/elk/logstash-7.15.1/output.log 2>&1 & elk@elk:~$ elk@elk:~$ crontab -e no crontab for elk - using an empty one Select an editor. To change later, run 'select-editor'. 1. /bin/nano <---- easiest 2. /usr/bin/vim.basic 3. /usr/bin/vim.tiny 4. /bin/ed Choose 1-4 [1]: 2 crontab: installing new crontab elk@elk:~$ elk@elk:~$ elk@elk:~$ crontab -l @reboot /home/elk/startup.sh elk@elk:~$logstash插件logstash是通过插件对其功能进行加强插件分类:inputs 输入codecs 解码filters 过滤outputs 输出在Gemfile文件里记录了logstash的插件elk@elk:~$ cd elk/logstash-7.15.1 elk@elk:~/elk/logstash-7.15.1$ ls Gemfile Gemfile elk@elk:~/elk/logstash-7.15.1$去其github上下载插件,地址为:https://github.com/logstash-plugins使用filter插件logstash-filter-mutateelk@elk:~/elk/logstash-7.15.1/config$ vim logstash2.conf #创建一个新的配置文件用来过滤 input { stdin { } } filter { mutate { split => ["message", "|"] } } output { stdout { } }当输入sss|sssni|akok223|23即会按照|分隔符进行分隔其数据处理流程:input–>解码–>filter–>解码–>output启动服务然后去启动logstash服务elk@elk:~$ nohup /home/elk/elk/logstash-7.15.1/bin/logstash -f /home/elk/elk/logstash-7.15.1/config/logstash2.conf >> /home/elk/elk/logstash-7.15.1/output.log 2>&1 & Linux运维交流社区Linux运维交流社区，互联网新闻以及技术交流。41篇原创内容公众号本文使用 文章同步助手 同步
-
Exchangis搭建安装 项目简介Exchangis是一个轻量级的、高扩展性的数据交换平台，支持对结构化及无结构化的异构数据源之间的数据传输，在应用层上具有数据权限管控、节点服务高可用和多租户资源隔离等业务特性，而在数据层上又具有传输架构多样化、模块插件化和组件低耦合等架构特点。Exchangis的传输交换能力依赖于其底层聚合的传输引擎，其顶层对各类数据源定义统一的参数模型，每种传输引擎对参数模型进行映射配置，转化为引擎的输入模型。每聚合一种引擎，都将增加Exchangis一类特性，对某类引擎的特性强化，都是对Exchangis特性的完善。默认聚合以及强化Alibaba的DataX传输引擎。核心特点数据源管理以绑定项目的方式共享自己的数据源；设置数据源对外权限，控制数据的流入和流出。多传输引擎支持传输引擎可横向扩展；当前版本完整聚合了离线批量引擎DataX、部分聚合了大数据批量导数引擎SQOOP近实时任务管控快速抓取传输任务日志以及传输速率等信息，实时关闭任务；可根据带宽状况对任务进行动态限流支持无结构化传输DataX框架改造，单独构建二进制流快速通道，适用于无数据转换的纯数据同步场景。任务状态自检监控长时间运行的任务和状态异常任务，及时释放占用的资源并发出告警。架构设计环境准备基础软件安装MySQL (5.5+) 必选，对应客户端可以选装, Linux服务上若安装mysql的客户端可以通过部署脚本快速初始化数据库JDK (1.8.0_141) 必选Maven (3.6.1+) 必选SQOOP (1.4.6) 可选，如果想要SQOOP做传输引擎，可以安装SQOOP，SQOOP安装依赖Hive,Hadoop环境，这里就不展开来讲Python (2.x) 可选，主要用于调度执行底层DataX的启动脚本，默认的方式是以Java子进程方式执行DataX，用户可以选择以Python方式来做自定义的改造mysql 数据库安装[root@localhost ~]# mkdir mysql [root@localhost ~]# cd mysql [root@localhost mysql]# wget https://downloads.mysql.com/archives/get/p/23/file/mysql-5.7.35-1.el7.x86_64.rpm-bundle.tar [root@localhost mysql]# tar xvf mysql-5.7.35-1.el7.x86_64.rpm-bundle.tar [root@localhost mysql]# yum install ./*.rpm [root@localhost mysql]# systemctl start mysqld.service [root@localhost mysql]# [root@localhost mysql]# systemctl enable mysqld.service [root@localhost mysql]# [root@localhost mysql]# [root@localhost mysql]# sudo grep 'temporary password' /var/log/mysqld.log 2021-10-25T06:57:46.569037Z 1 [Note] A temporary password is generated for root@localhost: (l5aFfIxfNuu [root@localhost mysql]# [root@localhost mysql]# [root@localhost mysql]# [root@localhost mysql]# mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3 Server version: 5.7.35 MySQL Community Server (GPL) Copyright (c) 2000, 2021, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'Cby123..'; Query OK, 0 rows affected (0.00 sec) mysql> mysql> mysql> mysql> use mysql; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> mysql> mysql> update user set host='%' where user ='root'; Query OK, 1 row affected (0.00 sec) Rows matched: 1 Changed: 1 Warnings: 0 mysql> set global validate_password_policy=0; Query OK, 0 rows affected (0.00 sec) mysql> set global validate_password_mixed_case_count=0; Query OK, 0 rows affected (0.01 sec) mysql> set global validate_password_number_count=3; Query OK, 0 rows affected (0.00 sec) mysql> set global validate_password_special_char_count=0; Query OK, 0 rows affected (0.00 sec) mysql> set global validate_password_length=3; Query OK, 0 rows affected (0.00 sec) mysql>jdk安装[root@localhost ~]# mkdir jdk [root@localhost ~]# cd jdk [root@localhost jdk]# [root@localhost jdk]# tar xf jdk-8u141-linux-x64.tar.gz [root@localhost jdk]# [root@localhost jdk]# ll total 181172 drwxr-xr-x. 8 10 143 255 Jul 12 2017 jdk1.8.0_141 -rw-r--r--. 1 root root 185516505 Jul 25 2017 jdk-8u141-linux-x64.tar.gz [root@localhost jdk]# [root@localhost jdk]# vim /etc/profile [root@localhost jdk]# tail -n 3 /etc/profile export JAVA_HOME=/root/jdk/jdk1.8.0_141/ export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar [root@localhost jdk]# [root@localhost jdk]# source /etc/profile [root@localhost jdk]# java -version java version "1.8.0_141" Java(TM) SE Runtime Environment (build 1.8.0_141-b15) Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode) [root@localhost jdk]#maven 安装[root@localhost ~]# [root@localhost ~]# mkdir maven [root@localhost ~]# cd maven [root@localhost maven]# wget https://archive.apache.org/dist/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz [root@localhost maven]# [root@localhost maven]# tar xf apache-maven-3.6.1-bin.tar.gz [root@localhost maven]# ll total 8924 drwxr-xr-x. 6 root root 99 Oct 25 15:08 apache-maven-3.6.1 -rw-r--r--. 1 root root 9136463 Sep 4 2019 apache-maven-3.6.1-bin.tar.gz [root@localhost maven]# vim /etc/profile [root@localhost maven]# [root@localhost maven]# tail -n 3 /etc/profile export MAVEN_HOME=/root/maven/apache-maven-3.6.1 export PATH=$MAVEN_HOME/bin:$PATH:$HOME/bin [root@localhost maven]# [root@localhost maven]# source /etc/profile [root@localhost maven]# mvn -version Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00) Maven home: /root/maven/apache-maven-3.6.1 Java version: 1.8.0_141, vendor: Oracle Corporation, runtime: /root/jdk/jdk1.8.0_141/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-1160.31.1.el7.x86_64", arch: "amd64", family: "unix" [root@localhost maven]#Exchangis安装[root@localhost ~]# [root@localhost ~]# mkdir Exchangis [root@localhost ~]# cd Exchangis [root@localhost Exchangis]# wget https://github.com/WeBankFinTech/Exchangis/releases/download/release-0.5.0/wedatasphere-exchangis-0.5.0.RELEASE.tar.gz [root@localhost Exchangis]# ll total 552904 -rw-r--r--. 1 root root 566172217 Oct 25 15:14 wedatasphere-exchangis-0.5.0.RELEASE.tar.gz [root@localhost Exchangis]# tar xf wedatasphere-exchangis-0.5.0.RELEASE.tar.gz [root@localhost Exchangis]# ll total 552904 drwxr-xr-x. 6 root root 91 Oct 25 15:14 wedatasphere-exchangis-0.5.0.RELEASE -rw-r--r--. 1 root root 566172217 Oct 25 15:14 wedatasphere-exchangis-0.5.0.RELEASE.tar.gz [root@localhost Exchangis]# cd wedatasphere-exchangis-0.5.0.RELEASE [root@localhost wedatasphere-exchangis-0.5.0.RELEASE]# ll total 20 drwxrwxrwx. 2 root root 120 Oct 29 2020 bin drwxrwxrwx. 4 root root 32 May 12 2020 docs drwxrwxrwx. 4 root root 57 May 12 2020 images -rwxrwxrwx. 1 root root 11357 Oct 29 2020 LICENSE drwxr-xr-x. 2 root root 198 Oct 25 15:14 packages -rwxrwxrwx. 1 root root 4582 Oct 29 2020 README.md [root@localhost wedatasphere-exchangis-0.5.0.RELEASE]# cd bin/ [root@localhost bin]# ./install.sh 2021-10-25 15:16:19.723 [INFO] (12476) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/bin/../modules]. 2021-10-25 15:16:19.728 [INFO] (12476) ####### Start To Uncompress Packages ###### 2021-10-25 15:16:19.730 [INFO] (12476) Uncompressing.... Do you want to decompress this package: [exchangis-eureka_0.5.0.RELEASE_1.tar.gz]? (Y/N)y 2021-10-25 15:16:22.691 [INFO] (12476) Uncompress package: [exchangis-eureka_0.5.0.RELEASE_1.tar.gz] to modules directory Do you want to decompress this package: [exchangis-executor_0.5.0.RELEASE_1.tar.gz]? (Y/N)y 2021-10-25 15:16:24.798 [INFO] (12476) Uncompress package: [exchangis-executor_0.5.0.RELEASE_1.tar.gz] to modules directory Do you want to decompress this package: [exchangis-gateway_0.5.0.RELEASE_1.tar.gz]? (Y/N)y 2021-10-25 15:16:31.947 [INFO] (12476) Uncompress package: [exchangis-gateway_0.5.0.RELEASE_1.tar.gz] to modules directory Do you want to decompress this package: [exchangis-service_0.5.0.RELEASE_1.tar.gz]? (Y/N)y 2021-10-25 15:16:35.029 [INFO] (12476) Uncompress package: [exchangis-service_0.5.0.RELEASE_1.tar.gz] to modules directory 2021-10-25 15:16:36.537 [INFO] (12476) ####### Finish To Umcompress Packages ###### Scan modules directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/bin/../modules] to find server under exchangis 2021-10-25 15:16:36.542 [INFO] (12476) ####### Start To Install Modules ###### 2021-10-25 15:16:36.545 [INFO] (12476) Module servers could be installed: [exchangis-eureka] [exchangis-executor] [exchangis-gateway] [exchangis-service] Do you want to confiugre and install [exchangis-eureka]? (Y/N)y 2021-10-25 15:16:37.676 [INFO] (12476) Install module server: [exchangis-eureka] 2021-10-25 15:16:37.706 [INFO] (12527) Start to build directory 2021-10-25 15:16:37.709 [INFO] (12527) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-eureka/bin/../logs]. 2021-10-25 15:16:37.779 [INFO] (12527) Directory or file: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-eureka/bin/../conf] has been exist 2021-10-25 15:16:37.782 [INFO] (12527) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-eureka/bin/../data]. Do you want to confiugre and install [exchangis-executor]? (Y/N)y 2021-10-25 15:16:38.529 [INFO] (12476) Install module server: [exchangis-executor] 2021-10-25 15:16:38.558 [INFO] (12565) Start to build directory 2021-10-25 15:16:38.561 [INFO] (12565) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-executor/bin/../logs]. 2021-10-25 15:16:38.596 [INFO] (12565) Directory or file: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-executor/bin/../conf] has been exist 2021-10-25 15:16:38.599 [INFO] (12565) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-executor/bin/../data]. Do you want to confiugre and install [exchangis-gateway]? (Y/N)y 2021-10-25 15:16:39.291 [INFO] (12476) Install module server: [exchangis-gateway] 2021-10-25 15:16:39.317 [INFO] (12603) Start to build directory 2021-10-25 15:16:39.320 [INFO] (12603) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-gateway/bin/../logs]. 2021-10-25 15:16:39.354 [INFO] (12603) Directory or file: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-gateway/bin/../conf] has been exist 2021-10-25 15:16:39.356 [INFO] (12603) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-gateway/bin/../data]. Do you want to confiugre and install [exchangis-service]? (Y/N)y 2021-10-25 15:16:39.991 [INFO] (12476) Install module server: [exchangis-service] 2021-10-25 15:16:40.017 [INFO] (12641) Start to build directory 2021-10-25 15:16:40.020 [INFO] (12641) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-service/bin/../logs]. 2021-10-25 15:16:40.056 [INFO] (12641) Directory or file: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-service/bin/../conf] has been exist 2021-10-25 15:16:40.059 [INFO] (12641) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-service/bin/../data]. 2021-10-25 15:16:40.099 [INFO] (12641) Scan out mysql command, so begin to initalize the database Do you want to initalize database with sql: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/bin/exchangis-init.sql]? (Y/N)y Please input the db host(default: 127.0.0.1): Please input the db port(default: 3306): Please input the db username(default: root): Please input the db password(default: ): Cby123.. Please input the db name(default: exchangis) mysql: [Warning] Using a password on the command line interface can be insecure. 2021-10-25 15:16:55.665 [INFO] (12476) ####### Finish To Install Modules ###### [root@localhost bin]# [root@localhost bin]# ./start-all.sh 2021-10-25 15:18:22.181 [INFO] (12691) Try To Start Modules In Order 2021-10-25 15:18:22.189 [INFO] (12699) ####### Begin To Start Module: [exchangis-eureka] ###### 2021-10-25 15:18:22.199 [INFO] (12707) load environment variables 2021-10-25 15:18:22.717 [INFO] (12707) /root/jdk/jdk1.8.0_141//bin/java 2021-10-25 15:18:22.721 [INFO] (12707) Waiting EXCHANGIS-EUREKA to start complete ... 2021-10-25 15:18:22.994 [INFO] (12707) EXCHANGIS-EUREKA start success 2021-10-25 15:18:23.003 [INFO] (13009) ####### Begin To Start Module: [exchangis-gateway] ###### 2021-10-25 15:18:23.012 [INFO] (13017) load environment variables 2021-10-25 15:18:23.493 [INFO] (13017) /root/jdk/jdk1.8.0_141//bin/java 2021-10-25 15:18:23.497 [INFO] (13017) Waiting EXCHANGIS-GATEWAY to start complete ... 2021-10-25 15:18:24.081 [INFO] (13017) EXCHANGIS-GATEWAY start success 2021-10-25 15:18:24.091 [INFO] (13321) ####### Begin To Start Module: [exchangis-service] ###### 2021-10-25 15:18:24.099 [INFO] (13329) load environment variables 2021-10-25 15:18:24.933 [INFO] (13329) /root/jdk/jdk1.8.0_141//bin/java 2021-10-25 15:18:24.936 [INFO] (13329) Waiting EXCHANGIS-SERVICE to start complete ... 2021-10-25 15:18:26.398 [INFO] (13329) EXCHANGIS-SERVICE start success 2021-10-25 15:18:26.410 [INFO] (13634) ####### Begin To Start Module: [exchangis-executor] ###### 2021-10-25 15:18:26.423 [INFO] (13643) load environment variables 2021-10-25 15:18:27.677 [INFO] (13643) /root/jdk/jdk1.8.0_141//bin/java 2021-10-25 15:18:27.681 [INFO] (13643) Waiting EXCHANGIS-EXECUTOR to start complete ... 2021-10-25 15:18:28.441 [INFO] (13643) EXCHANGIS-EXECUTOR start success [root@localhost bin]#**登陆访问 ** 注册中心：http://192.168.1.161:8500/访问地址：http://192.168.1.161:9503/账号：admin密码：adminLinux运维交流社区Linux运维交流社区，互联网新闻以及技术交流。43篇原创内容公众号本文使用 文章同步助手 同步
-
使用 Istioctl 安装 istio 使用 Istioctl 安装 istio下载 Istio转到 Istio 发布 页面，下载针对你操作系统的安装文件， 或用自动化工具下载并提取最新版本（Linux 或 macOS）：[root@k8s-master-node1 ~]# curl -L https://istio.io/downloadIstio | sh - 若无法下载可以手动写入文件进行执行 脚本内容：#!/bin/sh # Copyright Istio Authors # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # This file will be fetched as: curl -L https://git.io/getLatestIstio | sh - # so it should be pure bourne shell, not bash (and not reference other scripts) # # The script fetches the latest Istio release candidate and untars it. # You can pass variables on the command line to download a specific version # or to override the processor architecture. For example, to download # Istio 1.6.8 for the x86_64 architecture, # run curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.6.8 TARGET_ARCH=x86_64 sh -. set -e # Determines the operating system. OS="$(uname)" if [ "x${OS}" = "xDarwin" ] ; then OSEXT="osx" else OSEXT="linux" fi # Determine the latest Istio version by version number ignoring alpha, beta, and rc versions. if [ "x${ISTIO_VERSION}" = "x" ] ; then ISTIO_VERSION="$(curl -sL https://github.com/istio/istio/releases | \ grep -o 'releases/[0-9]*.[0-9]*.[0-9]*/' | sort -V | \ tail -1 | awk -F'/' '{ print $2}')" ISTIO_VERSION="${ISTIO_VERSION##*/}" fi LOCAL_ARCH=$(uname -m) if [ "${TARGET_ARCH}" ]; then LOCAL_ARCH=${TARGET_ARCH} fi case "${LOCAL_ARCH}" in x86_64) ISTIO_ARCH=amd64 ;; armv8*) ISTIO_ARCH=arm64 ;; aarch64*) ISTIO_ARCH=arm64 ;; armv*) ISTIO_ARCH=armv7 ;; amd64|arm64) ISTIO_ARCH=${LOCAL_ARCH} ;; *) echo "This system's architecture, ${LOCAL_ARCH}, isn't supported" exit 1 ;; esac if [ "x${ISTIO_VERSION}" = "x" ] ; then printf "Unable to get latest Istio version. Set ISTIO_VERSION env var and re-run. For example: export ISTIO_VERSION=1.0.4" exit 1; fi NAME="istio-$ISTIO_VERSION" URL="https://github.com/istio/istio/releases/download/${ISTIO_VERSION}/istio-${ISTIO_VERSION}-${OSEXT}.tar.gz" ARCH_URL="https://github.com/istio/istio/releases/download/${ISTIO_VERSION}/istio-${ISTIO_VERSION}-${OSEXT}-${ISTIO_ARCH}.tar.gz" with_arch() { printf "\nDownloading %s from %s ...\n" "$NAME" "$ARCH_URL" if ! curl -o /dev/null -sIf "$ARCH_URL"; then printf "\n%s is not found, please specify a valid ISTIO_VERSION and TARGET_ARCH\n" "$ARCH_URL" exit 1 fi curl -fsLO "$ARCH_URL" filename="istio-${ISTIO_VERSION}-${OSEXT}-${ISTIO_ARCH}.tar.gz" tar -xzf "${filename}" rm "${filename}" } without_arch() { printf "\nDownloading %s from %s ..." "$NAME" "$URL" if ! curl -o /dev/null -sIf "$URL"; then printf "\n%s is not found, please specify a valid ISTIO_VERSION\n" "$URL" exit 1 fi curl -fsLO "$URL" filename="istio-${ISTIO_VERSION}-${OSEXT}.tar.gz" tar -xzf "${filename}" rm "${filename}" } # Istio 1.6 and above support arch # Istio 1.5 and below do not have arch support ARCH_SUPPORTED="1.6" if [ "${OS}" = "Linux" ] ; then # This checks if ISTIO_VERSION is less than ARCH_SUPPORTED (version-sort's before it) if [ "$(printf '%s\n%s' "${ARCH_SUPPORTED}" "${ISTIO_VERSION}" | sort -V | head -n 1)" = "${ISTIO_VERSION}" ]; then without_arch else with_arch fi elif [ "x${OS}" = "xDarwin" ] ; then without_arch else printf "\n\n" printf "Unable to download Istio %s at this moment!\n" "$ISTIO_VERSION" printf "Please verify the version you are trying to download.\n\n" exit 1 fi printf "" printf "\nIstio %s Download Complete!\n" "$ISTIO_VERSION" printf "\n" printf "Istio has been successfully downloaded into the %s folder on your system.\n" "$NAME" printf "\n" BINDIR="$(cd "$NAME/bin" && pwd)" printf "Next Steps:\n" printf "See https://istio.io/latest/docs/setup/install/ to add Istio to your Kubernetes cluster.\n" printf "\n" printf "To configure the istioctl client tool for your workstation,\n" printf "add the %s directory to your environment path variable with:\n" "$BINDIR" printf "\t export PATH=\"\$PATH:%s\"\n" "$BINDIR" printf "\n" printf "Begin the Istio pre-installation check by running:\n" printf "\t istioctl x precheck \n" printf "\n" printf "Need more information? Visit https://istio.io/latest/docs/setup/install/ \n"[root@k8s-master-node1 ~]# bash istio.sh 转到 Istio 包目录。例如，如果包是 istio-1.11.4：[root@k8s-master-node1 ~]# cd istio-1.11.4/ [root@k8s-master-node1 ~/istio-1.11.4]# ll total 28 drwxr-x---. 2 root root 22 Oct 13 22:50 bin -rw-r--r--. 1 root root 11348 Oct 13 22:50 LICENSE drwxr-xr-x. 5 root root 52 Oct 13 22:50 manifests -rw-r-----. 1 root root 854 Oct 13 22:50 manifest.yaml -rw-r--r--. 1 root root 5866 Oct 13 22:50 README.md drwxr-xr-x. 21 root root 4096 Oct 13 22:50 samples drwxr-xr-x. 3 root root 57 Oct 13 22:50 tools [root@k8s-master-node1 ~/istio-1.11.4]#安装目录包含：samples/ 目录下的示例应用程序bin/ 目录下的 istioctl 客户端二进制文件 .将 istioctl 客户端加入搜索路径（Linux or macOS）:$ export PATH=$PWD/bin:$PATH export PATH=/root/istio-1.11.4/bin:$PATH [root@k8s-master-node1 ~/istio-1.11.4]# export PATH=$PWD/bin:$PATH [root@k8s-master-node1 ~/istio-1.11.4]# [root@k8s-master-node1 ~/istio-1.11.4]# vim /etc/profile [root@k8s-master-node1 ~/istio-1.11.4]# tail -n 2 /etc/profile export PATH=/root/istio-1.11.4/bin:$PATH [root@k8s-master-node1 ~/istio-1.11.4]#使用默认配置档安装 Istio最简单的选择是用下面命令安装 Istio 默认 配置档：[root@k8s-master-node1 ~]# istioctl version no running Istio pods in "istio-system" 1.11.4 [root@k8s-master-node1 ~]# [root@k8s-master-node1 ~]# [root@k8s-master-node1 ~]# istioctl install --set profile=demo -y ✔ Istio core installed ✔ Istiod installed ✔ Egress gateways installed ✔ Ingress gateways installed ✔ Installation complete Thank you for installing Istio 1.11. Please take a few minutes to tell us about your install/upgrade experience! https://forms.gle/kWULBRjUv7hHci7T6 [root@k8s-master-node1 ~]#查看istio相应的 namespace 和 pod 是否已经正常创建[root@k8s-master-node1 ~]# [root@k8s-master-node1 ~]# kubectl get pods -n istio-system NAME READY STATUS RESTARTS AGE istio-egressgateway-756d4db566-wh949 1/1 Running 0 2m istio-ingressgateway-8577c57fb6-2vrtg 1/1 Running 0 2m istiod-5847c59c69-l2dt2 1/1 Running 0 2m39s [root@k8s-master-node1 ~]# [root@k8s-master-node1 ~]# [root@k8s-master-node1 ~]# [root@k8s-master-node1 ~]#检查 istio 的 CRD 和 API 资源[root@k8s-master-node1 ~]# [root@k8s-master-node1 ~]# kubectl get crd |grep istio authorizationpolicies.security.istio.io 2021-11-01T09:43:55Z destinationrules.networking.istio.io 2021-11-01T09:43:55Z envoyfilters.networking.istio.io 2021-11-01T09:43:55Z gateways.networking.istio.io 2021-11-01T09:43:55Z istiooperators.install.istio.io 2021-11-01T09:43:55Z peerauthentications.security.istio.io 2021-11-01T09:43:55Z requestauthentications.security.istio.io 2021-11-01T09:43:55Z serviceentries.networking.istio.io 2021-11-01T09:43:55Z sidecars.networking.istio.io 2021-11-01T09:43:55Z telemetries.telemetry.istio.io 2021-11-01T09:43:55Z virtualservices.networking.istio.io 2021-11-01T09:43:55Z workloadentries.networking.istio.io 2021-11-01T09:43:55Z workloadgroups.networking.istio.io 2021-11-01T09:43:55Z [root@k8s-master-node1 ~]# [root@k8s-master-node1 ~]# kubectl api-resources |grep istio istiooperators iop,io install.istio.io/v1alpha1 true IstioOperator destinationrules dr networking.istio.io/v1beta1 true DestinationRule envoyfilters networking.istio.io/v1alpha3 true EnvoyFilter gateways gw networking.istio.io/v1beta1 true Gateway serviceentries se networking.istio.io/v1beta1 true ServiceEntry sidecars networking.istio.io/v1beta1 true Sidecar virtualservices vs networking.istio.io/v1beta1 true VirtualService workloadentries we networking.istio.io/v1beta1 true WorkloadEntry workloadgroups wg networking.istio.io/v1alpha3 true WorkloadGroup authorizationpolicies security.istio.io/v1beta1 true AuthorizationPolicy peerauthentications pa security.istio.io/v1beta1 true PeerAuthentication requestauthentications ra security.istio.io/v1beta1 true RequestAuthentication telemetries telemetry telemetry.istio.io/v1alpha1 true Telemetry [root@k8s-master-node1 ~]#安装 dashboard 组件。命令如下[root@k8s-master-node1 ~]# kubectl apply -f /root/istio-1.11.4/samples/addons/ -n istio-system serviceaccount/grafana created configmap/grafana created service/grafana created deployment.apps/grafana created configmap/istio-grafana-dashboards created configmap/istio-services-grafana-dashboards created deployment.apps/jaeger created service/tracing created service/zipkin created service/jaeger-collector created serviceaccount/kiali created configmap/kiali created clusterrole.rbac.authorization.k8s.io/kiali-viewer created clusterrole.rbac.authorization.k8s.io/kiali created clusterrolebinding.rbac.authorization.k8s.io/kiali created role.rbac.authorization.k8s.io/kiali-controlplane created rolebinding.rbac.authorization.k8s.io/kiali-controlplane created service/kiali created deployment.apps/kiali created serviceaccount/prometheus created configmap/prometheus created clusterrole.rbac.authorization.k8s.io/prometheus created clusterrolebinding.rbac.authorization.k8s.io/prometheus created service/prometheus created deployment.apps/prometheus created [root@k8s-master-node1 ~]# [root@k8s-master-node1 ~]# kubectl get pods -n istio-system NAME READY STATUS RESTARTS AGE grafana-68cc7d6d78-792cw 1/1 Running 0 88s istio-egressgateway-756d4db566-wh949 1/1 Running 0 6m9s istio-ingressgateway-8577c57fb6-2vrtg 1/1 Running 0 6m9s istiod-5847c59c69-l2dt2 1/1 Running 0 6m48s jaeger-5d44bc5c5d-n6zjq 1/1 Running 0 88s kiali-fd9f88575-svz7g 1/1 Running 0 87s prometheus-77b49cb997-7d4s9 2/2 Running 0 86s [root@k8s-master-node1 ~]#将istio-ingressgateway改为NodePort方式，方便访问[root@k8s-master-node1 ~]# kubectl patch service istio-ingressgateway -n istio-system -p '{"spec":{"type":"NodePort"}}' service/istio-ingressgateway patched [root@k8s-master-node1 ~]#Linux运维交流社区Linux运维交流社区，互联网新闻以及技术交流。46篇原创内容公众号 https://blog.csdn.net/qq_33921750https://my.oschina.net/u/3981543https://www.zhihu.com/people/chen-bu-yun-2https://segmentfault.com/u/hppyvyv6/articleshttps://juejin.cn/user/3315782802482007https://space.bilibili.com/352476552/articlehttps://cloud.tencent.com/developer/column/93230知乎、CSDN、开源中国、思否、掘金、哔哩哔哩、腾讯云本文使用 文章同步助手 同步
-
KubeSphere 升级 && 安装后启用插件 KubeSphere 升级root@master1:~# export KKZONE=cn root@master1:~# kk upgrade --with-kubernetes v1.22.1 --with-kubesphere v3.2.0 -f sample.yaml 启用插件用户可以使用 KubeSphere Web 控制台查看和操作不同的资源。要在安装后启用可插拔组件，只需要在控制台中进行略微调整。对于那些习惯使用 Kubernetes 命令行工具 kubectl 的人来说，由于该工具已集成到控制台中，因此使用 KubeSphere 将毫无困难。以 admin 身份登录控制台。点击左上角的平台管理 ，然后选择集群管理。集群管理点击 CRD，然后在搜索栏中输入 clusterconfiguration，点击搜索结果进入其详情页面。CRD编辑配置文件在该配置文件中，将对应组件 enabled 的 false 更改为 true，以启用要安装的组件。完成后，点击更新以保存配置。我的内容：apiVersion: installer.kubesphere.io/v1alpha1 kind: ClusterConfiguration metadata: labels: version: v3.2.0 name: ks-installer namespace: kubesphere-system spec: alerting: enabled: true auditing: enabled: true authentication: jwtSecret: '' common: core: console: enableMultiLogin: true port: 30880 type: NodePort es: basicAuth: enabled: true password: '' username: '' data: volumeSize: 20Gi elkPrefix: logstash externalElasticsearchPort: '' externalElasticsearchUrl: '' logMaxAge: 7 master: volumeSize: 4Gi gpu: kinds: - default: true resourceName: nvidia.com/gpu resourceType: GPU minio: volumeSize: 20Gi monitoring: GPUMonitoring: enabled: true endpoint: 'http://prometheus-operated.kubesphere-monitoring-system.svc:9090' openldap: enabled: true redis: enabled: true devops: enabled: true jenkinsJavaOpts_MaxRAM: 2g jenkinsJavaOpts_Xms: 512m jenkinsJavaOpts_Xmx: 512m jenkinsMemoryLim: 2Gi jenkinsMemoryReq: 1500Mi jenkinsVolumeSize: 8Gi etcd: endpointIps: 192.168.1.10 monitoring: false port: 2379 tlsEnable: true events: enabled: true kubeedge: cloudCore: cloudHub: advertiseAddress: - '' nodeLimit: '100' cloudhubHttpsPort: '10002' cloudhubPort: '10000' cloudhubQuicPort: '10001' cloudstreamPort: '10003' nodeSelector: node-role.kubernetes.io/worker: '' service: cloudhubHttpsNodePort: '30002' cloudhubNodePort: '30000' cloudhubQuicNodePort: '30001' cloudstreamNodePort: '30003' tunnelNodePort: '30004' tolerations: [] tunnelPort: '10004' edgeWatcher: edgeWatcherAgent: nodeSelector: node-role.kubernetes.io/worker: '' tolerations: [] nodeSelector: node-role.kubernetes.io/worker: '' tolerations: [] enabled: true logging: containerruntime: docker enabled: true logsidecar: enabled: true replicas: 2 metrics_server: enabled: true monitoring: gpu: nvidia_dcgm_exporter: enabled: true storageClass: '' multicluster: clusterRole: none network: ippool: type: none networkpolicy: enabled: true topology: type: none openpitrix: store: enabled: true persistence: storageClass: '' servicemesh: enabled: true 启用组件执行以下命令，使用 Web kubectl 来检查安装过程：root@master1:~# kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f 如果组件安装成功，输出将显示以下消息。##################################################### ### Welcome to KubeSphere! ### ##################################################### Console: http://192.168.0.2:30880 Account: admin Password: P@88w0rd NOTES： 1. After you log into the console, please check the monitoring status of service components in "Cluster Management". If any service is not ready, please wait patiently until all components are up and running. 2. Please change the default password after login. ##################################################### https://kubesphere.io 20xx-xx-xx xx:xx:xx #####################################################登录 KubeSphere 控制台，在系统组件中可以查看不同组件的状态。服务组件Linux运维交流社区Linux运维交流社区，互联网新闻以及技术交流。47篇原创内容公众号 https://blog.csdn.net/qq_33921750https://my.oschina.net/u/3981543https://www.zhihu.com/people/chen-bu-yun-2https://segmentfault.com/u/hppyvyv6/articleshttps://juejin.cn/user/3315782802482007https://space.bilibili.com/352476552/articlehttps://cloud.tencent.com/developer/column/93230知乎、CSDN、开源中国、思否、掘金、哔哩哔哩、腾讯云本文使用 文章同步助手 同步