diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..4ed90b9 --- /dev/null +++ b/LICENSE @@ -0,0 +1,208 @@ +Apache License + +Version 2.0, January 2004 + +http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, +AND DISTRIBUTION + + 1. Definitions. + + + +"License" shall mean the terms and conditions for use, reproduction, and distribution +as defined by Sections 1 through 9 of this document. + + + +"Licensor" shall mean the copyright owner or entity authorized by the copyright +owner that is granting the License. + + + +"Legal Entity" shall mean the union of the acting entity and all other entities +that control, are controlled by, or are under common control with that entity. +For the purposes of this definition, "control" means (i) the power, direct +or indirect, to cause the direction or management of such entity, whether +by contract or otherwise, or (ii) ownership of fifty percent (50%) or more +of the outstanding shares, or (iii) beneficial ownership of such entity. + + + +"You" (or "Your") shall mean an individual or Legal Entity exercising permissions +granted by this License. + + + +"Source" form shall mean the preferred form for making modifications, including +but not limited to software source code, documentation source, and configuration +files. + + + +"Object" form shall mean any form resulting from mechanical transformation +or translation of a Source form, including but not limited to compiled object +code, generated documentation, and conversions to other media types. + + + +"Work" shall mean the work of authorship, whether in Source or Object form, +made available under the License, as indicated by a copyright notice that +is included in or attached to the work (an example is provided in the Appendix +below). + + + +"Derivative Works" shall mean any work, whether in Source or Object form, +that is based on (or derived from) the Work and for which the editorial revisions, +annotations, elaborations, or other modifications represent, as a whole, an +original work of authorship. For the purposes of this License, Derivative +Works shall not include works that remain separable from, or merely link (or +bind by name) to the interfaces of, the Work and Derivative Works thereof. + + + +"Contribution" shall mean any work of authorship, including the original version +of the Work and any modifications or additions to that Work or Derivative +Works thereof, that is intentionally submitted to Licensor for inclusion in +the Work by the copyright owner or by an individual or Legal Entity authorized +to submit on behalf of the copyright owner. For the purposes of this definition, +"submitted" means any form of electronic, verbal, or written communication +sent to the Licensor or its representatives, including but not limited to +communication on electronic mailing lists, source code control systems, and +issue tracking systems that are managed by, or on behalf of, the Licensor +for the purpose of discussing and improving the Work, but excluding communication +that is conspicuously marked or otherwise designated in writing by the copyright +owner as "Not a Contribution." + + + +"Contributor" shall mean Licensor and any individual or Legal Entity on behalf +of whom a Contribution has been received by Licensor and subsequently incorporated +within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of this +License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, +no-charge, royalty-free, irrevocable copyright license to reproduce, prepare +Derivative Works of, publicly display, publicly perform, sublicense, and distribute +the Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of this License, +each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, +no-charge, royalty-free, irrevocable (except as stated in this section) patent +license to make, have made, use, offer to sell, sell, import, and otherwise +transfer the Work, where such license applies only to those patent claims +licensable by such Contributor that are necessarily infringed by their Contribution(s) +alone or by combination of their Contribution(s) with the Work to which such +Contribution(s) was submitted. If You institute patent litigation against +any entity (including a cross-claim or counterclaim in a lawsuit) alleging +that the Work or a Contribution incorporated within the Work constitutes direct +or contributory patent infringement, then any patent licenses granted to You +under this License for that Work shall terminate as of the date such litigation +is filed. + +4. Redistribution. You may reproduce and distribute copies of the Work or +Derivative Works thereof in any medium, with or without modifications, and +in Source or Object form, provided that You meet the following conditions: + +(a) You must give any other recipients of the Work or Derivative Works a copy +of this License; and + +(b) You must cause any modified files to carry prominent notices stating that +You changed the files; and + +(c) You must retain, in the Source form of any Derivative Works that You distribute, +all copyright, patent, trademark, and attribution notices from the Source +form of the Work, excluding those notices that do not pertain to any part +of the Derivative Works; and + +(d) If the Work includes a "NOTICE" text file as part of its distribution, +then any Derivative Works that You distribute must include a readable copy +of the attribution notices contained within such NOTICE file, excluding those +notices that do not pertain to any part of the Derivative Works, in at least +one of the following places: within a NOTICE text file distributed as part +of the Derivative Works; within the Source form or documentation, if provided +along with the Derivative Works; or, within a display generated by the Derivative +Works, if and wherever such third-party notices normally appear. The contents +of the NOTICE file are for informational purposes only and do not modify the +License. You may add Your own attribution notices within Derivative Works +that You distribute, alongside or as an addendum to the NOTICE text from the +Work, provided that such additional attribution notices cannot be construed +as modifying the License. + +You may add Your own copyright statement to Your modifications and may provide +additional or different license terms and conditions for use, reproduction, +or distribution of Your modifications, or for any such Derivative Works as +a whole, provided Your use, reproduction, and distribution of the Work otherwise +complies with the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, any +Contribution intentionally submitted for inclusion in the Work by You to the +Licensor shall be under the terms and conditions of this License, without +any additional terms or conditions. Notwithstanding the above, nothing herein +shall supersede or modify the terms of any separate license agreement you +may have executed with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade names, +trademarks, service marks, or product names of the Licensor, except as required +for reasonable and customary use in describing the origin of the Work and +reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or agreed to +in writing, Licensor provides the Work (and each Contributor provides its +Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied, including, without limitation, any warranties +or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR +A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness +of using or redistributing the Work and assume any risks associated with Your +exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, whether +in tort (including negligence), contract, or otherwise, unless required by +applicable law (such as deliberate and grossly negligent acts) or agreed to +in writing, shall any Contributor be liable to You for damages, including +any direct, indirect, special, incidental, or consequential damages of any +character arising as a result of this License or out of the use or inability +to use the Work (including but not limited to damages for loss of goodwill, +work stoppage, computer failure or malfunction, or any and all other commercial +damages or losses), even if such Contributor has been advised of the possibility +of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing the Work +or Derivative Works thereof, You may choose to offer, and charge a fee for, +acceptance of support, warranty, indemnity, or other liability obligations +and/or rights consistent with this License. However, in accepting such obligations, +You may act only on Your own behalf and on Your sole responsibility, not on +behalf of any other Contributor, and only if You agree to indemnify, defend, +and hold each Contributor harmless for any liability incurred by, or claims +asserted against, such Contributor by reason of your accepting any such warranty +or additional liability. END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work. + +To apply the Apache License to your work, attach the following boilerplate +notice, with the fields enclosed by brackets "[]" replaced with your own identifying +information. (Don't include the brackets!) The text should be enclosed in +the appropriate comment syntax for the file format. We also recommend that +a file or class name and description of purpose be included on the same "printed +page" as the copyright notice for easier identification within third-party +archives. + +Copyright [yyyy] [name of copyright owner] + +Licensed under the Apache License, Version 2.0 (the "License"); + +you may not use this file except in compliance with the License. + +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software + +distributed under the License is distributed on an "AS IS" BASIS, + +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + +See the License for the specific language governing permissions and + +limitations under the License. diff --git a/README.md b/README.md index 927981e..cd5f13c 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,98 @@ -# ccyunchina-deploy +# 说明 +*部署脚本于长城专有云平台* +部署脚本中包括了kubernetes底层组件、efk等一系列服务的安装。采用ansbile脚本实现自动安装，运维人员需要对ansible工具有一定简单了解。主要服务基本实现全容器化、k8s化部署，可以通过kubernetes dashboard监控到所有容器服务；部署脚本基于centos 7 或者kylin v10 sp1，要求内核版本为4以上；既提供一键快速安装方式，也提供分步执行安装方式。 +# 组件版本 +| 名称 | 版本号 | 备注 | +|-------------------------|--------------|----------| +| Kernel | 4以上 | | +| kube-apiserver | 1.21.0 | | +| kube-controller-manager | 1.21.0 | | +| kube-scheduler | 1.21.0 | | +| kube-proxy | 1.21.0 | | +| kubelet | 1.21.0 | | +| etcd | 3.3.15 | | +| calico | 3.3.1 | | +| docker | 18.06.3-ce | | +| coredns | 1.6.2 | | +| kubernets-dashboard | 1.10.1 | | +| traefik | 2.1.1 | | +| pause | 3.1 | | +| elasticsearch | 6.2.4 | | +| keepalived | 2.0.19-r0 | | +| Haproxy | 2.1.2 | | +| gluster | 4.0 | | +| heketi | 6.0 | | +| metrics-server | 0.2.0 | | +| node-problem-detector | 0.4.1 | | +| openvpn | 2.1 | | +| efk | 7.10.2 | | +| mysql | 8.0 | | +| redis | 5.0 | | +| rocketmq | 4.5.0 | | +| gw-proprietary-cloud | 0.1 | | +# 部署示意图 +![k8s部署图](https://github.com/toyangdon/k8s_deploy/blob/master/kubernetes%20%E7%BB%84%E7%BB%87%E5%9B%BE.png?raw=true) +# 快速安装 +1. 安装ansible +`yum install -y ansible` +2. 下载部署文件到部署节点的/etc/ansible目录下 +`git clone –depth=1 https://git.trustie.net/toyangdon/proprietary-cloud-deploy.git` +将部署文件复制到/etc/ansible目录下 +`cp -rf k8s_deploy/* /etc/ansible/` +3. 配置集群安装信息 +**根据实际情况修改`hosts`文件** +`vi /etc/ansible/hosts` +4. 配置ssh免密码 +`sh tools/ssh-key-copy.sh root ${passwd} #请输入实际的root用户密码` +5. 执行一键安装 +**centos** +`ansible-playbook setup.yml` +**kylin v10 sp1** +`ansible-playbook -e 'ansible_python_interpreter=/usr/bin/python3.7' setup.yml` +**单机部署** +`ansible-playbook -e 'apiserver_mem_requests=100Mi' -e 'apiserver_cpu_requests=100m' setup.yml` +# 分步安装 +`playbooks`目录提供分步安装的相关playbook，主要分为两大块`kubernetes`和`gpaas` +## `kubernetes` 部署 +1. `ansible-playbook playbooks/kubernetes/00.check.yml` 检查集群服务器 +1. `ansible-playbook playbooks/kubernetes/01.docker.yml` 在所有主机上安装并启动docker服务 +2. `ansible-playbook playbooks/kubernetes/02.prepare.yml` 服务器通用配置，生成并分发集群所需相关证书 +3. `ansible-playbook playbooks/kubernetes/03.harbor.yml` 部署harbor节点，安装并启动harbor服务（可选） +4. `ansible-playbook playbooks/kubernetes/04.lb.yml` 准备lb节点所需的相关安装文件，包括keepalived和haproxy +5. `ansible-playbook playbooks/kubernetes/05.kube-master.yml` 准备master节点所需的相关安装文件 +6. `ansible-playbook playbooks/kubernetes/06.kube-node.yml` 在主机上安装并启动kubelet服务，先启动lb,再启动master，最后启动kube-node +7. `ansible-playbook playbooks/kubernetes/07.calico.yml` 在主机上准备calico服务所需要的相关安装文件（与flannel可选） +8. `ansible-playbook playbooks/kubernetes/07.flannel.yml` 在主机上准备flannel服务所需要的相关安装文件（与calico可选） （暂时不可用） +9. `ansible-playbook playbooks/kubernetes/09.storage-nfs.yml` 安装nfs服务（与gfs可选）（暂时不可用） +10. `ansible-playbook playbooks/kubernetes/10.storage-gluster.yml` 准备安装gfs服务 +11. `ansible-playbook playbooks/kubernetes/20.addnode.yml` 新增节点 +12. `ansible-playbook playbooks/kubernetes/30.addons.yml` kubernetes所有插件服务的部署，包括kube-proxy、kubedns、calico、glusterfs等等 +13. `ansible-playbook playbooks/kubernetes/90.setup.yml` 一键安装kubernetes,即顺序执行以上所有步骤（除了20.addnode） +14. `ansible-playbook playbooks/kubernetes/99.clean.yml` 一键清理kubernetes集群（慎用） + +## `gpass` 部署 +目前分为`efk`和`monitor`二部分 + +### `efk` 部署 +1. `ansible-playbook playbooks/gpaas/elk/01.es.yml` es部署 +3. `ansible-playbook playbooks/gpaas/elk/02.fluentd.yml` fluentd +4. `ansible-playbook playbooks/gpaas/elk/03.kibana.yml` kibana部署 +5. `ansible-playbook playbooks/gpaas/elk/90.setup.yml` 一键安装elk，即顺序执行以上所有步骤 + +### `monitor` 部署 +1. `ansible-playbook playbooks/gpaas/monitor/01.prometheus.yml` prometheus部署 +2. `ansible-playbook playbooks/gpaas/monitor/90.setup.yml` 一键安装监控平台，即顺序执行以上所有步骤 + +### 一键部署`gpass` +1. `ansible-playbook playbooks/gpaas/90.setup.yml` + +## `专有云组件`部署 +1. `ansible-playbook playbooks/proprietary-cloud/90.setup.yml` + +# `ansible`容器化 +1. 安装docker +`systemctl stop firewalld` #关闭防火墙 +`sh tools/docker/install.sh` +2. 运行ansible镜像执行k8s安装 +`/opt/k8s/bin/docker run --name ansible -it --privileged -v /root/k8s-install:/etc/ansible toyangdon/ansible:latest` #进入容器内执行ansible脚本 diff --git a/bin/ansible/PyYAML-3.10-11.el7.aarch64.rpm b/bin/ansible/PyYAML-3.10-11.el7.aarch64.rpm new file mode 100644 index 0000000..06b6ffd Binary files /dev/null and b/bin/ansible/PyYAML-3.10-11.el7.aarch64.rpm differ diff --git a/bin/ansible/ansible-2.8.5-1.el7.noarch.rpm b/bin/ansible/ansible-2.8.5-1.el7.noarch.rpm new file mode 100644 index 0000000..8fac727 Binary files /dev/null and b/bin/ansible/ansible-2.8.5-1.el7.noarch.rpm differ diff --git a/bin/ansible/libyaml-0.1.4-11.el7.aarch64.rpm b/bin/ansible/libyaml-0.1.4-11.el7.aarch64.rpm new file mode 100644 index 0000000..27e6e4a Binary files /dev/null and b/bin/ansible/libyaml-0.1.4-11.el7.aarch64.rpm differ diff --git a/bin/ansible/python-babel-0.9.6-8.el7.noarch.rpm b/bin/ansible/python-babel-0.9.6-8.el7.noarch.rpm new file mode 100644 index 0000000..d481181 Binary files /dev/null and b/bin/ansible/python-babel-0.9.6-8.el7.noarch.rpm differ diff --git a/bin/ansible/python-backports-1.0-8.el7.aarch64.rpm b/bin/ansible/python-backports-1.0-8.el7.aarch64.rpm new file mode 100644 index 0000000..998eb66 Binary files /dev/null and b/bin/ansible/python-backports-1.0-8.el7.aarch64.rpm differ diff --git a/bin/ansible/python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch.rpm b/bin/ansible/python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch.rpm new file mode 100644 index 0000000..1ee4fc1 Binary files /dev/null and b/bin/ansible/python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch.rpm differ diff --git a/bin/ansible/python-cffi-1.6.0-5.el7.aarch64.rpm b/bin/ansible/python-cffi-1.6.0-5.el7.aarch64.rpm new file mode 100644 index 0000000..4e99345 Binary files /dev/null and b/bin/ansible/python-cffi-1.6.0-5.el7.aarch64.rpm differ diff --git a/bin/ansible/python-enum34-1.0.4-1.el7.noarch.rpm b/bin/ansible/python-enum34-1.0.4-1.el7.noarch.rpm new file mode 100644 index 0000000..4d3c89a Binary files /dev/null and b/bin/ansible/python-enum34-1.0.4-1.el7.noarch.rpm differ diff --git a/bin/ansible/python-httplib2-0.9.2-1.el7.noarch.rpm b/bin/ansible/python-httplib2-0.9.2-1.el7.noarch.rpm new file mode 100644 index 0000000..65c0763 Binary files /dev/null and b/bin/ansible/python-httplib2-0.9.2-1.el7.noarch.rpm differ diff --git a/bin/ansible/python-idna-2.4-1.el7.noarch.rpm b/bin/ansible/python-idna-2.4-1.el7.noarch.rpm new file mode 100644 index 0000000..de64244 Binary files /dev/null and b/bin/ansible/python-idna-2.4-1.el7.noarch.rpm differ diff --git a/bin/ansible/python-ipaddress-1.0.16-2.el7.noarch.rpm b/bin/ansible/python-ipaddress-1.0.16-2.el7.noarch.rpm new file mode 100644 index 0000000..d3c7d2b Binary files /dev/null and b/bin/ansible/python-ipaddress-1.0.16-2.el7.noarch.rpm differ diff --git a/bin/ansible/python-jinja2-2.7.2-4.el7.noarch.rpm b/bin/ansible/python-jinja2-2.7.2-4.el7.noarch.rpm new file mode 100644 index 0000000..d7239c8 Binary files /dev/null and b/bin/ansible/python-jinja2-2.7.2-4.el7.noarch.rpm differ diff --git a/bin/ansible/python-markupsafe-0.11-10.el7.aarch64.rpm b/bin/ansible/python-markupsafe-0.11-10.el7.aarch64.rpm new file mode 100644 index 0000000..6f0f749 Binary files /dev/null and b/bin/ansible/python-markupsafe-0.11-10.el7.aarch64.rpm differ diff --git a/bin/ansible/python-paramiko-2.1.1-9.el7.noarch.rpm b/bin/ansible/python-paramiko-2.1.1-9.el7.noarch.rpm new file mode 100644 index 0000000..8c79030 Binary files /dev/null and b/bin/ansible/python-paramiko-2.1.1-9.el7.noarch.rpm differ diff --git a/bin/ansible/python-ply-3.4-11.el7.noarch.rpm b/bin/ansible/python-ply-3.4-11.el7.noarch.rpm new file mode 100644 index 0000000..1b607e3 Binary files /dev/null and b/bin/ansible/python-ply-3.4-11.el7.noarch.rpm differ diff --git a/bin/ansible/python-pycparser-2.14-1.el7.noarch.rpm b/bin/ansible/python-pycparser-2.14-1.el7.noarch.rpm new file mode 100644 index 0000000..dd2e6c9 Binary files /dev/null and b/bin/ansible/python-pycparser-2.14-1.el7.noarch.rpm differ diff --git a/bin/ansible/python-setuptools-0.9.8-7.el7.noarch.rpm b/bin/ansible/python-setuptools-0.9.8-7.el7.noarch.rpm new file mode 100644 index 0000000..7fa9685 Binary files /dev/null and b/bin/ansible/python-setuptools-0.9.8-7.el7.noarch.rpm differ diff --git a/bin/ansible/python-six-1.9.0-2.el7.noarch.rpm b/bin/ansible/python-six-1.9.0-2.el7.noarch.rpm new file mode 100644 index 0000000..d6f3ca5 Binary files /dev/null and b/bin/ansible/python-six-1.9.0-2.el7.noarch.rpm differ diff --git a/bin/ansible/python2-cryptography-1.7.2-2.el7.aarch64.rpm b/bin/ansible/python2-cryptography-1.7.2-2.el7.aarch64.rpm new file mode 100644 index 0000000..ef8c8bf Binary files /dev/null and b/bin/ansible/python2-cryptography-1.7.2-2.el7.aarch64.rpm differ diff --git a/bin/ansible/python2-jmespath-0.9.0-3.el7.noarch.rpm b/bin/ansible/python2-jmespath-0.9.0-3.el7.noarch.rpm new file mode 100644 index 0000000..8e09135 Binary files /dev/null and b/bin/ansible/python2-jmespath-0.9.0-3.el7.noarch.rpm differ diff --git a/bin/ansible/python2-pyasn1-0.1.9-7.el7.noarch.rpm b/bin/ansible/python2-pyasn1-0.1.9-7.el7.noarch.rpm new file mode 100644 index 0000000..7c4351a Binary files /dev/null and b/bin/ansible/python2-pyasn1-0.1.9-7.el7.noarch.rpm differ diff --git a/bin/ansible/sshpass-1.06-2.el7.aarch64.rpm b/bin/ansible/sshpass-1.06-2.el7.aarch64.rpm new file mode 100644 index 0000000..55e4396 Binary files /dev/null and b/bin/ansible/sshpass-1.06-2.el7.aarch64.rpm differ diff --git a/bin/calico/calicoctl b/bin/calico/calicoctl new file mode 100644 index 0000000..f626569 Binary files /dev/null and b/bin/calico/calicoctl differ diff --git a/bin/cfssl/cfssl b/bin/cfssl/cfssl new file mode 100755 index 0000000..24486ff Binary files /dev/null and b/bin/cfssl/cfssl differ diff --git a/bin/cfssl/cfssljson b/bin/cfssl/cfssljson new file mode 100755 index 0000000..e5b1919 Binary files /dev/null and b/bin/cfssl/cfssljson differ diff --git a/bin/cni/bridge b/bin/cni/bridge new file mode 100644 index 0000000..778b51e Binary files /dev/null and b/bin/cni/bridge differ diff --git a/bin/cni/calico-ipam b/bin/cni/calico-ipam new file mode 100644 index 0000000..1580a7f Binary files /dev/null and b/bin/cni/calico-ipam differ diff --git a/bin/cni/dhcp b/bin/cni/dhcp new file mode 100644 index 0000000..080d3ea Binary files /dev/null and b/bin/cni/dhcp differ diff --git a/bin/cni/flannel b/bin/cni/flannel new file mode 100644 index 0000000..b7bcfca Binary files /dev/null and b/bin/cni/flannel differ diff --git a/bin/cni/host-device b/bin/cni/host-device new file mode 100644 index 0000000..7e5128f Binary files /dev/null and b/bin/cni/host-device differ diff --git a/bin/cni/host-local b/bin/cni/host-local new file mode 100644 index 0000000..3a950e7 Binary files /dev/null and b/bin/cni/host-local differ diff --git a/bin/cni/ipvlan b/bin/cni/ipvlan new file mode 100644 index 0000000..b3e97fc Binary files /dev/null and b/bin/cni/ipvlan differ diff --git a/bin/cni/loopback b/bin/cni/loopback new file mode 100644 index 0000000..91e9386 Binary files /dev/null and b/bin/cni/loopback differ diff --git a/bin/cni/macvlan b/bin/cni/macvlan new file mode 100644 index 0000000..bdee03a Binary files /dev/null and b/bin/cni/macvlan differ diff --git a/bin/cni/portmap b/bin/cni/portmap new file mode 100644 index 0000000..677ee51 Binary files /dev/null and b/bin/cni/portmap differ diff --git a/bin/cni/ptp b/bin/cni/ptp new file mode 100644 index 0000000..bea856b Binary files /dev/null and b/bin/cni/ptp differ diff --git a/bin/cni/tuning b/bin/cni/tuning new file mode 100644 index 0000000..8a1549f Binary files /dev/null and b/bin/cni/tuning differ diff --git a/bin/cni/vlan b/bin/cni/vlan new file mode 100644 index 0000000..6f7c086 Binary files /dev/null and b/bin/cni/vlan differ diff --git a/bin/conntrack-tools/conntrack-tools-1.4.4-9.ky10.aarch64.rpm b/bin/conntrack-tools/conntrack-tools-1.4.4-9.ky10.aarch64.rpm new file mode 100644 index 0000000..a1bee44 Binary files /dev/null and b/bin/conntrack-tools/conntrack-tools-1.4.4-9.ky10.aarch64.rpm differ diff --git a/bin/conntrack-tools/libnetfilter_cthelper-1.0.0-15.ky10.aarch64.rpm b/bin/conntrack-tools/libnetfilter_cthelper-1.0.0-15.ky10.aarch64.rpm new file mode 100644 index 0000000..9cc9a29 Binary files /dev/null and b/bin/conntrack-tools/libnetfilter_cthelper-1.0.0-15.ky10.aarch64.rpm differ diff --git a/bin/conntrack-tools/libnetfilter_cttimeout-1.0.0-13.ky10.aarch64.rpm b/bin/conntrack-tools/libnetfilter_cttimeout-1.0.0-13.ky10.aarch64.rpm new file mode 100644 index 0000000..f33216f Binary files /dev/null and b/bin/conntrack-tools/libnetfilter_cttimeout-1.0.0-13.ky10.aarch64.rpm differ diff --git a/bin/conntrack-tools/libnetfilter_queue-1.0.2-13.ky10.aarch64.rpm b/bin/conntrack-tools/libnetfilter_queue-1.0.2-13.ky10.aarch64.rpm new file mode 100644 index 0000000..7b9e213 Binary files /dev/null and b/bin/conntrack-tools/libnetfilter_queue-1.0.2-13.ky10.aarch64.rpm differ diff --git a/bin/docker/containerd b/bin/docker/containerd new file mode 100755 index 0000000..d7bf9c8 Binary files /dev/null and b/bin/docker/containerd differ diff --git a/bin/docker/containerd-shim b/bin/docker/containerd-shim new file mode 100755 index 0000000..132c62d Binary files /dev/null and b/bin/docker/containerd-shim differ diff --git a/bin/docker/ctr b/bin/docker/ctr new file mode 100755 index 0000000..4b08592 Binary files /dev/null and b/bin/docker/ctr differ diff --git a/bin/docker/docker b/bin/docker/docker new file mode 100755 index 0000000..7bae52b Binary files /dev/null and b/bin/docker/docker differ diff --git a/bin/docker/docker-init b/bin/docker/docker-init new file mode 100755 index 0000000..2f25a24 Binary files /dev/null and b/bin/docker/docker-init differ diff --git a/bin/docker/docker-proxy b/bin/docker/docker-proxy new file mode 100755 index 0000000..10476b3 Binary files /dev/null and b/bin/docker/docker-proxy differ diff --git a/bin/docker/dockerd b/bin/docker/dockerd new file mode 100755 index 0000000..5070191 Binary files /dev/null and b/bin/docker/dockerd differ diff --git a/bin/docker/runc b/bin/docker/runc new file mode 100755 index 0000000..c2d0b0d Binary files /dev/null and b/bin/docker/runc differ diff --git a/bin/etcd/etcdctl b/bin/etcd/etcdctl new file mode 100644 index 0000000..944a2d2 Binary files /dev/null and b/bin/etcd/etcdctl differ diff --git a/bin/glusterfs/ding-libs-0.6.1-42.ky10.aarch64.rpm b/bin/glusterfs/ding-libs-0.6.1-42.ky10.aarch64.rpm new file mode 100644 index 0000000..a583e1a Binary files /dev/null and b/bin/glusterfs/ding-libs-0.6.1-42.ky10.aarch64.rpm differ diff --git a/bin/glusterfs/glusterfs-7.0-4.ky10.aarch64.rpm b/bin/glusterfs/glusterfs-7.0-4.ky10.aarch64.rpm new file mode 100644 index 0000000..0476a24 Binary files /dev/null and b/bin/glusterfs/glusterfs-7.0-4.ky10.aarch64.rpm differ diff --git a/bin/glusterfs/gssproxy-0.8.0-11.ky10.aarch64.rpm b/bin/glusterfs/gssproxy-0.8.0-11.ky10.aarch64.rpm new file mode 100644 index 0000000..8ca13ce Binary files /dev/null and b/bin/glusterfs/gssproxy-0.8.0-11.ky10.aarch64.rpm differ diff --git a/bin/glusterfs/krb5-1.17-9.ky10.aarch64.rpm b/bin/glusterfs/krb5-1.17-9.ky10.aarch64.rpm new file mode 100644 index 0000000..daba837 Binary files /dev/null and b/bin/glusterfs/krb5-1.17-9.ky10.aarch64.rpm differ diff --git a/bin/glusterfs/nfs-utils-2.4.2-2.ky10.aarch64.rpm b/bin/glusterfs/nfs-utils-2.4.2-2.ky10.aarch64.rpm new file mode 100644 index 0000000..d5ab039 Binary files /dev/null and b/bin/glusterfs/nfs-utils-2.4.2-2.ky10.aarch64.rpm differ diff --git a/bin/glusterfs/python3-gluster-7.0-4.ky10.aarch64.rpm b/bin/glusterfs/python3-gluster-7.0-4.ky10.aarch64.rpm new file mode 100644 index 0000000..b9a116e Binary files /dev/null and b/bin/glusterfs/python3-gluster-7.0-4.ky10.aarch64.rpm differ diff --git a/bin/glusterfs/python3-prettytable-0.7.2-18.ky10.noarch.rpm b/bin/glusterfs/python3-prettytable-0.7.2-18.ky10.noarch.rpm new file mode 100644 index 0000000..cc0bc0f Binary files /dev/null and b/bin/glusterfs/python3-prettytable-0.7.2-18.ky10.noarch.rpm differ diff --git a/bin/glusterfs/quota-4.05-1.ky10.aarch64.rpm b/bin/glusterfs/quota-4.05-1.ky10.aarch64.rpm new file mode 100644 index 0000000..e4961f1 Binary files /dev/null and b/bin/glusterfs/quota-4.05-1.ky10.aarch64.rpm differ diff --git a/bin/glusterfs/rdma-core-20.1-6.ky10.aarch64.rpm b/bin/glusterfs/rdma-core-20.1-6.ky10.aarch64.rpm new file mode 100644 index 0000000..55c8961 Binary files /dev/null and b/bin/glusterfs/rdma-core-20.1-6.ky10.aarch64.rpm differ diff --git a/bin/glusterfs/rpcbind-1.2.5-2.ky10.aarch64.rpm b/bin/glusterfs/rpcbind-1.2.5-2.ky10.aarch64.rpm new file mode 100644 index 0000000..e74fb3d Binary files /dev/null and b/bin/glusterfs/rpcbind-1.2.5-2.ky10.aarch64.rpm differ diff --git a/bin/kubernetes/kubectl b/bin/kubernetes/kubectl new file mode 100644 index 0000000..0211f9d Binary files /dev/null and b/bin/kubernetes/kubectl differ diff --git a/bin/kubernetes/kubelet b/bin/kubernetes/kubelet new file mode 100755 index 0000000..243dae5 Binary files /dev/null and b/bin/kubernetes/kubelet differ diff --git a/bin/kubernetes/kustomize b/bin/kubernetes/kustomize new file mode 100644 index 0000000..891d864 Binary files /dev/null and b/bin/kubernetes/kustomize differ diff --git a/bin/openvpn/openvpn-docker-image.tar b/bin/openvpn/openvpn-docker-image.tar new file mode 100644 index 0000000..b12b861 Binary files /dev/null and b/bin/openvpn/openvpn-docker-image.tar differ diff --git a/dockerfiles/ansible/Dockerfile b/dockerfiles/ansible/Dockerfile new file mode 100644 index 0000000..46584a3 --- /dev/null +++ b/dockerfiles/ansible/Dockerfile @@ -0,0 +1,9 @@ +FROM centos +RUN yum install centos-release-ansible-29 -y && yum install ansible openssh-server openssh-clients -y +RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N '' && ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N "" && ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N "" &&\ + ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa && cat /root/.ssh/id_rsa.pub >/root/.ssh/authorized_keys +RUN chmod +x -R /usr/bin/ + +ENV LANG C.UTF-8 + +RUN echo "/usr/sbin/sshd" >>/root/.bash_profile diff --git a/dockerfiles/es/es-index-clean/Dockerfile b/dockerfiles/es/es-index-clean/Dockerfile new file mode 100644 index 0000000..3da3703 --- /dev/null +++ b/dockerfiles/es/es-index-clean/Dockerfile @@ -0,0 +1,6 @@ +FROM centos:7 +MAINTAINER yangdong +WORKDIR /home/gwcloud +COPY es_index_clean_config clean.sh /home/gwcloud/ +RUN chmod +x /home/gwcloud/clean.sh +ENTRYPOINT sh /home/gwcloud/clean.sh diff --git a/dockerfiles/es/es-index-clean/es_index_clean_config b/dockerfiles/es/es-index-clean/es_index_clean_config new file mode 100644 index 0000000..c76705b --- /dev/null +++ b/dockerfiles/es/es-index-clean/es_index_clean_config @@ -0,0 +1,10 @@ +applog 3 +filebeat 7 +metricbeat 3 +elasticsearch 7 +gfs 3 +harbor 3 +k8s 3 +mongodb 3 +rabbitmq 3 +zipkin 3 diff --git a/example/hosts.m-masters.example b/example/hosts.m-masters.example new file mode 100644 index 0000000..9f8e162 --- /dev/null +++ b/example/hosts.m-masters.example @@ -0,0 +1,69 @@ +[deploy] +localhost + +#镜像仓库 +[registry] +10.2.1.212 + +#节点配置 +#管理节点 +[kube-master] +10.2.1.212 +10.2.1.170 +10.2.1.148 + +#计算节点 +[kube-compute-node] + +#存储节点 +[kube-storage-node] +10.2.1.212 +10.2.1.170 + +[kube-node:children] +kube-compute-node +kube-storage-node + +[kube-cluster:children] +kube-node +kube-master +lb + + +#代理节点 +#多管理节点时必须配置有代理节点 +#LB_ROLE：master、backup、lb_only，master\backup表示使用keepalived实现haproxy高可用,lb_only表示不使用keepalived +[lb] +10.2.1.212 +10.2.1.170 + +[elasticsearch] +10.2.1.212 +10.2.1.170 + +[mysql] +10.2.1.148 + +# 预留组，后续添加node节点使用 +[new-node] + +[kube-cluster:vars] +ansible_python_interpreter=/usr/bin/python3 + +[all:vars] +#集群 MASTER IP 如果单master部署，则该值为master ip，如果多master部署,则改值为lb的vip +MASTER_IP="10.2.1.164" + +#gfs磁盘设备名 +gfs_device=/dev/sdb + +zstackServerIP=10.1.1.9 +zstackServerAdminUser=admin +zstackServerAdminPassword=Greatwall@123 + +#默认区域 +defaultRegionCode=cn-cs-1 +defaultRegionName=长沙一区 + +#专有云访问域名 +gwcloudDomain=gwcloud.com diff --git a/group_vars/all b/group_vars/all new file mode 100644 index 0000000..3ff9526 --- /dev/null +++ b/group_vars/all @@ -0,0 +1,107 @@ +# ---------集群主要参数--------------- +MASTER_PORT: 6443 + +#TLS Bootstrapping 使用的 Token，使用 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 生成 +BOOTSTRAP_TOKEN: "0a3417df589e82d359ff6fc3d3c90fc0" + +# 集群网络插件，目前支持calico +CLUSTER_NETWORK: "calico" + +# 服务网段 (Service CIDR），部署前路由不可达，部署后集群内使用 IP:Port 可达 +SERVICE_CIDR: "10.68.0.0/16" + +# POD 网段 (Cluster CIDR），部署前路由不可达，**部署后**路由可达 +CLUSTER_CIDR: "172.20.0.0/16" + +# 服务端口范围 (NodePort Range) +NODE_PORT_RANGE: "10000-32767" + +# kubernetes 服务 IP (预分配，一般是 SERVICE_CIDR 中第一个IP) +CLUSTER_KUBERNETES_SVC_IP: "10.68.0.1" + +# 集群 DNS 服务 IP (从 SERVICE_CIDR 中预分配) +CLUSTER_DNS_SVC_IP: "10.68.0.2" + +# 集群 DNS 域名 +CLUSTER_DNS_DOMAIN: "cluster.local." + +#默认二进制文件目录 +bin_dir: "/opt/k8s/bin" + +#集群安装目录 +cluster_dir: "/etc/kubernetes" + +#证书目录 +ca_dir: "{{cluster_dir}}/ssl" + +#部署目录，即 ansible 工作目录 +base_dir: "/etc/ansible" + +#数据目录 +data_dir: "/data" + +# 基础镜像 +k8s_pod_infra_container_image: toyangdon/pause-arm64:3.4.1 + +#存储卷类型 +storage_type: glusterfs + +#默认主机名，要求唯一，默认规则：主机组名-主机ip最后一段 ，如果hosts文件中不是配置的ip(有可能是域名)，那么就直接取域名 +NODE_ID: >- + node-{{inventory_hostname.split('.')[3]}} + +#资源限制 +#生产配置 +#KUBE_RESERVED: "{'cpu':'200m','memory':'500Mi','ephemeral-storage':'1Gi'}" +system_reserved: >- + {'cpu': '{{ ansible_processor_vcpus * 1000 * 0.08 }}m','memory': '{{ansible_memtotal_mb * 0.05}}Mi','ephemeral-storage':'500Mi'} +kube_reserved: >- + {'cpu': '{{ansible_processor_vcpus * 1000 * 0.08}}m','memory': '{{ansible_memtotal_mb * 0.05}}Mi','ephemeral-storage':'500Mi'} +eviction_hard: >- + {'memory.available': '{{ansible_memtotal_mb * 0.05}}Mi','nodefs.available':'5%','imagefs.available':'15%','nodefs.inodesFree': '5%'} + +node_labels: >- + {%- if NODE_LABELS is defined -%}{{NODE_LABELS}},{%- endif -%} + {%- for group_name in group_names|reject("match","kube-cluster|kube-node") -%}node.kubernetes.io/{{group_name|replace("kube-","")}}=true{%- if not loop.last-%},{%- endif-%}{%- endfor -%} + +node_taints: "" + +#本地镜像仓库端口 +registry_port: 6550 + +BASE_IMAGE_URL: >- + {{ groups['registry'][0]}}:{{registry_port}} + +#docker insecure registry 如果有多个用逗号分开 +docker_insecure_registry: >- + {{BASE_IMAGE_URL}} + +#是否部署glusterfs +deploy_gfs: >- + {%- if groups['kube-storage-node']|length !=0 -%}true{%- else -%}false{%- endif -%} + +#是否部署监控告警 +deploy_monitor: true + +#是否离线 +deploy_offline: true + +#是否部署本地镜像仓库 +deploy_docker_registry: true + +#是否加载镜像 +load_images: true + +#是否推送镜像 +push_images: true + + +#工具镜像（证书） +gw_tools_image: toyangdon/gw-tools:1.1 + +#证书生成命令 +#cfssl_cmd: "{{bin_dir}}/docker run --rm -v {{ ca_dir }}:/workdir {{ gw_tools_image }} sh -c" +cfssl_cmd: "cd {{ ca_dir }} && export PATH=$PATH:{{bin_dir}} && sh -c" + +#专有云业务镜像仓库 +CLOUD_IMAGE_URL: "{{BASE_IMAGE_URL}}" diff --git a/group_vars/deploy b/group_vars/deploy new file mode 100644 index 0000000..e69de29 diff --git a/group_vars/elasticsearch b/group_vars/elasticsearch new file mode 100644 index 0000000..fb5f400 --- /dev/null +++ b/group_vars/elasticsearch @@ -0,0 +1,11 @@ +# es运行的jvm参数，两个值建议设置相同，并且为虚拟机内存的一半 +#es_jvm_xms_size: "-Xms6g" +#es_jvm_xmx_size: "-Xmx6g" + +es_storage_capacity: auto + +#es_cpu_limit: 4000m +#es_cpu_requests: 1000m + +#es_mem_limit: 7Gi +#es_mem_requests: 1Gi \ No newline at end of file diff --git a/group_vars/harbor b/group_vars/harbor new file mode 100644 index 0000000..afb7c63 --- /dev/null +++ b/group_vars/harbor @@ -0,0 +1,14 @@ +harbor_admin_password: Harbor12345 + +harbor_ext_endpoint: >- + {{ inventory_hostname }} + +images_src_path: >- + {{base_dir}}/images + +images_path: /data/tmp/images + +images_file_name: images.tar.gz + +images_file: >- + {{ images_path }}/{{ images_file_name }} \ No newline at end of file diff --git a/group_vars/kube-cluster b/group_vars/kube-cluster new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/group_vars/kube-cluster @@ -0,0 +1 @@ + diff --git a/group_vars/kube-master b/group_vars/kube-master new file mode 100644 index 0000000..9b6ec98 --- /dev/null +++ b/group_vars/kube-master @@ -0,0 +1,7 @@ +secure_port: 6444 + +#保留值 +system_reserved: >- + {'cpu': '500m','memory': '1024Mi','ephemeral-storage':'500Mi'} +kube_reserved: >- + {'cpu': '500m','memory': '1024Mi','ephemeral-storage':'500Mi'} diff --git a/group_vars/kube-node b/group_vars/kube-node new file mode 100644 index 0000000..e69de29 diff --git a/group_vars/kube-storage-node b/group_vars/kube-storage-node new file mode 100644 index 0000000..dc215dd --- /dev/null +++ b/group_vars/kube-storage-node @@ -0,0 +1,10 @@ +node_labels: >- + {%- if NODE_LABELS is defined -%}{{NODE_LABELS}},{%- endif -%} + {%- for group_name in group_names|reject("match","kube-cluster|kube-node") -%}node.kubernetes.io/{{group_name|replace("kube-","")}}=true{%- if not loop.last-%},{%- endif-%}{%- endfor -%} + {%- if groups["kube-storage-node"][0] == inventory_hostname -%},node.kubernetes.io/heketi=true{%- endif -%} + +#保留值 +system_reserved: >- + {'cpu': '500m','memory': '1024Mi','ephemeral-storage':'500Mi'} +kube_reserved: >- + {'cpu': '500m','memory': '1024Mi','ephemeral-storage':'500Mi'} diff --git a/group_vars/lb b/group_vars/lb new file mode 100644 index 0000000..0717a41 --- /dev/null +++ b/group_vars/lb @@ -0,0 +1,17 @@ +# 区分多个instance的VRRP组播，同网段不能重复，取值在0-255之间 +VIRTUAL_ROUTER_ID: >- + {{MASTER_IP.split('.')[3]}} + +#lb节点部署方式 process/container +lb_type: container + +LB_ROLE: >- + {%- if groups["lb"][0] == inventory_hostname -%}master + {%- else -%}backup + {%- endif -%} + +#保留值 +system_reserved: >- + {'cpu': '500m','memory': '1024Mi','ephemeral-storage':'500Mi'} +kube_reserved: >- + {'cpu': '500m','memory': '1024Mi','ephemeral-storage':'500Mi'} \ No newline at end of file diff --git a/group_vars/logstash b/group_vars/logstash new file mode 100644 index 0000000..05a2076 --- /dev/null +++ b/group_vars/logstash @@ -0,0 +1,10 @@ +#logstash_jvm_xms_size: -Xms6g +#logstash_jvm_xmx_size: -Xmx6g + +logstash_type: container + +#logstash_cpu_limit: 4000m +#logstash_cpu_requests: 1000m + +#logstash_mem_limit: 7Gi +#logstash_mem_requests: 1Gi \ No newline at end of file diff --git a/group_vars/new-node b/group_vars/new-node new file mode 100644 index 0000000..14d0c5d --- /dev/null +++ b/group_vars/new-node @@ -0,0 +1,9 @@ +node_labels: >- + {%- if NODE_LABELS is defined -%}{{NODE_LABELS}},{%- endif -%}node.kubernetes.io/kube-compute-node=true + +node_taints: >- + {%- if NODE_TAINTS is defined -%}{{NODE_TAINTS}}{%- endif -%} + +#默认主机名，要求唯一，默认规则：主机组名-主机ip最后一段 ，如果hosts文件中不是配置的ip(有可能是域名)，那么就直接取域名 +NODE_ID: >- + kube-compute-node-{{inventory_hostname.split('.')[3]}} \ No newline at end of file diff --git a/group_vars/registry b/group_vars/registry new file mode 100644 index 0000000..5502ee5 --- /dev/null +++ b/group_vars/registry @@ -0,0 +1,9 @@ +images_src_path: >- + {{base_dir}}/images + +images_path: /data/tmp/images + +images_file_name: images.tar.gz + +images_file: >- + {{ images_path }}/{{ images_file_name }} diff --git a/hosts b/hosts new file mode 100644 index 0000000..481b6e7 --- /dev/null +++ b/hosts @@ -0,0 +1,82 @@ +[deploy] +localhost + +#镜像仓库 +[registry] +10.2.1.212 + +#节点配置 +#管理节点 +[kube-master] +10.2.1.212 +10.2.1.170 +10.2.1.148 + +#计算节点 +[kube-compute-node] + +#存储节点 +[kube-storage-node] +10.2.1.212 +10.2.1.170 + +[kube-node:children] +kube-compute-node +kube-storage-node + +[kube-cluster:children] +kube-node +kube-master +lb + + +#代理节点 +#多管理节点时必须配置有代理节点 +#LB_ROLE：master、backup、lb_only，master\backup表示使用keepalived实现haproxy高可用,lb_only表示不使用keepalived +[lb] +10.2.1.212 +10.2.1.170 + +[elasticsearch] +10.2.1.212 +10.2.1.170 + +[mysql] +10.2.1.148 +10.2.1.170 + +#openvpn服务，请另外创建一台虚拟机（4c8g）作为openvpn服务器，该虚拟机可用于暴露私有云内部网络，openvpnExternalIp表示服务器的外部IP，openvpnRouteCIDR表示需求经过vpn路由的ip范围 +[openvpn] +10.2.1.224 openvpnExternalIp=10.2.1.224 openvpnRouteCIDR=10.2.1.0/16 ansible_ssh_user=root ansible_ssh_pass=Greatwall@123 + +# 预留组，后续添加node节点使用 +[new-node] + +[kube-cluster:vars] +ansible_python_interpreter=/usr/bin/python3 + +[all:vars] +#集群 MASTER IP 如果单master部署，则该值为master ip，如果多master部署,则改值为lb的vip +MASTER_IP="10.2.1.164" + +#gfs磁盘设备名 +gfs_device=/dev/sdb + +zstackServerIP=10.1.1.9 +zstackServerAdminUser=admin +zstackServerAdminPassword=Greatwall@123 + +#默认区域 +defaultRegionCode=cn-cs-1 +defaultRegionName=长沙一区 + +#专有云访问域名 +gwcloudDomain=gwcloud.com + +#专有云告警邮箱服务器 +monitorEmailEnable=true +monitorEmailHost=mail.greatwall.com.cn +monitorEmailPort=465 +monitorEmailUsername=hnck +monitorEmailFrom=hnck@greatwall.com.cn +monitorEmailPassword=Eghucn-91406 diff --git a/package/Jenkinsfile_for_package b/package/Jenkinsfile_for_package new file mode 100644 index 0000000..01903a2 --- /dev/null +++ b/package/Jenkinsfile_for_package @@ -0,0 +1,81 @@ +def projectProperties = [ + [$class: 'BuildDiscarderProperty', strategy: [$class: 'LogRotator', numToKeepStr: '5']], + parameters([ + string(name: 'BRANCH_NAME', defaultValue: 'master', description: '代码分支'), + string(name: 'CLEAN_FLAG', defaultValue: "false", description: '是否清理工作空间'), + string(name: 'VERSION', defaultValue: "1.0", description: '包版本'), + string(name: 'GENERATE_IMAGES_LISTS', defaultValue: "true", description: '是否生成images_list文件'), + string(name: 'INCLUDE_APPS_IMAGES', defaultValue: "true", description: '是否包含应用镜像'), + string(name: 'DEST_PATH_PREFIX', defaultValue: "/var/jenkins_home", description: '打包文件保存目录'), + string(name: 'HARBOR_SRC', defaultValue: 'docker.io', description: '源镜像仓库') + ]) +] + +properties(projectProperties) + +node('master') { + + //部署文件路径 + DEPLOY_PROJECT_PATH="${WORKSPACE}/k8s_deploy"; + + //镜像文件保存目录 + IMAGES_PATH="${DEPLOY_PROJECT_PATH}/images" + + //基础镜像文件列表 + IMAGES_LIST="${DEPLOY_PROJECT_PATH}/tools/images/images_list" + + //应用商店镜像文件列表 + APPS_IMAGES_LIST="${DEPLOY_PROJECT_PATH}/tools/images/apps_images_list" + + //打包文件保存目录 + DEST_PATH="${DEST_PATH_PREFIX}/k8s_data/${VERSION}" + + //源harobr镜像库地址 + HARBOR_SRC="docker.io" + + + stage('拉取k8s部署文件') { + if("${CLEAN_FLAG}" == "true"){ + cleanWs() + } + dir(DEPLOY_PROJECT_PATH){ + git branch: "${BRANCH_NAME}", url: 'https://git.trustie.net/toyangdon/proprietary-cloud-deploy.git' + } + } + + if( "${GENERATE_IMAGES_LISTS}" == "true" ){ + stage('生成基础镜像列表'){ + sh "grep '_image:' -hr ${DEPLOY_PROJECT_PATH}/roles ${DEPLOY_PROJECT_PATH}/group_vars|cut -d':' -f2,3|sed 's/\"//g'|sed 's/ //g' |sort -u >${IMAGES_LIST} " + } + } + + stage('拉取并打包基础镜像') { + sh "docker login -u toyangdon -pyd880309" + sh "apt install pigz -y" + if( "${INCLUDE_APPS_IMAGES}" == "true" ){ + sh "cat ${APPS_IMAGES_LIST} >> ${IMAGES_LIST}" + sh "rm -f ${APPS_IMAGES_LIST}" + } + sh "sh ${DEPLOY_PROJECT_PATH}/tools/images/images_batch_save.sh ${IMAGES_PATH} ${HARBOR_SRC} ${IMAGES_LIST}" + + } + + stage('拉取并导出ansible镜像') { + sh "docker pull toyangdon/ansible:latest" + sh "docker save toyangdon/ansible:latest -o k8s_deploy/ansible_image.tar" + } + + + + stage('打包部署文件') { + sh "rm -rf k8s_deploy/.git" + sh "tar --use-compress-program=pigz -cvpf k8s_deploy.tar.gz k8s_deploy" + } + + stage('归档部署文件'){ + //根据版本号创建目录，如果目录已经存在则报错退出 + sh "mkdir -p ${DEST_PATH}" + sh "cp k8s_deploy.tar.gz ${DEST_PATH}/" + } +} + diff --git a/package/image_push/image_list b/package/image_push/image_list new file mode 100644 index 0000000..a5d49dc --- /dev/null +++ b/package/image_push/image_list @@ -0,0 +1,8 @@ +dev-docker-registry.ccyunchina.com/uat/system-management:v0.3.beta-20210907.1467.45 toyangdon/system-management:v0.3 +dev-docker-registry.ccyunchina.com/uat/cloud-service:v0.3.beta-20210907.1460.31 toyangdon/cloud-service:v0.3 +dev-docker-registry.ccyunchina.com/uat/trade-service:v0.3.beta-20210907.1460.43 toyangdon/trade-service:v0.3 +dev-docker-registry.ccyunchina.com/uat/file-service:v0.3.beta-20210907.496.20 toyangdon/file-service:v0.3 +dev-docker-registry.ccyunchina.com/uat/cloud-base:v0.3.beta-20210907.1462.55 toyangdon/cloud-base:v0.3 +dev-docker-registry.ccyunchina.com/uat/vue-cloud:v0.3.beta-20210907.1414.63 toyangdon/vue-cloud:v0.3 +dev-docker-registry.ccyunchina.com/uat/vue-admin:v0.3.beta-20210907.1415.70 toyangdon/vue-admin:v0.3 +dev-docker-registry.ccyunchina.com/dev/activiti-service:v0.3.beta-20210907.1470.16 toyangdon/activiti-service:v0.3 diff --git a/package/image_push/push.sh b/package/image_push/push.sh new file mode 100755 index 0000000..967427c --- /dev/null +++ b/package/image_push/push.sh @@ -0,0 +1,36 @@ +#!/bin/bash + +set -x + +image_list_file=image_list +#docker_registry_desc=dockerhub.ccyunchina.com +#docker_registry_desc=registry.cn-hangzhou.aliyuncs.com +docker_registry_desc=docker.io + +#docker login $docker_registry_desc -u 153307747@qq.com -pyd880309 +docker login $docker_registry_desc -u toyangdon -pyd880309 +if [ $? != 0 ];then + echo "error:docker registry login fail! please check username or password is correct" + exit 1 + fi + +while read line +do + src_image=`echo $line |awk '{print $1}'` + desc_image=${docker_registry_desc}/`echo $line |awk '{print $2}'` + docker pull $src_image + if [ $? != 0 ];then + echo "error:pull image ${src_image} fail! please check image name is correct" + exit 1 + fi + docker tag $src_image $desc_image + docker push $desc_image + if [ $? != 0 ];then + echo "error:push image ${image} fail!" + exit 1 + fi + echo "------------------$name push success!-------------" +done < $image_list_file +docker logout + +echo "------------------all images push success-------------" diff --git a/package/jenkins.yml b/package/jenkins.yml new file mode 100644 index 0000000..8bcc960 --- /dev/null +++ b/package/jenkins.yml @@ -0,0 +1,126 @@ +apiVersion: v1 +kind: PersistentVolume +metadata: + name: jenkins +spec: + accessModes: + - ReadWriteMany + capacity: + storage: 30Gi + local: + path: /data/jenkins + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: + - "kube-compute-node-240" + persistentVolumeReclaimPolicy: Delete + storageClassName: local-storage +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: jenkins +spec: + accessModes: + - ReadWriteMany + resources: + requests: + storage: 30Gi + volumeMode: Filesystem + storageClassName: local-storage +--- +# Source: jenkins/templates/svc.yaml +apiVersion: v1 +kind: Service +metadata: + name: jenkins + labels: + app: jenkins +spec: + type: LoadBalancer + externalTrafficPolicy: "Cluster" + ports: + - name: http + port: 80 + targetPort: http + - name: https + port: 443 + targetPort: https + selector: + app: jenkins +--- +# Source: jenkins/templates/deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: jenkins + labels: + app: jenkins +spec: + selector: + matchLabels: + app: jenkins + template: + metadata: + labels: + app: jenkins + spec: + containers: + - name: jenkins + image: calmmopey/jenkins-arm64-2.277.4:fixed + env: + - name: JAVA_OPTS + value: "-Xmx3072m -Xms512m -XX:ErrorFile=/var/jenkins_home/hs_err_pid.log" + ports: + - name: http + containerPort: 8080 + - name: https + containerPort: 8443 + livenessProbe: + httpGet: + path: /login + port: http + initialDelaySeconds: 300 + periodSeconds: 10 + timeoutSeconds: 5 + successThreshold: 1 + failureThreshold: 10 + readinessProbe: + httpGet: + path: /login + port: http + initialDelaySeconds: 30 + periodSeconds: 5 + timeoutSeconds: 3 + successThreshold: 1 + failureThreshold: 3 + resources: + limits: + cpu: 2 + memory: 4Gi + requests: + cpu: 300m + memory: 512Mi + volumeMounts: + - name: jenkins-data + mountPath: /var/jenkins_home + - name: docker-sock + mountPath: /var/run/docker.sock + - name: docker + mountPath: /bin/docker + volumes: + - name: jenkins-data + persistentVolumeClaim: + claimName: jenkins + - name: docker-sock + hostPath: + path: /var/run/docker.sock + type: '' + - name: docker + hostPath: + path: /opt/k8s/bin/docker + type: '' diff --git a/playbooks/gpaas/90.setup.yml b/playbooks/gpaas/90.setup.yml new file mode 100644 index 0000000..b0e44a6 --- /dev/null +++ b/playbooks/gpaas/90.setup.yml @@ -0,0 +1,3 @@ +- import_playbook: efk/90.setup.yml + +- import_playbook: monitor/90.setup.yml diff --git a/playbooks/gpaas/efk/01.es.yml b/playbooks/gpaas/efk/01.es.yml new file mode 100644 index 0000000..061fce7 --- /dev/null +++ b/playbooks/gpaas/efk/01.es.yml @@ -0,0 +1,4 @@ +- hosts: + - elasticsearch + roles: + - gpaas/efk/elasticsearch diff --git a/playbooks/gpaas/efk/03.fluentd.yml b/playbooks/gpaas/efk/03.fluentd.yml new file mode 100644 index 0000000..f293297 --- /dev/null +++ b/playbooks/gpaas/efk/03.fluentd.yml @@ -0,0 +1,4 @@ +# 插件部署 +- hosts: deploy + roles: + - gpaas/efk/fluentd diff --git a/playbooks/gpaas/efk/04.kibana.yml b/playbooks/gpaas/efk/04.kibana.yml new file mode 100644 index 0000000..042da06 --- /dev/null +++ b/playbooks/gpaas/efk/04.kibana.yml @@ -0,0 +1,4 @@ +# 插件部署 +- hosts: deploy + roles: + - gpaas/efk/kibana diff --git a/playbooks/gpaas/efk/90.setup.yml b/playbooks/gpaas/efk/90.setup.yml new file mode 100644 index 0000000..72c00d8 --- /dev/null +++ b/playbooks/gpaas/efk/90.setup.yml @@ -0,0 +1,7 @@ +- import_playbook: 01.es.yml + +#- import_playbook: 02.logstash.yml + +- import_playbook: 03.fluentd.yml + +- import_playbook: 04.kibana.yml diff --git a/playbooks/gpaas/monitor/01.prometheus.yml b/playbooks/gpaas/monitor/01.prometheus.yml new file mode 100644 index 0000000..39193b0 --- /dev/null +++ b/playbooks/gpaas/monitor/01.prometheus.yml @@ -0,0 +1,4 @@ +# 插件部署 +- hosts: deploy + roles: + - gpaas/monitor/metricbeat \ No newline at end of file diff --git a/playbooks/gpaas/monitor/02.metricbeat.yml b/playbooks/gpaas/monitor/02.metricbeat.yml new file mode 100644 index 0000000..39193b0 --- /dev/null +++ b/playbooks/gpaas/monitor/02.metricbeat.yml @@ -0,0 +1,4 @@ +# 插件部署 +- hosts: deploy + roles: + - gpaas/monitor/metricbeat \ No newline at end of file diff --git a/playbooks/gpaas/monitor/03.grafana.yml b/playbooks/gpaas/monitor/03.grafana.yml new file mode 100644 index 0000000..bbd8d06 --- /dev/null +++ b/playbooks/gpaas/monitor/03.grafana.yml @@ -0,0 +1,4 @@ +# 插件部署 +- hosts: deploy + roles: + - gpaas/monitor/grafana \ No newline at end of file diff --git a/playbooks/gpaas/monitor/04.alert.yml b/playbooks/gpaas/monitor/04.alert.yml new file mode 100644 index 0000000..4a596a7 --- /dev/null +++ b/playbooks/gpaas/monitor/04.alert.yml @@ -0,0 +1,4 @@ +# 插件部署 +- hosts: deploy + roles: + - gpaas/monitor/alert \ No newline at end of file diff --git a/playbooks/gpaas/monitor/05.inspect.yml b/playbooks/gpaas/monitor/05.inspect.yml new file mode 100644 index 0000000..c51bf37 --- /dev/null +++ b/playbooks/gpaas/monitor/05.inspect.yml @@ -0,0 +1,4 @@ +# 插件部署 +- hosts: deploy + roles: + - gpaas/monitor/inspect \ No newline at end of file diff --git a/playbooks/gpaas/monitor/06.push.yml b/playbooks/gpaas/monitor/06.push.yml new file mode 100644 index 0000000..07312c3 --- /dev/null +++ b/playbooks/gpaas/monitor/06.push.yml @@ -0,0 +1,4 @@ +# 插件部署 +- hosts: deploy + roles: + - gpaas/monitor/push \ No newline at end of file diff --git a/playbooks/gpaas/monitor/90.setup.yml b/playbooks/gpaas/monitor/90.setup.yml new file mode 100644 index 0000000..1439ec0 --- /dev/null +++ b/playbooks/gpaas/monitor/90.setup.yml @@ -0,0 +1,14 @@ +#- import_playbook: ../elk/01.es.yml + # when: deploy_es == 'true' + +#- import_playbook: 02.metricbeat.yml + +#- import_playbook: 03.grafana.yml + +#- import_playbook: 04.alert.yml + +#- import_playbook: 05.inspect.yml + +#- import_playbook: 06.push.yml + +- import_playbook: monitor.yml diff --git a/playbooks/gpaas/monitor/monitor.yml b/playbooks/gpaas/monitor/monitor.yml new file mode 100644 index 0000000..de1a2c4 --- /dev/null +++ b/playbooks/gpaas/monitor/monitor.yml @@ -0,0 +1,4 @@ +# 插件部署 +- hosts: deploy + roles: + - gpaas/monitor/prometheus diff --git a/playbooks/kubernetes/00.check.yml b/playbooks/kubernetes/00.check.yml new file mode 100644 index 0000000..6051fe2 --- /dev/null +++ b/playbooks/kubernetes/00.check.yml @@ -0,0 +1,16 @@ +# 所有主机进行基础环境配置 +#- hosts: all +# gather_facts: yes + +- hosts: + - kube-cluster + - openvpn + roles: + - kubernetes/os-env + +- hosts: kube-storage-node + tasks: + - name: 检测存储节点中是否存在指定的磁盘设备 + fail: + msg: "存储节点中找不到指定的磁盘设备" + when: 'ansible_devices[gfs_device|replace("/dev/","")] is not defined' diff --git a/playbooks/kubernetes/01.docker.yml b/playbooks/kubernetes/01.docker.yml new file mode 100644 index 0000000..93ce6fb --- /dev/null +++ b/playbooks/kubernetes/01.docker.yml @@ -0,0 +1,5 @@ +- hosts: + - kube-cluster + - openvpn + roles: + - kubernetes/docker diff --git a/playbooks/kubernetes/02.prepare.yml b/playbooks/kubernetes/02.prepare.yml new file mode 100644 index 0000000..74210d8 --- /dev/null +++ b/playbooks/kubernetes/02.prepare.yml @@ -0,0 +1,11 @@ +- hosts: deploy + roles: +# - { role: kubernetes/registry , vars: { stage : "load"} ,when: deploy_docker_registry } + - kubernetes/deploy + +# 集群节点的公共配置任务 +- hosts: + - kube-cluster + - deploy + roles: + - kubernetes/prepare diff --git a/playbooks/kubernetes/03.registry.yml b/playbooks/kubernetes/03.registry.yml new file mode 100644 index 0000000..4d7d479 --- /dev/null +++ b/playbooks/kubernetes/03.registry.yml @@ -0,0 +1,6 @@ +- hosts: registry + roles: + - { role: kubernetes/registry , vars: { stage : "load"} ,when: deploy_docker_registry } + - { role: kubernetes/registry , vars: { stage : "deploy"} } + - { role: kubernetes/kube-node } + - { role: kubernetes/registry , vars: { stage : "push"} } diff --git a/playbooks/kubernetes/04.lb.yml b/playbooks/kubernetes/04.lb.yml new file mode 100644 index 0000000..de7b2c4 --- /dev/null +++ b/playbooks/kubernetes/04.lb.yml @@ -0,0 +1,7 @@ +# [可选]多master部署时的负载均衡配置 +#- hosts: lb +# gather_facts: yes + +- hosts: lb + roles: + - kubernetes/lb diff --git a/playbooks/kubernetes/05.kube-master.yml b/playbooks/kubernetes/05.kube-master.yml new file mode 100644 index 0000000..3326be1 --- /dev/null +++ b/playbooks/kubernetes/05.kube-master.yml @@ -0,0 +1,6 @@ +#- hosts: kube-master +# gather_facts: yes + +- hosts: kube-master + roles: + - kubernetes/kube-master diff --git a/playbooks/kubernetes/06.kube-node.yml b/playbooks/kubernetes/06.kube-node.yml new file mode 100644 index 0000000..c446ea6 --- /dev/null +++ b/playbooks/kubernetes/06.kube-node.yml @@ -0,0 +1,21 @@ +#- hosts: +# - lb +# - kube-master +# - kube-node +# gather_facts: yes + +#先lb节点 +#- hosts: +# - lb +# roles: +# - kubernetes/kube-node +#再master节点 +- hosts: + - kube-master + roles: + - kubernetes/kube-node +#最后计算节点 +#- hosts: +# - kube-node +# roles: +# - kubernetes/kube-node diff --git a/playbooks/kubernetes/07.calico.yml b/playbooks/kubernetes/07.calico.yml new file mode 100644 index 0000000..4c11257 --- /dev/null +++ b/playbooks/kubernetes/07.calico.yml @@ -0,0 +1,3 @@ +- hosts: kube-cluster + roles: + - kubernetes/calico diff --git a/playbooks/kubernetes/07.flannel.yml b/playbooks/kubernetes/07.flannel.yml new file mode 100644 index 0000000..021e8b3 --- /dev/null +++ b/playbooks/kubernetes/07.flannel.yml @@ -0,0 +1,3 @@ +- hosts: kube-cluster + roles: + - kubernetes/flannel diff --git a/playbooks/kubernetes/09.storage-nfs.yml b/playbooks/kubernetes/09.storage-nfs.yml new file mode 100644 index 0000000..659aaee --- /dev/null +++ b/playbooks/kubernetes/09.storage-nfs.yml @@ -0,0 +1,12 @@ +- hosts: storage-nfs-node + roles: + - kubernetes/storage-nfs + +- hosts: deploy + tasks: + - name: 创建 deplyment-nfs.yaml文件 + template: src={{ base_dir }}/roles/storage-nfs/templates/deployment-nfs.yaml dest={{ base_dir }}/manifests/deployment-nfs.yaml + + - name: 部署 deplyment-nfs.yaml + shell: "{{bin_dir}}/kubectl apply -f {{ base_dir }}/manifests/deployment-nfs.yaml" + diff --git a/playbooks/kubernetes/10.storage-gluster.yml b/playbooks/kubernetes/10.storage-gluster.yml new file mode 100644 index 0000000..ffec0a3 --- /dev/null +++ b/playbooks/kubernetes/10.storage-gluster.yml @@ -0,0 +1,7 @@ +#- hosts: kube-storage-node +# gather_facts: yes + +- hosts: kube-storage-node + roles: + - kubernetes/storage-glusterfs + diff --git a/playbooks/kubernetes/20.addnode.yml b/playbooks/kubernetes/20.addnode.yml new file mode 100644 index 0000000..9d6b5ab --- /dev/null +++ b/playbooks/kubernetes/20.addnode.yml @@ -0,0 +1,16 @@ +#TO_DO 改成import 其它playbook + +- hosts: new-node + gather_facts: yes + roles: + - kubernetes/os-env + - kubernetes/prepare + - kubernetes/docker + - { role: kubernetes/calico, when: "CLUSTER_NETWORK == 'calico'" } + - { role: kubernetes/flannel, when: "CLUSTER_NETWORK == 'flannel'" } + - { role: kubernetes/kube-node , node_role: compute } +- hosts: deploy + tasks: + - name: 批准新增node节点 + shell: "sleep 15 && {{ bin_dir }}/kubectl get csr|grep 'Pending' | awk 'NR>0{print $1}'| xargs {{ bin_dir }}/kubectl certificate approve" + ignore_errors: true diff --git a/playbooks/kubernetes/30.addons.yml b/playbooks/kubernetes/30.addons.yml new file mode 100644 index 0000000..7665f1c --- /dev/null +++ b/playbooks/kubernetes/30.addons.yml @@ -0,0 +1,6 @@ +- hosts: kube-storage-node + gather_facts: yes + +- hosts: deploy + roles: + - kubernetes/addons diff --git a/playbooks/kubernetes/40.upgrade.yml b/playbooks/kubernetes/40.upgrade.yml new file mode 100644 index 0000000..a86bc99 --- /dev/null +++ b/playbooks/kubernetes/40.upgrade.yml @@ -0,0 +1,90 @@ +- hosts: deploy + tags: + - addons + tasks: + - name: 升级kubectl二进制文件 + copy: src={{ base_dir }}/bin/kubectl dest={{ bin_dir }}/kubectl mode=075 + - name: 升级calico yaml + template: src={{base_dir}}/roles/addons/templates/calico.yaml dest={{ base_dir }}/manifests/calico.yaml + when: CLUSTER_NETWORK == "calico" + - name: 升级calico + shell: '{{bin_dir}}/kubectl apply -f {{ base_dir }}/manifests/calico.yaml' + when: CLUSTER_NETWORK == "calico" + - name: 触发calico更新 + shell: '{{bin_dir}}/kubectl patch daemonset calico-node -n kube-system -p "{\"spec\":{\"updateStrategy\":{\"type\":\"RollingUpdate\"}}}"' + when: CLUSTER_NETWORK == "calico" + ignore_errors: true + - name: 升级 kubedns yaml文件 + template: src={{base_dir}}/roles/addons/templates/kubedns.yaml dest={{ base_dir }}/manifests/kubedns.yaml + - name: 升级 kubedns + shell: "{{bin_dir}}/kubectl apply -f {{ base_dir }}/manifests/kubedns.yaml" + - name: 升级 kubernetes-dashboard yaml文件 + template: src={{base_dir}}/roles/addons/templates/kubernetes-dashboard.yaml dest={{ base_dir }}/manifests/kubernetes-dashboard.yaml + - name: 升级 kubernetes-dashboard + shell: "{{bin_dir}}/kubectl apply -f {{ base_dir }}/manifests/kubernetes-dashboard.yaml" + +- hosts: kube-cluster + tasks: + - name: 升级kube-node二进制文件 + copy: src={{ base_dir }}/bin/{{ item }} dest={{ bin_dir }}/{{ item }} mode=0755 + with_items: + - kubectl + - kube-proxy + - kubelet + +- hosts: kube-master + tags: + - master + tasks: + - name: 备份etcd数据 + shell: export ETCDCTL_API=3;{{bin_dir}}/etcdctl snapshot save {{data_dir}}/etcd/backup_`date +%Y%m%d.%s`.db + tags: + - etcd + - name: 更新kube-apiserver.yml文件 + template: src={{base_dir}}/roles/kube-master/templates/apiserver.yml.conf.j2 dest=/etc/kubernetes/manifests/kube-apiserver.yml + when: master_prepare is not defined + - name: 更新kube-controller-manager.yml文件 + template: src={{base_dir}}/roles/kube-master/templates/manager.yml.conf.j2 dest=/etc/kubernetes/manifests/kube-controller-manager.yml + when: master_prepare is not defined + - name: 更新kube-scheduler.yml文件 + template: src={{base_dir}}/roles/kube-master/templates/scheduler.yml.conf.j2 dest=/etc/kubernetes/manifests/kube-scheduler.yml + when: master_prepare is not defined + - name: 更新etcd.yml文件 + template: src={{base_dir}}/roles/kube-master/templates/etcd.yml.conf.j2 dest=/etc/kubernetes/manifests/etcd.yml + when: master_prepare is not defined + tags: + - etcd + - name: 重启kubelet、kubeproxy + shell: systemctl restart kubelet kube-proxy + tags: + - restart + +- hosts: kube-node + tags: + - node + tasks: + - name: 重启kubelet、kubeproxy + shell: systemctl restart kubelet kube-proxy + tags: + - restart + +- hosts: kube-master + tasks: + - name: 创建目录 + file: name=/etc/kubernetes/update_prepare state=directory + - name: 准备kube-apiserver.yml文件 + template: src={{base_dir}}/roles/kube-master/templates/apiserver.yml.conf.j2 dest=/etc/kubernetes/update_prepare/kube-apiserver.yml + when: master_prepare is defined + - name: 准备kube-controller-manager.yml文件 + template: src={{base_dir}}/roles/kube-master/templates/manager.yml.conf.j2 dest=/etc/kubernetes/update_prepare/kube-controller-manager.yml + when: master_prepare is defined + - name: 准备kube-scheduler.yml文件 + template: src={{base_dir}}/roles/kube-master/templates/scheduler.yml.conf.j2 dest=/etc/kubernetes/update_prepare/kube-scheduler.yml + when: master_prepare is defined + - name: 准备etcd.yml文件 + template: src={{base_dir}}/roles/kube-master/templates/etcd.yml.conf.j2 dest=/etc/kubernetes/update_prepare/etcd.yml + when: master_prepare is defined + tags: + - etcd + tags: + - master_prepare diff --git a/playbooks/kubernetes/90.setup.yml b/playbooks/kubernetes/90.setup.yml new file mode 100644 index 0000000..d9c041c --- /dev/null +++ b/playbooks/kubernetes/90.setup.yml @@ -0,0 +1,28 @@ +- import_playbook: 00.check.yml + +- import_playbook: 01.docker.yml + +- import_playbook: 02.prepare.yml + +- import_playbook: 03.registry.yml + when: deploy_docker_registry + +- import_playbook: 04.lb.yml + +- import_playbook: 05.kube-master.yml + +- import_playbook: 06.kube-node.yml + +- import_playbook: 07.calico.yml + when: "CLUSTER_NETWORK == 'calico'" + +- import_playbook: 07.flannel.yml + when: "CLUSTER_NETWORK == 'flannel'" + +#- import_playbook: 09.storage-nfs.yml +# when: "storage_type == 'nfs'" + +#- import_playbook: 10.storage-gluster.yml +# when: "storage_type == 'glusterfs'" + +- import_playbook: 30.addons.yml diff --git a/playbooks/kubernetes/99.clean.yml b/playbooks/kubernetes/99.clean.yml new file mode 100644 index 0000000..6944a61 --- /dev/null +++ b/playbooks/kubernetes/99.clean.yml @@ -0,0 +1,141 @@ +# 警告：此脚本将清理真个K8S集群，包括所有POD、ETCD数据等 +# 请三思后运行此脚本 ansible-playbook 99.clean.yml + +- hosts: + - kube-cluster + - lb + - elasticsearch + - logstash + - harbor + tasks: + - name: stop kube-node service + shell: "systemctl stop kubelet" + ignore_errors: true + + - name: clean docker container + shell: "{{bin_dir}}/docker stop $({{bin_dir}}/docker ps -q) && {{bin_dir}}/docker rm $({{bin_dir}}/docker ps -aq)" + ignore_errors: true + + - name: stop docker service + shell: "systemctl stop docker" + ignore_errors: true + + - name: umount kubelet 挂载的目录 + shell: "mount | grep '/data/kubelet'| awk '{print $3}'|xargs -r umount" + ignore_errors: true + + - name: 清理目录和文件 + file: name={{ item }} state=absent + with_items: + - "{{data_dir}}/kubelet/" + - "/etc/kubernetes/" + - "/etc/systemd/system/kubelet.service" + - "/etc/systemd/system/kube-proxy.service" + - "{{bin_dir}}/" + - "/data/kubelet/" + - "/data/kube-proxy/" + - "/data/log/" + - "/data/mysql/" + - "/data/es" + + # 因为calico-kube-controller使用了host网络，相当于使用了docker -net=host，需要 + # 卸载 /var/run/docker/netns/default + - name: 卸载docker 相关fs0 + mount: path=/var/run/docker/netns/default state=unmounted + - name: 卸载docker 相关fs1 + mount: path=/run/docker/netns/* state=unmounted + + - name: 卸载docker 相关fs2 + mount: path={{data_dir}}/docker/overlay2/*/merged state=unmounted + + - name: 卸载docker 相关fs3 + mount: path={{data_dir}}/docker/overlay2 state=unmounted + + - name: 卸载docker 相关fs4 + mount: path={{data_dir}}/docker/plugins state=unmounted + + - name: 卸载docker 相关fs5 + mount: path={{data_dir}}/docker/containers/*/shm state=unmounted + + - name: 清理目录和文件 + file: name={{ item }} state=absent + with_items: + - "/etc/cni/" + - "/etc/flannel/" + - "/var/run/flannel/" + - "/var/log/flannel/" + - "{{data_dir}}/docker/" + - "{{data_dir}}/registry/" + - "/var/run/docker/" + - "/etc/systemd/system/docker.service" + - "/etc/systemd/system/docker.service.requires/" + - "/var/run/calico/" + - "/etc/calico/" + - "/root/.kube/" + + - name: 清理 iptables + shell: "iptables -F && iptables -X \ + && iptables -F -t nat && iptables -X -t nat \ + && iptables -F -t raw && iptables -X -t raw \ + && iptables -F -t mangle && iptables -X -t mangle" + + - name: 清理网络 + shell: "ip link del docker0; ip link del flannel.1;ip link del cni0;ip link set tunl0 down;ip link del tunl0; systemctl restart network" + ignore_errors: true + + - name: 清理目录和文件 + file: name={{ item }} state=absent + with_items: + - "{{data_dir}}/etcd/" + - "/etc/etcd/ssl" + +- hosts: lb + tasks: + - name: stop keepalived service + shell: systemctl disable keepalived && systemctl stop keepalived + ignore_errors: true + + - name: stop haproxy service + shell: systemctl disable haproxy && systemctl stop haproxy + ignore_errors: true + + - name: stop nginx service + shell: systemctl disable nginx && systemctl stop nginx + ignore_errors: true + + - name: 清理LB 配置文件目录 + file: name={{ item }} state=absent + with_items: + - "/etc/haproxy" + - "/etc/keepalived" + - "/etc/nginx/nginx.conf" + - "/etc/kubernetes/ssl" + ignore_errors: true + +- hosts: deploy + tasks: + - name: 清理目录和文件 + file: name={{ item }} state=absent + with_items: + - "/etc/ansible/manifests/" + - "/root/.kube/" + - "/etc/kubernetes/ssl" + +- hosts: kube-storage-node + tasks: + - name: 清理存储节点数据 + file: name={{ item }} state=absent + with_items: + - "/var/lib/heketi" + - "/etc/glusterfs" + - "/var/log/glusterfs" + - "/var/lib/glusterd" + - "/var/lib/misc/glusterfsd" + - "/data/heketi" + - name: reboot + shell: shutdown -r + - name: wait for connecting + wait_for_connection: delay=70 + - name: 清理存储节点磁盘分区数据 + shell: vgdisplay -s -S pv_name={{gfs_device}} |awk '{print $1}'|sed 's/"//g'|xargs -r vgremove -y + ignore_errors: true diff --git a/playbooks/openvpn.yml b/playbooks/openvpn.yml new file mode 100644 index 0000000..041ce53 --- /dev/null +++ b/playbooks/openvpn.yml @@ -0,0 +1,3 @@ +- hosts: openvpn + roles: + - openvpn diff --git a/playbooks/proprietary-cloud/90.setup.yml b/playbooks/proprietary-cloud/90.setup.yml new file mode 100644 index 0000000..8fb6a0a --- /dev/null +++ b/playbooks/proprietary-cloud/90.setup.yml @@ -0,0 +1,15 @@ +- hosts: mysql + tasks: + - name: 创建数据目录 + file: name=/data/mysql state=directory mode=0755 owner=999 group=999 + +- hosts: kube-master + tasks: + - name: 创建rocketmq nameserver数据目录 + file: name=/data/rocketmq/nameserver state=directory mode=0755 recurse=yes + - name: 创建rocketmq broker数据目录 + file: name=/data/rocketmq/broker state=directory mode=0755 recurse=yes + +- hosts: deploy + roles: + - proprietary-cloud diff --git a/roles/gpaas/efk/elasticsearch/defaults/main.yml b/roles/gpaas/efk/elasticsearch/defaults/main.yml new file mode 100644 index 0000000..66572ed --- /dev/null +++ b/roles/gpaas/efk/elasticsearch/defaults/main.yml @@ -0,0 +1,6 @@ +elasticsearch_image: toyangdon/elasticsearch-oss:7.10.2 +alpine_image: library/alpine:3.6 +elasticsearch_indices_cleanup_image: toyangdon/es-index-cleaner:v0.1 + +es_data_path: >- + {{data_dir}}/es diff --git a/roles/gpaas/efk/elasticsearch/tasks/container.yml b/roles/gpaas/efk/elasticsearch/tasks/container.yml new file mode 100644 index 0000000..9a741b5 --- /dev/null +++ b/roles/gpaas/efk/elasticsearch/tasks/container.yml @@ -0,0 +1,20 @@ +- name: 创建es-statefulset.yaml文件 + template: src=es-statefulset.yaml dest={{ base_dir }}/manifests/es-statefulset.yaml + delegate_to: localhost + run_once: true + when: not ansible_check_mode + +- name: 部署es-statefulset + shell: "{{bin_dir}}/kubectl apply -f {{ base_dir }}/manifests/es-statefulset.yaml" + delegate_to: localhost + run_once: true + +- name: 创建es-indices-cleanup.yaml文件 + template: src=es-indices-cleanup.yaml dest={{ base_dir }}/manifests/es-indices-cleanup.yaml + delegate_to: localhost + run_once: true + +- name: 部署es-index-clean-job + shell: "{{bin_dir}}/kubectl apply -f {{ base_dir }}/manifests/es-indices-cleanup.yaml" + delegate_to: localhost + run_once: true diff --git a/roles/gpaas/efk/elasticsearch/tasks/main.yml b/roles/gpaas/efk/elasticsearch/tasks/main.yml new file mode 100644 index 0000000..08ce51d --- /dev/null +++ b/roles/gpaas/efk/elasticsearch/tasks/main.yml @@ -0,0 +1,29 @@ +- name: 创建数据目录 + file: name={{es_data_path}}/data state=directory mode=0755 owner=1000 group=1000 recurse=yes + +- name: 创建日志目录 + file: name={{es_data_path}}/logs state=directory mode=0755 owner=1000 group=1000 recurse=yes + +#- name: 修改系统文件限制 +# sysctl: +# name: vm.max_map_count +# value: 655360 +# sysctl_set: yes +# state: present +# reload: yes + +- include_tasks : container.yml + +#- name: 等待cloud-service服务启动成功 +# shell: "{{bin_dir}}/kubectl get statefulset elasticsearch -n kube-system -o jsonpath='{.status.readyReplicas}'" +# register: status_result +# until: 'groups["elasticsearch"]|length|string == status_result.stdout' +# retries: 50 +# delay: 5 +# run_once: true + +#- name: 重启es节点,目前发现部署elasticsearch节点会出现网络问题，重启可以恢复 +# shell: shutdown -r + +#- name: wait for connecting +# wait_for_connection: delay=60 diff --git a/roles/gpaas/efk/elasticsearch/templates/es-indices-cleanup.yaml b/roles/gpaas/efk/elasticsearch/templates/es-indices-cleanup.yaml new file mode 100644 index 0000000..725a9dd --- /dev/null +++ b/roles/gpaas/efk/elasticsearch/templates/es-indices-cleanup.yaml @@ -0,0 +1,18 @@ +apiVersion: batch/v1beta1 +kind: CronJob +metadata: + name: es-index-cleaner + namespace: kube-system +spec: + # 每天1点3分执行 + schedule: "3 1 */1 * *" + jobTemplate: + spec: + template: + spec: + containers: + - name: es-index-cleaner + image: {{BASE_IMAGE_URL}}/{{elasticsearch_indices_cleanup_image}} + restartPolicy: OnFailure + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 3 diff --git a/roles/gpaas/efk/elasticsearch/templates/es-statefulset.yaml b/roles/gpaas/efk/elasticsearch/templates/es-statefulset.yaml new file mode 100644 index 0000000..6760fcb --- /dev/null +++ b/roles/gpaas/efk/elasticsearch/templates/es-statefulset.yaml @@ -0,0 +1,248 @@ +# RBAC authn and authz +apiVersion: v1 +kind: ServiceAccount +metadata: + name: elasticsearch-logging + namespace: kube-system + labels: + k8s-app: elasticsearch-logging + addonmanager.kubernetes.io/mode: Reconcile +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: elasticsearch-logging + labels: + k8s-app: elasticsearch-logging + addonmanager.kubernetes.io/mode: Reconcile +rules: +- apiGroups: + - "" + resources: + - "services" + - "namespaces" + - "endpoints" + verbs: + - "get" +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + namespace: kube-system + name: elasticsearch-logging + labels: + k8s-app: elasticsearch-logging + addonmanager.kubernetes.io/mode: Reconcile +subjects: +- kind: ServiceAccount + name: elasticsearch-logging + namespace: kube-system + apiGroup: "" +roleRef: + kind: ClusterRole + name: elasticsearch-logging + apiGroup: "" +--- + +apiVersion: v1 +kind: ConfigMap +metadata: + name: es-config + namespace: kube-system +data: + elasticsearch.yml: | + path.data: {{es_data_path}}/data + path.logs: {{es_data_path}}/logs + #http.host: 0.0.0.0 + http.cors.enabled: true + http.cors.allow-origin: "*" + network.host: 0.0.0.0 + node.name: ${HOSTNAME}.elasticsearch + cluster.initial_master_nodes: [{%- for node in groups['elasticsearch'] -%}"elasticsearch-{{loop.index - 1}}.elasticsearch"{% if not loop.last %},{% endif %}{%- endfor -%}] + cluster.name: elasticsearch + discovery.zen.ping.unicast.hosts: [{%- for node in groups['elasticsearch'] -%}"elasticsearch-{{loop.index - 1}}.elasticsearch"{% if not loop.last %},{% endif %}{%- endfor -%}] + discovery.zen.minimum_master_nodes: {{(groups['elasticsearch']|length) // 2 + 1}} + discovery.zen.ping_timeout: 5s + +--- +--- +# https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.2/cluster/addons/fluentd-elasticsearch/es-statefulset.yaml +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: elasticsearch + namespace: kube-system + labels: + k8s-app: elasticsearch + addonmanager.kubernetes.io/mode: Reconcile +spec: + serviceName: elasticsearch + replicas: {{groups['elasticsearch']|length}} + selector: + matchLabels: + k8s-app: elasticsearch + template: + metadata: + labels: + k8s-app: elasticsearch + kubernetes.io/cluster-service: "true" + spec: +# hostNetwork: true + tolerations: +{% for taint in node_taints.split(",") %} + - operator: "Exists" + effect: "NoSchedule" + key: {{taint|replace("=:NoSchedule","")}} +{% endfor %} + nodeSelector: + "node.kubernetes.io/elasticsearch": "true" + affinity: +# nodeAffinity: +# requiredDuringSchedulingIgnoredDuringExecution: +# nodeSelectorTerms: +# - matchExpressions: +# - key: node-role.kubernetes.io/es +# operator: Exists + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: k8s-app + operator: In + values: + - elasticsearch + topologyKey: kubernetes.io/hostname + containers: + - image: "{{BASE_IMAGE_URL}}/{{ elasticsearch_image }}" + name: elasticsearch + resources: + # need more cpu upon initialization, therefore burstable class + limits: + cpu: 4 + memory: "6Gi" + requests: + cpu: 2 + memory: 3Gi + ports: + - containerPort: 9200 + name: db + protocol: TCP + - containerPort: 9300 + name: transport + protocol: TCP + livenessProbe: + tcpSocket: + port: transport + initialDelaySeconds: 30 + timeoutSeconds: 10 + readinessProbe: + tcpSocket: + port: transport + initialDelaySeconds: 30 + timeoutSeconds: 10 + volumeMounts: + - name: es-config + mountPath: /usr/share/elasticsearch/config/elasticsearch.yml + subPath: elasticsearch.yml + - name: es-persistent-storage + mountPath: /data/es + env: + - name: "NAMESPACE" + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: "ES_JAVA_OPTS" + value: "-Xms3g -Xmx3g" + securityContext: + privileged: true + volumes: + - name: es-config + configMap: + name: es-config + items: + - key: elasticsearch.yml + path: elasticsearch.yml + initContainers: + - image: {{BASE_IMAGE_URL}}/{{ alpine_image }} + command: ["/sbin/sysctl", "-w", "vm.max_map_count=262144"] + name: elasticsearch-init + securityContext: + privileged: true + volumeClaimTemplates: + - metadata: + name: es-persistent-storage + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 100Gi + storageClassName: local-storage + selector: + matchLabels: + pv: elasticsearch +--- +kind: Service +apiVersion: v1 +metadata: + labels: + elastic-app: elasticsearch-service + name: elasticsearch-service + namespace: kube-system +spec: + ports: + - port: 9200 + name: db + targetPort: 9200 + - port: 9300 + name: transport + targetPort: 9300 + selector: + k8s-app: elasticsearch +--- +kind: Service +apiVersion: v1 +metadata: + labels: + elastic-app: elasticsearch-service + name: elasticsearch + namespace: kube-system +spec: + ports: + - port: 9200 + name: db + targetPort: 9200 + - port: 9300 + name: transport + targetPort: 9300 + selector: + k8s-app: elasticsearch + clusterIP: None + +--- +{% for node in groups['elasticsearch'] %} +--- +apiVersion: v1 +kind: PersistentVolume +metadata: + name: es-persistent-storage-{{hostvars[node].NODE_ID}} + labels: + pv: elasticsearch +spec: + accessModes: + - ReadWriteOnce + capacity: + storage: 100Gi + local: + path: {{es_data_path}} + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: + - "{{hostvars[node].NODE_ID}}" + persistentVolumeReclaimPolicy: Retain + storageClassName: local-storage +{% endfor %} diff --git a/roles/gpaas/efk/fluentd/defaults/main.yml b/roles/gpaas/efk/fluentd/defaults/main.yml new file mode 100644 index 0000000..d7f2574 --- /dev/null +++ b/roles/gpaas/efk/fluentd/defaults/main.yml @@ -0,0 +1 @@ +fluentd_image: toyangdon/fluentd-es-image-arm:1.12.0 diff --git a/roles/gpaas/efk/fluentd/tasks/container.yml b/roles/gpaas/efk/fluentd/tasks/container.yml new file mode 100644 index 0000000..2a185b7 --- /dev/null +++ b/roles/gpaas/efk/fluentd/tasks/container.yml @@ -0,0 +1,7 @@ +- name: 创建fluentd.yaml文件 + template: src=fluentd.yaml dest={{ base_dir }}/manifests/fluentd.yaml + delegate_to: localhost + +- name: 部署fluentd.yaml + shell: "{{bin_dir}}/kubectl apply -f {{ base_dir }}/manifests/fluentd.yaml -n kube-system" + delegate_to: localhost diff --git a/roles/gpaas/efk/fluentd/tasks/main.yml b/roles/gpaas/efk/fluentd/tasks/main.yml new file mode 100644 index 0000000..dffa6bc --- /dev/null +++ b/roles/gpaas/efk/fluentd/tasks/main.yml @@ -0,0 +1 @@ +- include_tasks : container.yml \ No newline at end of file diff --git a/roles/gpaas/efk/fluentd/templates/fluentd.yaml b/roles/gpaas/efk/fluentd/templates/fluentd.yaml new file mode 100644 index 0000000..630f41a --- /dev/null +++ b/roles/gpaas/efk/fluentd/templates/fluentd.yaml @@ -0,0 +1,481 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: fluentd-es + labels: + k8s-app: fluentd-es + addonmanager.kubernetes.io/mode: Reconcile +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: fluentd-es + labels: + k8s-app: fluentd-es + addonmanager.kubernetes.io/mode: Reconcile +rules: +- apiGroups: + - "" + resources: + - "namespaces" + - "pods" + verbs: + - "get" + - "watch" + - "list" +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: fluentd-es + labels: + k8s-app: fluentd-es + addonmanager.kubernetes.io/mode: Reconcile +subjects: +- kind: ServiceAccount + name: fluentd-es + namespace: kube-system + apiGroup: "" +roleRef: + kind: ClusterRole + name: fluentd-es + apiGroup: "" +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: fluentd-es-v3.1.1 + labels: + k8s-app: fluentd-es + version: v3.1.1 + addonmanager.kubernetes.io/mode: Reconcile +spec: + selector: + matchLabels: + k8s-app: fluentd-es + version: v3.1.1 + template: + metadata: + labels: + k8s-app: fluentd-es + version: v3.1.1 + spec: + priorityClassName: system-node-critical + serviceAccountName: fluentd-es + containers: + - name: fluentd-es + image: {{BASE_IMAGE_URL}}/{{fluentd_image}} + env: + - name: FLUENTD_ARGS + value: --no-supervisor -q + resources: + limits: + memory: 500Mi + requests: + cpu: 100m + memory: 200Mi + volumeMounts: + - name: varlog + mountPath: /var/log + - name: datalog + mountPath: /data/log + - name: varlibdockercontainers + mountPath: /data/docker/containers + readOnly: true + - name: config-volume + mountPath: /etc/fluent/config.d + ports: + - containerPort: 24231 + name: prometheus + protocol: TCP + livenessProbe: + tcpSocket: + port: prometheus + initialDelaySeconds: 5 + timeoutSeconds: 10 + readinessProbe: + tcpSocket: + port: prometheus + initialDelaySeconds: 5 + timeoutSeconds: 10 + terminationGracePeriodSeconds: 30 + volumes: + - name: varlog + hostPath: + path: /var/log + - name: datalog + hostPath: + path: /data/log + - name: varlibdockercontainers + hostPath: + path: /data/docker/containers + - name: config-volume + configMap: + name: fluentd-es-config-v0.2.1 +--- +kind: ConfigMap +apiVersion: v1 +metadata: + name: fluentd-es-config-v0.2.1 + labels: + addonmanager.kubernetes.io/mode: Reconcile +data: + system.conf: |- + + root_dir /tmp/fluentd-buffers/ + + containers.input.conf: |- + # This configuration file for Fluentd / td-agent is used + # to watch changes to Docker log files. The kubelet creates symlinks that + # capture the pod name, namespace, container name & Docker container ID + # to the docker logs for pods in the /var/log/containers directory on the host. + # If running this fluentd configuration in a Docker container, the /var/log + # directory should be mounted in the container. + # + # These logs are then submitted to Elasticsearch which assumes the + # installation of the fluent-plugin-elasticsearch & the + # fluent-plugin-kubernetes_metadata_filter plugins. + # See https://github.com/uken/fluent-plugin-elasticsearch & + # https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter for + # more information about the plugins. + # + # Example + # ======= + # A line in the Docker log file might look like this JSON: + # + # {"log":"2014/09/25 21:15:03 Got request with path wombat\n", + # "stream":"stderr", + # "time":"2014-09-25T21:15:03.499185026Z"} + # + # The time_format specification below makes sure we properly + # parse the time format produced by Docker. This will be + # submitted to Elasticsearch and should appear like: + # $ curl 'http://elasticsearch-logging:9200/_search?pretty' + # ... + # { + # "_index" : "logstash-2014.09.25", + # "_type" : "fluentd", + # "_id" : "VBrbor2QTuGpsQyTCdfzqA", + # "_score" : 1.0, + # "_source":{"log":"2014/09/25 22:45:50 Got request with path wombat\n", + # "stream":"stderr","tag":"docker.container.all", + # "@timestamp":"2014-09-25T22:45:50+00:00"} + # }, + # ... + # + # The Kubernetes fluentd plugin is used to write the Kubernetes metadata to the log + # record & add labels to the log record if properly configured. This enables users + # to filter & search logs on any metadata. + # For example a Docker container's logs might be in the directory: + # + # /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b + # + # and in the file: + # + # 997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log + # + # where 997599971ee6... is the Docker ID of the running container. + # The Kubernetes kubelet makes a symbolic link to this file on the host machine + # in the /var/log/containers directory which includes the pod name and the Kubernetes + # container name: + # + # synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log + # -> + # /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log + # + # The /var/log directory on the host is mapped to the /var/log directory in the container + # running this instance of Fluentd and we end up collecting the file: + # + # /var/log/containers/synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log + # + # This results in the tag: + # + # var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log + # + # The Kubernetes fluentd plugin is used to extract the namespace, pod name & container name + # which are added to the log message as a kubernetes field object & the Docker container ID + # is also added under the docker field object. + # The final tag is: + # + # kubernetes.var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log + # + # And the final log record look like: + # + # { + # "log":"2014/09/25 21:15:03 Got request with path wombat\n", + # "stream":"stderr", + # "time":"2014-09-25T21:15:03.499185026Z", + # "kubernetes": { + # "namespace": "default", + # "pod_name": "synthetic-logger-0.25lps-pod", + # "container_name": "synth-lgr" + # }, + # "docker": { + # "container_id": "997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b" + # } + # } + # + # This makes it easier for users to search for logs by pod name or by + # the name of the Kubernetes container regardless of how many times the + # Kubernetes pod has been restarted (resulting in a several Docker container IDs). + # Json Log Example: + # {"log":"[info:2016-02-16T16:04:05.930-08:00] Some log text here\n","stream":"stdout","time":"2016-02-17T00:04:05.931087621Z"} + # CRI Log Example: + # 2016-02-17T00:04:05.931087621Z stdout F [info:2016-02-16T16:04:05.930-08:00] Some log text here + + @id fluentd-containers.log + @type tail + path /var/log/containers/*.log + pos_file /var/log/es-containers.log.pos + tag raw.kubernetes.* + read_from_head true + + @type multi_format + + format json + time_key time + time_format %Y-%m-%dT%H:%M:%S.%NZ + + + format /^(? + + + # Detect exceptions in the log output and forward them as one log entry. + + @id raw.kubernetes + @type detect_exceptions + remove_tag_prefix raw + message log + stream stream + multiline_flush_interval 5 + max_bytes 500000 + max_lines 1000 + + # Concatenate multi-line logs + + @id filter_concat + @type concat + key message + multiline_end_regexp /\n$/ + separator "" + + # Enriches records with Kubernetes metadata + + @id filter_kubernetes_metadata + @type kubernetes_metadata + + # Fixes json fields in Elasticsearch + + @id filter_parser + @type parser + key_name log + reserve_data true + remove_key_name_field true + + @type multi_format + + format json + + + format none + + + + system.input.conf: |- + # Example: + # 2015-12-21 23:17:22,066 [salt.state ][INFO ] Completed state [net.ipv4.ip_forward] at time 23:17:22.066081 + + @id minion + @type tail + format /^(?