Linux命令工具-find

1.简介#

能够在多层级目录中递归查找文件的工具

2.命令格式#

find [path...] [expression]

[path] - 指定递归查找的根目录(默认当前目录)
[expression] 可以由3部分组成:
options - 选项(相当于全局配置)
tests - 测试(即判断指定条件是否满足)
actions - 动作(即找到符合条件的文件之后,对每个文件进行的操作)(默认-print)

3.示例#

find . -mindepth 2 -name '*.txt' -size +1M -exec ls -sailh {} \;

path: .
expression.options: -mindepth 2
expression.tests: -name '*.txt' -size +1M
expression.actions: -exec ls -sailh {} \;

{} 表示查找到的每个文件, \; 表示 -exec 的结束

该示例表示的意思是,从当前目录开始递归查找,文件路径最小深度为2(即>=2),文件名以 .txt 结尾,并且文件大小大于1M,查找完之后的动作为使用 ls -sailh 命令将其展示出来

4.资料#

  • linux man find

ab - Apache HTTP服务器基准测试工具

1.简介#

HTTP服务器基准测试工具

2.摘要#

1
2
3
4
5
6
ab [ -A auth-username:password ] [ -c concurrency ] [ -C cookie-name=value ] [ -d ]
[ -e csv-file ] [ -g gnuplot-file ] [ -h ] [ -H custom-header ] [ -i ] [ -k ]
[ -n requests ] [ -p POST-file ] [ -P proxy-auth-username:password ] [ -q ]
[ -s ] [ -S ] [ -t timelimit ] [ -T content-type ] [ -v verbosity] [ -V ]
[ -w ] [ -x <table>-attributes ] [ -X proxy[:port] ] [ -y <tr>-attributes ]
[ -z <td>-attributes ] [http://]hostname[:port]/path

3.选项#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
-A auth-username:password
Http Basic认证
-c concurrency
并发数,默认1
-C cookie-name=value
添加一个Cookie,示例 -C 'XSESSIONID=aaaa;key2=bbbb'
-d
不要显示"Percentage of the requests served within a certain time (ms)"这个表格(遗留支持)
-e csv-file
输入csv文件
-g gnuplot-file
输出gnuplot文件
-h
显示帮助信息
-H custom-header
添加头部,示例 -H "Accept-Encoding: zip/zop;8bit"
-i
发送HEAD请求
-k
开启http KeepAlive特性,默认关闭
-n requests
请求数,默认1
-p POST-file
指定数据文件并发送POST请求
-P proxy-auth-username:password
代理
-q
不显示执行百分比信息
-s
https相关,该特性为实验性,一般不会用到
-S
不显示中位数和标准偏差值,还有不显示其他的一些值。默认仅显示最小值/平均值/最大值(遗留支持)
-t timelimit
基准测试的最大时间(秒)
-T content-type
为POST数据指定Content-Type头部
-v verbosity
设置详情级别,详细程度从4到2递减
-V
显示版本号
-w
打印输出结果为HTML表格(源码)
-x <table>-attributes
String to use as attributes for <table>. Attributes are inserted <table here >.
设置<table>属性,示例 -x bgcolor=blue => <table bgcolor=blue>
-X proxy[:port]
为请求设置一个代理服务器
-y <tr>-attributes
设置<tr>属性
-z <td>-attributes
设置<td>属性

不同版本参数不完全相同,具体见版本的帮助文档 ab -h

4.示例#

1
# ab -n 100 -c 10 'https://publib.boulder.ibm.com/httpserv/manual70/programs/ab.html'
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking publib.boulder.ibm.com (be patient).....done


Server Software:
Server Hostname: publib.boulder.ibm.com
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,4096,128

Document Path: /httpserv/manual70/programs/ab.html
Document Length: 10193 bytes

Concurrency Level: 10
Time taken for tests: 17.458 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 1045400 bytes
HTML transferred: 1019300 bytes
Requests per second: 5.73 [#/sec] (mean)
Time per request: 1745.789 [ms] (mean)
Time per request: 174.579 [ms] (mean, across all concurrent requests)
Transfer rate: 58.48 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 695 951 198.0 925 1629
Processing: 454 582 57.3 596 693
Waiting: 228 293 30.9 298 368
Total: 1152 1533 233.7 1526 2267

Percentage of the requests served within a certain time (ms)
50% 1526
66% 1558
75% 1594
80% 1614
90% 1652
95% 2222
98% 2267
99% 2267
100% 2267 (longest request)

5.资料#

SpringBoot集成Elasticsearch

1.Maven配置#

1
2
3
4
5
6
<!-- Spring Data Elasticsearch -->
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-elasticsearch</artifactId>
<!-- 版本随spring-boot-starter-parent -->
</dependency>

2.代码配置#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
package com.yeshimin.ysmspace.config;

import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.elasticsearch.client.ClientConfiguration;
import org.springframework.data.elasticsearch.client.RestClients;
import org.springframework.data.elasticsearch.config.AbstractElasticsearchConfiguration;

@Configuration
public class ElasticsearchConfiguration extends AbstractElasticsearchConfiguration {

@Override
public RestHighLevelClient elasticsearchClient() {
final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo("vhost4:9200") // 指定es服务地址
.build();

return RestClients.create(clientConfiguration).rest();
}
}

3.数据访问层代码#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
package com.yeshimin.ysmspace.model.elasticsearch;

// ... import

@Data
@Document(indexName = Common.PROJECT_NAME + "_product")
public class ProductDoc {

/**
* 主键ID
*/
@Id
@Field(name = "id", type = FieldType.Long)
private Long id;

/**
* 创建时间
*/
@Field(name = "create_time", type = FieldType.Date, format = DateFormat.basic_date_time)
private LocalDateTime createTime;

/**
* 商品名称
*/
@Field(name = "name", type = FieldType.Keyword)
private String name;

// ...
}
1
2
3
4
5
6
7
package com.yeshimin.ysmspace.dal.elasticsearch;

// ... import

@Repository
public interface ProductEsRepository extends ElasticsearchRepository<ProductDoc, Long> {
}

4.使用#

遵循spring data规则

示例:

1
2
3
4
5
6
@Autowired
private ProductEsRepository productEsRepository;

ProductDoc productDoc = new ProductDoc();
// ...
productEsRepository.save(productDoc);

5.版本信息#

  • springboot parent - 2.3.5.RELEASE
  • spring data elasticsearch - 7.6.2
  • elasticsearch - 7.9.3

SpringBoot集成RabbitMQ-实现订单超时取消

1.解决方案#

利用RabbitMQ的死信队列机制实现订单超时自动取消;
具体为下单后发送设置了TTL的订单消息到队列A,当超时未消费时,消息被转发到对应的死信队列B,监听队列B的消费者收到消息对其进行处理;
消费者对订单进行取消操作需保证幂等,因为可能订单已经被用户主动取消;

2.Maven配置#

1
2
3
4
5
6
<!-- Spring RabbitMQ -->
<dependency>
<groupId>org.springframework.amqp</groupId>
<artifactId>spring-rabbit</artifactId>
<!-- 版本随spring-boot-starter-parent -->
</dependency>

3.YML配置#

1
2
3
4
5
6
7
spring:
rabbitmq:
host: 192.168.31.208
port: 5672
virtual-host: ysmspace
username: username1
password: password1

4.代码配置#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
package com.yeshimin.ysmspace.config;

import org.springframework.amqp.core.AmqpAdmin;
import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.DirectExchange;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.rabbit.connection.ConnectionFactory;
import org.springframework.amqp.rabbit.core.RabbitAdmin;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.HashMap;
import java.util.Map;

@Configuration
public class RabbitMqConfiguration {

public static final String EXCHANGE_DEFAULT_DIRECT = "amq.direct";
public static final String QUEUE_ORDER_PENDING = "q.order.pending";
public static final String ROUTING_ORDER_PENDING = "rk.order.pending";
public static final String DLX_ORDER_PENDING = EXCHANGE_DEFAULT_DIRECT;
public static final String DLQ_ORDER_PENDING = "q.dl.order.pending";
public static final String DLR_ORDER_PENDING = "rk.dl.order.pending";

@Bean
public AmqpAdmin amqpAdmin(ConnectionFactory connectionFactory) {
return new RabbitAdmin(connectionFactory);
}

@Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
return new RabbitTemplate(connectionFactory);
}

@Bean
public DirectExchange defaultDirectExchange() {
return new DirectExchange(EXCHANGE_DEFAULT_DIRECT);
}

@Bean
public Queue orderPendingQueue() {
Map<String, Object> args = new HashMap<>();
// args.put("x-message-ttl", 30 * 60 * 1000); ttl由消息实体指定
args.put("x-dead-letter-exchange", DLX_ORDER_PENDING);
args.put("x-dead-letter-routing-key", DLR_ORDER_PENDING);
return new Queue(QUEUE_ORDER_PENDING, true, false, false, args);
}

@Bean
public Queue orderPendingDeadLetterQueue() {
return new Queue(DLQ_ORDER_PENDING);
}

@Bean
public Binding orderPendingBinding() {
return new Binding(QUEUE_ORDER_PENDING, Binding.DestinationType.QUEUE,
EXCHANGE_DEFAULT_DIRECT, ROUTING_ORDER_PENDING, null);
}

@Bean
public Binding orderPendingDeadLetterBinding() {
return new Binding(DLQ_ORDER_PENDING, Binding.DestinationType.QUEUE,
DLX_ORDER_PENDING, DLR_ORDER_PENDING, null);
}
}

5.消息发布和消费及相关配置#

5.1 消息后置处理器-用于设置消息的超时时间#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
package com.yeshimin.ysmspace.rabbitmq;

import org.springframework.amqp.AmqpException;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.core.MessagePostProcessor;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;

@Component
public class DefaultMessagePostProcessor implements MessagePostProcessor {

@Value("${order.timeout}")
private Integer orderTimeout;

@Override
public Message postProcessMessage(Message message) throws AmqpException {
message.getMessageProperties().setExpiration(String.valueOf(orderTimeout));
return message;
}
}

5.2 消息发布#

1
2
3
4
5
6
7
8
9
10
11
12
13
@Autowired
private RabbitTemplate rabbitTemplate;
@Autowired
private DefaultMessagePostProcessor defaultMessagePostProcessor;

// ...

rabbitTemplate.convertAndSend(
RabbitMqConfiguration.EXCHANGE_DEFAULT_DIRECT,
RabbitMqConfiguration.ROUTING_SYNC_TO_ES,
JSON.toJSONString(message), defaultMessagePostProcessor);

// ...

5.3 消息消费#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
package com.yeshimin.ysmspace.rabbitmq;

// ... import

@Slf4j
@Component
public class DefaultRabbitListener {

@Autowired
private OrderAppService orderAppService;

@RabbitListener(queues = RabbitMqConfiguration.DLQ_ORDER_PENDING)
public void listen(String message) {
log.debug("listen(), message: {}", message);

OrderPendingMessage jsonData = JSON.parseObject(message, OrderPendingMessage.class);

try {
orderAppService.cancel(jsonData.getOrderId());
} catch (Exception e) {
log.warn("未成功取消订单!");
e.printStackTrace();
}
}
}

6.版本信息#

  • springboot parent - 2.3.5.RELEASE
  • rabbitmq - 3.3.5

MySQL复制之一主二从配置

1.环境#

主机:vhost4, vhost5, vhost6
系统:CentOS7
vhost4: MySQL.slave
vhost5: MySQL.master
vhost6: MySQL.slave
MySQL: v5.7.36

2.配置#

2.1 (一个)master配置#

2.1.1 my.cnf配置#

cat /etc/my.cnf

1
2
3
4
5
6
7
8
9
10
11
12
...略

[mysqld]
# 开启binary log
log-bin=mysql-bin
# 设置server_id
server-id=1
# For the greatest possible durability and consistency in a replication setup using InnoDB with transactions, you should use innodb_flush_log_at_trx_commit=1 andsync_binlog=1 in the source's my.cnf file
innodb_flush_log_at_trx_commit=1
sync_binlog=1

...略

2.1.2 确保MySQL的 skip_networking 变量是OFF的#

1
SHOW VARIABLES LIKE '%skip_networking%';

2.1.3 重启MySQL服务#

1
systemctl restart mysqld

2.1.4 在master上创建用于复制的用户账号#

1
2
CREATE USER 'repl'@'%' IDENTIFIED BY '密码';
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%';

2.1.5 获取master上binary log的坐标#

1
2
3
4
5
6
7
8
mysql> FLUSH TABLES WITH READ LOCK; # 锁定只读
mysql> SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000354 | 4649036 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.01 sec)

2.1.6 同步现有数据,从master上导出数据#

1
mysqldump -uroot -p --all-databases --master-data > dbdump.db

2.1.7 解除锁定#

在master上执行UNLOCK TABLES;(对应上面的FLUSH TABLES WITH READ LOCK;)

2.2 (每个)slave配置#

2.2.1 replica设置(一般不需要开启binary log)#

cat /etc/my.cnf
两个slave的server-id分别配置为2和3

1
2
3
4
5
6
...略

[mysqld]
server-id=2

...略

2.2.2 在replica上配置master#

MASTER_LOG_FILE和MASTER_LOG_POST参数为master上binary log的坐标

1
2
3
4
5
6
CHANGE MASTER TO
MASTER_HOST='vhost5',
MASTER_USER='repl',
MASTER_PASSWORD='密码',
MASTER_LOG_FILE='mysql-bin.000354',
MASTER_LOG_POS=4649036;

2.2.3 启动slave开始复制#

1.导入master的数据
2.start slave

3.资料#

PowerShell脚本简单应用

1.背景#

Windows系统的HyperV中的虚拟机,在正常关机、开机后不能正确(符合预期地)保存/恢复虚拟机状态;
具体表现在CentOS7 Linux虚拟机中prometheus服务启动出错(详细错误未记录);
在尝试通过HyperV的配置使达到预期未果后,转而使用其他方式;
只要使虚拟机经历正常关机和开机就能使结果符合预期;
最终使用Windows PowerShell自动化脚本的方式。

2.代码#

2.1 启动脚本:StartHyperVMs.ps1#

1
2
3
4
5
6
7
8
9
10
11
12
# Self-elevate the script if required
# 这段代码会使脚本已管理员角色运行,不添加会报权限错误
if (-Not ([Security.Principal.WindowsPrincipal] [Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] 'Administrator')) {
if ([int](Get-CimInstance -Class Win32_OperatingSystem | Select-Object -ExpandProperty BuildNumber) -ge 6000) {
$Command = "-File `"" + $MyInvocation.MyCommand.Path + "`" " + $MyInvocation.UnboundArguments
Start-Process -FilePath PowerShell.exe -Verb RunAs -ArgumentList $Command
Exit
}
}

# Place your script here
start-vm -name vhost*

2.2 关机脚本:StopHyperVMs.ps1#

1
2
3
4
5
6
7
8
9
10
11
12
# Self-elevate the script if required
# 这段代码会使脚本已管理员角色运行,不添加会报权限错误
if (-Not ([Security.Principal.WindowsPrincipal] [Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] 'Administrator')) {
if ([int](Get-CimInstance -Class Win32_OperatingSystem | Select-Object -ExpandProperty BuildNumber) -ge 6000) {
$Command = "-File `"" + $MyInvocation.MyCommand.Path + "`" " + $MyInvocation.UnboundArguments
Start-Process -FilePath PowerShell.exe -Verb RunAs -ArgumentList $Command
Exit
}
}

# Place your script here
stop-vm -name vhost*

3.配置#

3.1 打开【本地组策略编辑器】#

Win + r 输入 gpedit.msc

3.1 配置启动脚本#

计算机配置 -> Windows设置 -> 脚本(启动/关机) -> 【启动】 -> PowerShell脚本 -> 添加 -> 脚本名(配置为StartHyperVMs.p1的路径)

3.1 配置关机脚本#

计算机配置 -> Windows设置 -> 脚本(启动/关机) -> 【关机】 -> PowerShell脚本 -> 添加 -> 脚本名(配置为StopHyperVMs.p1的路径)

4.资料#

Prometheus+Grafana监控主机和MySQL数据库

1.环境#

主机:vhost4, vhost5, vhost6
系统:CentOS7
vhost4: MySQL.slave, Prometheus, NodeExporter, MysqlExporter
vhost5: MySQL.master, NodeExporter, MysqlExporter
vhost6: MySQL.slave, NodeExporter, MysqlExporter
Prometheus: v2.24.1
Grafana: v7.4.0
NodeExporter: v1.1.0
MysqlExporter: v0.14.0

2.Prometheus#

2.1 下载压缩包并解压#

1
# ls -al /opt/prometheus/prometheus/prometheus-2.24.1.linux-amd64
1
2
3
4
5
6
7
8
9
10
11
总用量 165812
drwxr-xr-x 5 3434 3434 144 5月 10 16:35 .
drwxr-xr-x 3 root root 84 5月 5 13:51 ..
drwxr-xr-x 2 3434 3434 38 1月 20 2021 console_libraries
drwxr-xr-x 2 3434 3434 173 1月 20 2021 consoles
drwxr-xr-x 13 root root 4096 5月 10 17:00 data
-rw-r--r-- 1 3434 3434 11357 1月 20 2021 LICENSE
-rw-r--r-- 1 3434 3434 3420 1月 20 2021 NOTICE
-rwxr-xr-x 1 3434 3434 89894679 1月 20 2021 prometheus
-rw-r--r-- 1 root root 1324 5月 10 16:35 prometheus.yml
-rwxr-xr-x 1 3434 3434 79870425 1月 20 2021 promtool

2.2 配置#

1
# cat /opt/prometheus/prometheus/prometheus-2.24.1.linux-amd64/prometheus.yml 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'

# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.

static_configs:
- targets: ['localhost:9090']

- job_name: 'node'
static_configs:
- targets: ['vhost4:9100', 'vhost5:9100', 'vhost6:9100']

- job_name: 'mysqld'
static_configs:
- targets: ['vhost4:9104','vhost5:9104','vhost6:9104']

2.3 服务化#

1
# cat /usr/lib/systemd/system/prometheus.service 
1
2
3
4
5
6
7
8
9
10
[Unit]
Description=Prometheus

[Service]
WorkingDirectory=/opt/prometheus/prometheus/prometheus-2.24.1.linux-amd64
ExecStart=/opt/prometheus/prometheus/prometheus-2.24.1.linux-amd64/prometheus --config.file=/opt/prometheus/prometheus/prometheus-2.24.1.linux-amd64/prometheus.yml
User=root

[Install]
WantedBy=multi-user.target

3.Grafana#

3.1 下载安装包并安装#

1
# ls -al /opt/prometheus/grafana/
1
2
3
4
总用量 49732
drwxr-xr-x 2 root root 40 2月 8 2021 .
drwxr-xr-x 6 root root 83 5月 10 16:33 ..
-rw-r--r-- 1 root root 50921720 2月 8 2021 grafana-7.4.0-1.x86_64.rpm
1
# rpm -ivh /opt/prometheus/grafana/grafana-7.4.0-1.x86_64.rpm

3.2 配置#

默认

3.3 服务化#

1
2
# systemctl start grafana-server.service
# systemctl enable grafana-server.service

4.NodeExporter#

4.1 下载压缩包并解压#

1
# ls -al node_exporter/node_exporter-1.1.0.linux-amd64
1
2
3
4
5
6
总用量 18720
drwxr-xr-x 2 root root 56 2月 6 2021 .
drwxr-xr-x 3 root root 88 2月 8 2021 ..
-rw-r--r-- 1 root root 11357 2月 6 2021 LICENSE
-rwxr-xr-x 1 root root 19151814 2月 6 2021 node_exporter
-rw-r--r-- 1 root root 463 2月 6 2021 NOTICE

4.2 配置#

4.3 服务化#

1
# cat /usr/lib/systemd/system/node-exporter.service
1
2
3
4
5
6
7
8
9
[Unit]
Description=NodeExporter

[Service]
ExecStart=/opt/prometheus/node_exporter/node_exporter-1.1.0.linux-amd64/node_exporter
User=root

[Install]
WantedBy=multi-user.target

5.MysqlExporter#

5.1 下载压缩包并解压#

1
# ls -al /opt/prometheus/mysqld_exporter/mysqld_exporter-0.14.0.linux-amd64/
1
2
3
4
5
6
7
总用量 14828
drwxr-xr-x 2 root root 72 5月 10 16:37 .
drwxr-xr-x 3 root root 48 5月 10 16:34 ..
-rw-r--r-- 1 root root 11357 5月 10 16:34 LICENSE
-rw-r--r-- 1 root root 35 5月 10 16:37 my.cnf
-rwxr-xr-x 1 root root 15163162 5月 10 16:34 mysqld_exporter
-rw-r--r-- 1 root root 65 5月 10 16:34 NOTICE

5.2 配置#

1
# cat /opt/prometheus/mysqld_exporter/mysqld_exporter-0.14.0.linux-amd64/my.cnf 
1
2
3
[client]
user=数据库账号
password=数据库密码

按需还可以指定 host 和 port

5.3 服务化#

1
# cat /usr/lib/systemd/system/mysqld-exporter.service 
1
2
3
4
5
6
7
8
9
[Unit]
Description=MysqldExporter

[Service]
ExecStart=/opt/prometheus/mysqld_exporter/mysqld_exporter-0.14.0.linux-amd64/mysqld_exporter --config.my-cnf=/opt/prometheus/mysqld_exporter/mysqld_exporter-0.14.0.linux-amd64/my.cnf
User=root

[Install]
WantedBy=multi-user.target

6.资料#

阿里云服务器开启HTTPS/SSL by Nginx

1.获取免费SSL证书#

云服务器是阿里云的,同时阿里云也赠送免费DV单域名证书(0元购买)

1
2
每个实名主体个人/企业,一个自然年内可以领取一次数量为20的免费证书资源包
免费资源包到自然年结束时,会自动清除未签发的数量(每个自然年12月31日24:00)

为每个单域名申请证书,等待签发后,然后根据服务器类型(Tomcat,Apache,Nginx等)下载相应的证书

2.配置Nginx支持HTTPS#

1
cat /etc/nginx/conf.d/vhost/ysmblog.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
server {
listen 80;
listen [::]:80;
server_name blog.yeshimin.com;
rewrite ^(.*) https://$server_name$1 permanent;
}

server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name blog.yeshimin.com;

ssl_certificate "/etc/nginx/ssl/blog.yeshimin.com.pem";
ssl_certificate_key "/etc/nginx/ssl/blog.yeshimin.com.key";
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 10m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

location / {
root /var/www/ysmblog;
}
}
1
nginx -s reload

3.资料#

4.版本#

  • Nginx 1.20.1

SpringCloudGateway集成Swagger

1.背景#

微服务架构下,各个服务单独集成使用Swagger不方便,所以需要将其集中到网关使用

2.原理#

SwaggerResourcesProvider接口负责提供文档信息
官方默认提供实现类InMemorySwaggerResourcesProvider,只针对单服务场景
可以自定义实现类并配合网关或服务发现机制提供各服务的文档信息

3.实现#

自定义实现类核心代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
@Primary
@Component
public class CustomSwaggerResourcesProvider implements SwaggerResourcesProvider {

@Lazy
@Autowired
private RouteLocator routeLocator;

@Override
public List<SwaggerResource> get() {
List<Route> routes = new ArrayList<>();
routeLocator.getRoutes().subscribe(routes::add);
return routes.stream().map(route -> this.resource(route.getUri().getHost())).collect(Collectors.toList());
}

private SwaggerResource resource(String service) {
SwaggerResource swaggerResource = new SwaggerResource();
swaggerResource.setName(service);
swaggerResource.setUrl("/" + service + "/v3/api-docs");
swaggerResource.setSwaggerVersion("3.0.3");
return swaggerResource;
}
}

利用SpringCloudGateway的RouteLocator获取服务信息,组装并返回文档信息
@Primary注解使该自定义实现类代替默认的InMemorySwaggerResourcesProvider
网关只做此简单配置,Docket在各服务配置

4.补充#

问题:
如上配置后,在网关已能打开Swagger并出现各服务列表
但是发现服务接口调用不通,原因是网关截掉了路径中的服务信息,导致此处Swagger的Servers列表中的基地址缺少服务信息,所以无法路由到具体服务
而下游服务中的Swagger过滤器WebMvcBasePathAndHostnameTransformationFilter也未对此种情况进行支持(即便网关向下游服务提供了X-Forwarded-相关头部信息),所以提供给上游网关的Servers信息中才没有服务信息

解决方案:
在下游服务中自定义WebMvcOpenApiTransformationFilter实现类(可直接继承官方默认实现类WebMvcBasePathAndHostnameTransformationFilter),补充服务信息
确保优先级小于WebMvcBasePathAndHostnameTransformationFilter

自定义Filter核心代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
@Order(Ordered.HIGHEST_PRECEDENCE + 100)
public class PrefixedWebMvcOpenApiTransformationFilter extends WebMvcBasePathAndHostnameTransformationFilter {

private static final String HEADER_X_FORWARDED_PREFIX = "X-Forwarded-Prefix";

public PrefixedWebMvcOpenApiTransformationFilter(String oasPath) {
super(oasPath);
}

@Override
public OpenAPI transform(OpenApiTransformationContext<HttpServletRequest> context) {
OpenAPI openApi = context.getSpecification();
context.request().ifPresent(servletRequest -> {
String headerValue = servletRequest.getHeader(HEADER_X_FORWARDED_PREFIX);
if (!StringUtils.isEmpty(headerValue)) {
openApi.getServers().forEach(server -> server.setUrl(server.getUrl() + headerValue));
}
});
return openApi;
}
}

配置该自定义Filter,核心代码如下:

1
2
3
4
5
6
7
8
9
@Configuration
public class SwaggerConfiguration {

@Bean
public WebMvcOpenApiTransformationFilter prefixedWebMvcOpenApiTransformer(
@Value(springfox.documentation.oas.web.SpecGeneration.OPEN_API_SPECIFICATION_PATH) String oasPath) {
return new PrefixedWebMvcOpenApiTransformationFilter(oasPath);
}
}

5.资料#

6.版本#

  • Spring Cloud Gateway 2.2.5
  • SpringFox(Swagger) 3.0.0

基于SpringCloudGateway+SpringSecurity的微服务鉴权方案

1.Spring Cloud Gateway集成Spring Security#

网关只进行服务级别的认证,不做授权逻辑判断

1.1 Spring Secuty核心配置#

核心代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
@EnableWebFluxSecurity
public class WebSecurityConfiguration {

@Autowired
private JwtTokenAuthenticationManager jwtTokenAuthenticationManager;

@Bean
public SecurityWebFilterChain springSecurityFilterChain(ServerHttpSecurity http) {
// 鉴权配置
AuthenticationWebFilter authenticationWebFilter = new AuthenticationWebFilter(jwtTokenAuthenticationManager);
authenticationWebFilter.setServerAuthenticationConverter(new JwtTokenAuthenticationConverter());
authenticationWebFilter.setSecurityContextRepository(new WebSessionServerSecurityContextRepository());
http.addFilterAt(authenticationWebFilter, SecurityWebFiltersOrder.AUTHENTICATION);

// 清除一次请求后的SecurityContext
http.addFilterBefore(new ClearExchangeSecurityContextWebFilter(), SecurityWebFiltersOrder.REACTOR_CONTEXT);

// 跨域配置
http.cors().configurationSource(corsConfigurationSource());

http.csrf().disable();

http.authorizeExchange()
.pathMatchers("/**/login").permitAll()
.pathMatchers("/**/logout").authenticated()
.pathMatchers("/upms/**").authenticated()
.pathMatchers("/product/**").authenticated()
.anyExchange().denyAll();

return http.build();
}

@Bean
public CorsConfigurationSource corsConfigurationSource() {
UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
CorsConfiguration config = new CorsConfiguration();

config.addAllowedHeader("*");
config.addAllowedMethod("*");
config.addAllowedOrigin("*");
config.setAllowCredentials(true);
config.setMaxAge(3600L);

source.registerCorsConfiguration("/**", config);
return source;
}
}

上述鉴权配置使用Spring Security自带的AuthenticationWebFilter类,对其进行定制

1.2 定制ServerAuthenticationConverter,用于匹配并生成所需Authentication(即JwtTokenAuthenticationToken)#

自定义JwtTokenAuthenticationConverter核心代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
public class JwtTokenAuthenticationConverter implements ServerAuthenticationConverter {

@Override
public Mono<Authentication> convert(ServerWebExchange exchange) {
return apply(exchange);
}

public Mono<Authentication> apply(ServerWebExchange exchange) {
ServerHttpRequest request = exchange.getRequest();

String token = request.getHeaders().getFirst(HeaderNames.X_TOKEN);
if (StringUtils.isBlank(token)) {
return Mono.empty();
}

return Mono.just(new JwtTokenAuthenticationToken(token));
}
}

自定义JwtTokenAuthenticationToken核心代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
public class JwtTokenAuthenticationToken extends AbstractAuthenticationToken {

private String token;
private Long userId;

public JwtTokenAuthenticationToken(Collection<? extends GrantedAuthority> authorities) {
super(authorities);
}

public JwtTokenAuthenticationToken(String token) {
super(null);
this.token = token;
}

public void setUserId(Long userId) {
this.userId = userId;
}

public Long getUserId() {
return this.userId;
}

@Override
public Object getCredentials() {
return token;
}

@Override
public Object getPrincipal() {
return null;
}

@Override
public boolean implies(Subject subject) {
return false;
}

1.3 定制ReactiveAuthenticationManager,用于执行鉴权逻辑#

自定义JwtTokenAuthenticationManager核心代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
@Slf4j
@Component
public class JwtTokenAuthenticationManager implements ReactiveAuthenticationManager {

@Autowired
private AuthFeignService authFeignService;

@Override
public Mono<Authentication> authenticate(Authentication authentication) throws AuthenticationException {
JwtTokenAuthenticationToken requestToken = (JwtTokenAuthenticationToken) authentication;
String token = (String) requestToken.getCredentials();

AuthDto authDto = new AuthDto();
authDto.setToken(token);
// 服务级别只认证,不做授权逻辑判断
authDto.setOnlyAuthenticate(true);
ResultVo<AuthVo> resultVo = authFeignService.auth(authDto);
if (resultVo.getCode() != ErrorCodeEnum.SUCCESS.getCode() || !resultVo.getData().getAuthenticated()) {
log.error("调用auth服务鉴权失败!");
return Mono.error(new BadCredentialsException("调用auth服务鉴权失败!"));
}

// set authorities
List<GrantedAuthority> authorities = Collections.emptyList();
JwtTokenAuthenticationToken resultToken = new JwtTokenAuthenticationToken(authorities);
resultToken.setAuthenticated(true);
resultToken.setUserId(resultVo.getData().getUserId());

return Mono.just(resultToken);
}
}

1.4 定制ServerSecurityContextRepository,用于存取SecurityContext(安全上下文)#

由于Spring Cloud Gateway基于WebFlux构建,有别于javax servlet模型,不可使用基于ThreadLocal的SecurityContextHolder存取SecurityContext
可以使用Spring Security的基于session的WebSessionServerSecurityContextRepository类
代码见源码:WebSessionServerSecurityContextRepository

1.5 清除安全上下文#

由于访问凭证使用的是jwt token方案,而Spring Security的WebSessionServerSecurityContextRepository会使用到session
出现一个问题,当执行一次请求,触发鉴权操作,会生成一个session,即使jwt token失效后或者不带上jwt token,浏览器会自动带上sessionid,而服务端的session可能还未失效,这时该次请求会从session中获取缓存的已认证通过的Authentication
解决方案之一:使session和jwt token的过期时间保持一致,配置Spring Security的logout以使调用登出接口时清除相应SecurityContext
解决方案之二:自定义ServerSecurityContextRepository,避开session
解决方案之三:每次请求结束前清除相应的SecurityContext

此处使用方案三,核心代码如下:

1
2
3
4
5
6
7
8
9
public class ClearExchangeSecurityContextWebFilter implements WebFilter {

private ServerSecurityContextRepository securityContextRepository = new WebSessionServerSecurityContextRepository();

@Override
public Mono<Void> filter(ServerWebExchange exchange, WebFilterChain chain) {
return chain.filter(exchange).then(securityContextRepository.save(exchange, null));
}
}

1.6 网关向下游服务传递Header参数#

网关做完鉴权操作后,要向下游服务传递所需的(Header)参数,比如userId
使用自定义的Spring Cloud Gateway的GlobalFilter

核心代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
public class GatewayWebContextGlobalFilter implements GlobalFilter {

@Override
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
return ReactiveSecurityContextHolder.getContext()
.map(SecurityContext::getAuthentication)
.filter(authentication -> authentication instanceof JwtTokenAuthenticationToken)
.map(authentication -> (JwtTokenAuthenticationToken) authentication)
.flatMap(authentication -> {
String headerName = HeaderNames.X_USER_ID;
String headerValue = String.valueOf(authentication.getUserId());

ServerHttpRequest request = exchange.getRequest().mutate().header(headerName, headerValue).build();

return chain.filter(exchange.mutate().request(request).build());
})
.switchIfEmpty(chain.filter(exchange));
}
}

配置自定义GlobalFilter,核心代码如下:

1
2
3
4
5
6
7
8
@Configuration
public class GatewayConfiguration {

@Bean
public GatewayWebContextGlobalFilter gatewayWebContextGlobalFilter() {
return new GatewayWebContextGlobalFilter();
}
}

2.下游服务集成Spring Security#

下游服务做接口方法级别的鉴权
需要开启@EnableGlobalMethodSecurity
需要鉴权的接口方法添加@PreAuthorize注解

2.1 Spring Security核心配置#

核心代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
@EnableGlobalMethodSecurity(prePostEnabled = true)
@EnableWebSecurity
public class WebSecurityConfiguration extends WebSecurityConfigurerAdapter {

@Autowired
private JwtTokenAuthenticationProvider jwtTokenAuthenticationProvider;

@Override
protected void configure(HttpSecurity http) throws Exception {
http.csrf().disable();
http.cors().disable();

http.addFilterAfter(new JwtTokenAuthenticationFilter(authenticationManagerBean()), LogoutFilter.class);
}

/**
* 如果重写了authenticationManagerBean(),需要同时重写该方法
*
* @see WebSecurityConfigurerAdapter#authenticationManagerBean()
*/
@Override
protected void configure(AuthenticationManagerBuilder auth) {
auth.authenticationProvider(jwtTokenAuthenticationProvider);
}

@Bean
@Override
public AuthenticationManager authenticationManagerBean() throws Exception {
return super.authenticationManagerBean();
}
}

2.2 自定义Authentication#

核心代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
public class JwtTokenAuthenticationToken extends AbstractAuthenticationToken {

private String token;

public JwtTokenAuthenticationToken(Collection<? extends GrantedAuthority> authorities) {
super(authorities);
}

public JwtTokenAuthenticationToken(String token) {
super(null);
this.token = token;
}

@Override
public Object getCredentials() {
return token;
}

@Override
public Object getPrincipal() {
return null;
}

@Override
public boolean implies(Subject subject) {
return false;
}
}

2.3 自定义Filter#

核心代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
@Slf4j
public class JwtTokenAuthenticationFilter extends OncePerRequestFilter {

private AuthenticationManager authenticationManager;

public JwtTokenAuthenticationFilter(AuthenticationManager authenticationManager) {
this.authenticationManager = authenticationManager;
}

@Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
throws ServletException, IOException {

logger.debug("doFilterInternal()");

// If required, authenticate and set web context
authenticateIfRequired(request);

filterChain.doFilter(request, response);

// Clear web context
WebContextUtils.clear();
}

/**
* Authenticate if required
*/
private void authenticateIfRequired(HttpServletRequest request) {
String token = request.getHeader(Common.TOKEN_HEADER_KEY);
if (StringUtils.isBlank(token)) {
log.debug("no token, as anonymous in later AnonymousAuthenticationFilter");
// // 在过滤链AnonymousAuthenticationFilter中设置为匿名用户
// // 这里要先clear,否则在匿名Filter中会存在JwtTokenAuthenticationToken;具体原因暂时未知
SecurityContextHolder.clearContext();
return;
}

// wrap request authentication
Authentication authRequest = new JwtTokenAuthenticationToken(token);
// authenticate for result (delegate by ProviderManager)
Authentication authResult = authenticationManager.authenticate(authRequest);
// set to security context
SecurityContextHolder.getContext().setAuthentication(authResult);
}
}

2.4 自定义AuthenticationProvider#

在网关已经做了认证,此处主要是做授权

核心代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
@Slf4j
@Component
public class JwtTokenAuthenticationProvider implements AuthenticationProvider {

@Autowired
private AuthFeignService authFeignService;

@Override
public Authentication authenticate(Authentication authentication) throws AuthenticationException {
JwtTokenAuthenticationToken requestToken = (JwtTokenAuthenticationToken) authentication;
String token = (String) requestToken.getCredentials();

AuthDto authDto = new AuthDto();
authDto.setToken(token);
ResultVo<AuthVo> resultVo = authFeignService.auth(authDto);
if (resultVo.getCode() != ErrorCodeEnum.SUCCESS.getCode() || !resultVo.getData().getAuthenticated()) {
log.error("调用auth服务鉴权失败!");
throw new BaseException(ErrorCodeEnum.AUTH_FAIL, "调用auth服务鉴权失败!");
}

// set web context
WebContextUtils.setToken(token);
WebContextUtils.setUserId(resultVo.getData().getUserId());

AuthVo authVo = resultVo.getData();
// role
List<String> listAuthority = authVo.getRoles().stream().map(s -> "ROLE_" + s).collect(Collectors.toList());
// add resource
listAuthority.addAll(authVo.getResources());
String commaSeparatedRoles = String.join(",", listAuthority);

// set authorities
List<GrantedAuthority> authorities = AuthorityUtils.commaSeparatedStringToAuthorityList(commaSeparatedRoles);
// new authentication
JwtTokenAuthenticationToken resultToken = new JwtTokenAuthenticationToken(authorities);
resultToken.setAuthenticated(true);

return resultToken;
}

@Override
public boolean supports(Class<?> authentication) {
return JwtTokenAuthenticationToken.class.isAssignableFrom(authentication);
}
}

2.5 网关向下游服务传递Header参数,此处做提取并放置到WebContext(Utils)中#

核心代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
@Slf4j
public class ServiceWebContextFilter extends OncePerRequestFilter {

@Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
throws ServletException, IOException {

String token = request.getHeader(HeaderNames.X_TOKEN);
String userId = request.getHeader(HeaderNames.X_USER_ID);

log.debug("WebContext.token: {}", token);
log.debug("WebContext.userId: {}", userId);

if (token != null) {
WebContextUtils.setToken(token);
}
if (userId != null) {
WebContextUtils.setUserId(Long.parseLong(userId));
}

filterChain.doFilter(request, response);
}
}

配置提取Header参数的Filter,核心代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
@Configuration
public class ContextConfiguration {

@Bean
public FilterRegistrationBean<ServiceWebContextFilter> serviceWebContextFilter() {
FilterRegistrationBean<ServiceWebContextFilter> registrationBean = new FilterRegistrationBean<>();

registrationBean.setFilter(new ServiceWebContextFilter());
registrationBean.setOrder(Ordered.LOWEST_PRECEDENCE);

return registrationBean;
}
}

3.网关跨域的问题#

Spring Cloud Gateway集成Spring Security后出现跨域(Gateway配置中已经开启跨域配置并允许所有)

解决方案:
Spring Security也添加跨域配置

1
2
3
4
5
6
7
8
9
10
11
12
@Bean
public CorsConfigurationSource corsConfigurationSource() {
UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
CorsConfiguration config = new CorsConfiguration();
config.addAllowedHeader("*");
config.addAllowedMethod("*");
config.addAllowedOrigin("*");
config.setAllowCredentials(true);
config.setMaxAge(3600L);
source.registerCorsConfiguration("/**", config);
return source;
}

仅添加以上配置即可(Spring Security会获取到),也可以额外显式设置一下,在 SecurityWebFilterChain 配置中:
http.cors().configurationSource(corsConfigurationSource());

以上解决方案的原因:
按照网上的说法,Gateway的跨域处理较Spring Security靠后,Spring Security发现跨域问题后直接响应了
但是对Spring Security的cors进行关闭也无效:http.cors().disable();
具体细节暂不清楚

4.网关OpenFeign调用错误的问题#

错误信息:feign.codec.EncodeException: No qualifying bean of type ‘org.springframework.boot.autoconfigure.http.HttpMessageConverters’ available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}
原因:OpenFeign与基于WebFlux的Gateway不能完美适配导致报错
解决方案:

1
2
3
4
5
@Bean
@ConditionalOnMissingBean
public HttpMessageConverters messageConverters(ObjectProvider<HttpMessageConverter<?>> converters) {
return new HttpMessageConverters(converters.orderedStream().collect(Collectors.toList()));
}

按照官方说法,以上解决方案有缺陷,因为HttpMessageConvert会阻塞,可能会造成WebFlux应用中断
https://github.com/spring-cloud/spring-cloud-openfeign/issues/235

Spring官方目前推荐的做法是使用第三方的替代方案(feign-reactive https://github.com/Playtika/feign-reactive)
https://cloud.spring.io/spring-cloud-openfeign/reference/html/#reactive-support

由于不熟悉后者的reactive编程,所以暂时仍然使用前者

5.版本#

  • Spring Cloud Gateway 2.2.5
  • Spring Security 5.3.5