流量回放是什么意思- 畅鱼网

一、Goreplay 介绍

Goreplay 使用 Golang 写的一个 HTTP 实时流量复制工具。功能更强大，支持流量的放大、缩小，频率限制，还支持把请求记录到文件，方便回放和分析，也支持和 ElasticSearch 集成，将流量存入 ES 进行实时分析。
GoReplay 不是代理，而是监听网络接口上的流量，不需要更改生产基础架构，而是在与服务相同的计算机上运行 GoReplay 守护程序。
特点：简单易用
地址：***/buger/goreplay
stars：14.1k

与 TCPCopy 相比它的架构更简单，只有一个 gor 组件，如下：

只需要在生产服务器上启动一个 gor 进程，它负责所有的工作包括监听、过滤和转发。它的设计遵循 Unix 设计哲学：一切都是由管道组成的，各种输入将数据复用为输出。

二、Golang环境安装

一、首先安装Golang及相关依赖环境在官网上下载安装包或者go的一些中文网站上下载安装包

***/dl/
***/dl

我下载的是: go1.15.5.linux-amd64.tar.gz

二、解压到/usr/local目录下

tar -C /usr/local -zxvf go1.14.4.linux-amd64.tar.gz

三、配置环境变量

# 打开vim /etc/profile# 添加export GOROOT=/usr/local/goexport PATH=$PATH:$GOROOT/bin# 编译生效source /etc/profile

四、验证

go env

[root@vm ~]# go versiongo version go1.15.5 linux/amd64

三、Goreplay 安装

从***/buger/gor/releases 下载最新的 Gor 二进制文件（提供 Windows、Linux x64 和 Mac OS 的预编译二进制文件），也可以自己编译。

下载二进制包：

[root@vm-1 ~]# curl -L -O ***/buger/goreplay/releases/download/v1.3.1/gor_1.3.1_x64.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 626 100 626 0 0 741 0 --:--:-- --:--:-- --:--:-- 741100 10.5M 100 10.5M 0 0 3258k 0 0:00:03 0:00:03 --:--:-- 5951k[root@vm-1 ~]# tar xvzf gor_1.3.1_x64.tar.gz gor

解压缩包后，您可以从当前目录运行 Gor，或者您可能希望将二进制文件复制到您的 PATH（对于 Linux 和 Mac OS，它可以是 /usr/local/bin）。

四、Goreplay 使用示例

1、准备 RESTful API 环境

接口设计：

请求类型 请求路径 功能 Get /person 查询所有人 Post /person/two 事务测试

打包程序分别放到两台服务器 run 起来。

服务器一：

[root@vm-1 ~]# java -jar spirng-boot-demo-0.0.1-SNAPSHOT.jar . ____ _ __ _ _ /\ / ___'_ __ _ _(_)_ __ __ _    ( ( )___ | '_ | '_| | '_ / _` |     \/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |___, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.3.8.RELEASE)2021-08-17 14:19:10.467 WARN 17537 --- [ main] o.s.boot.StartupInfoLogger : InetAddress.getLocalHost().getHostName() took 10014 milliseconds to respond. Please verify your network configuration.2021-08-17 14:19:20.494 INFO 17537 --- [ main] com.zuozewei.SpirngbootdemoApplication : Starting SpirngbootdemoApplication v0.0.1-SNAPSHOT on vm-jmeter with PID 17537 (/root/spirng-boot-demo-0.0.1-SNAPSHOT.jar started by root in /root)2021-08-17 14:19:20.495 INFO 17537 --- [ main] com.zuozewei.SpirngbootdemoApplication : The following profiles are active: a2021-08-17 14:19:22.225 INFO 17537 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.2021-08-17 14:19:22.355 INFO 17537 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 111ms. Found 1 JPA repository interfaces.2021-08-17 14:19:23.642 INFO 17537 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8082 (*** 17537 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]2021-08-17 14:19:23.667 INFO 17537 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.41]2021-08-17 14:19:23.804 INFO 17537 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext2021-08-17 14:19:23.805 INFO 17537 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 3179 ms2021-08-17 14:19:24.314 INFO 17537 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default]2021-08-17 14:19:24.479 INFO 17537 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.27.Final2021-08-17 14:19:25.002 INFO 17537 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}2021-08-17 14:19:25.228 INFO 17537 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...2021-08-17 14:19:25.635 INFO 17537 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.2021-08-17 14:19:25.706 INFO 17537 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL57Dialect2021-08-17 14:19:27.086 INFO 17537 --- [ main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]2021-08-17 14:19:27.112 INFO 17537 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'2021-08-17 14:19:27.990 WARN 17537 --- [ main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning2021-08-17 14:19:28.261 INFO 17537 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'2021-08-17 14:19:28.749 INFO 17537 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8082 (*** 17537 --- [ main] com.zuozewei.SpirngbootdemoApplication : Started SpirngbootdemoApplication in 39.553 seconds (JVM running for 40.431)

服务器二：

[root@vm-2 ~]# java -jar spirng-boot-demo-0.0.1-SNAPSHOT.jar . ____ _ __ _ _ /\ / ___'_ __ _ _(_)_ __ __ _    ( ( )___ | '_ | '_| | '_ / _` |     \/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |___, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.3.8.RELEASE)2021-08-17 14:18:34.814 INFO 12775 --- [ main] com.zuozewei.SpirngbootdemoApplication : Starting SpirngbootdemoApplication v0.0.1-SNAPSHOT on vm-hsyjy-znkzb-vbi-1.novalocal with PID 12775 (/root/spirng-boot-demo-0.0.1-SNAPSHOT.jar started by root in /root)2021-08-17 14:18:34.820 INFO 12775 --- [ main] com.zuozewei.SpirngbootdemoApplication : The following profiles are active: a2021-08-17 14:18:36.293 INFO 12775 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.2021-08-17 14:18:36.419 INFO 12775 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 107ms. Found 1 JPA repository interfaces.2021-08-17 14:18:37.694 INFO 12775 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8082 (*** 12775 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]2021-08-17 14:18:37.717 INFO 12775 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.41]2021-08-17 14:18:37.841 INFO 12775 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext2021-08-17 14:18:37.841 INFO 12775 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2886 ms2021-08-17 14:18:38.305 INFO 12775 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default]2021-08-17 14:18:38.444 INFO 12775 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.27.Final2021-08-17 14:18:38.864 INFO 12775 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}2021-08-17 14:18:39.119 INFO 12775 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...2021-08-17 14:18:39.556 INFO 12775 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.2021-08-17 14:18:39.648 INFO 12775 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL57Dialect2021-08-17 14:18:41.052 INFO 12775 --- [ main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]2021-08-17 14:18:41.086 INFO 12775 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'2021-08-17 14:18:42.042 WARN 12775 --- [ main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning2021-08-17 14:18:42.330 INFO 12775 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'2021-08-17 14:18:42.768 INFO 12775 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8082 (*** 12775 --- [ main] com.zuozewei.SpirngbootdemoApplication : Started SpirngbootdemoApplication in 8.988 seconds (JVM running for 9.939)

2、捕获服务器一流量保存到本地文件

在终端运行这个命令：

#将端口 8082 流量保存到本地的文件sudo ./gor --input-raw :8082 --output-file=requests.gor

[root@vm-1 ~]# sudo ./gor --input-raw :8082 --output-file=requests.gorInterface: eth0 . BPF Filter: ((tcp dst port 8082) and (dst host 172.16.106.149 or dst host fe80::f599:6e65:3dc9:b84a))Interface: lo . BPF Filter: ((tcp dst port 8082) and (dst host 127.0.0.1 or dst host ::1))Interface: virbr0 . BPF Filter: ((tcp dst port 8082) and (dst host 192.168.124.1))2021/08/17 14:26:41 [PPID 17654 and PID 17655] Version:1.3.0

注意：

使用–output-file-append 选项可以将捕获流量保存为一个单独的文件使用sudo并要求输入密码：为了分析网络，Gor 需要只有超级用户才能使用的权限。但是，可以为非 root 用户配置运行Gor 。

在 Postman 上向服务器一发送请求：

我们查看服务一日志：

Hibernate: select person0_.id as id1_0_, person0_.age as age2_0_, person0_.name as name3_0_ from person person0_Hibernate: select next_val as id_val from hibernate_sequence for updateHibernate: update hibernate_sequence set next_val= ? where next_val=?Hibernate: select next_val as id_val from hibernate_sequence for updateHibernate: update hibernate_sequence set next_val= ? where next_val=?Hibernate: insert into person (age, name, id) values (?, ?, ?)Hibernate: insert into person (age, name, id) values (?, ?, ?)

可以看到服务已经正常处理了。

我们停掉在服务器一上运行的 Gor，可以看到已经生成了录制的请求文件：

[root@vm-1 ~]# ls -l | grep "requests"-rw-r----- 1 root root 300 8月 17 14:33 requests_0.gor

我们看下请求文件的内容：

[root@vm-1 ~]# cat requests_0.gor 1 04481f92ac106afb42ce55ca 1629182282437724065 0GET /person ***ept: */*Cache-Control: no-cachePostman-Token: ab405d20-ef93-4ffb-b2f6-a16cf1205e7cHost: 172.16.106.149:8082Accept-Encoding: gzip, deflate, brConnection: keep-alive

3、将流量从文件回放到服务器二

现在是时候将原始流量回放到服务二中了。如果是多个请求并且它们将按照与记录的顺序和时间完全相同的顺序重放。

–output-http 提供第二个服务的 URL

sudo ./gor --input-file requests_0.gor --output-http="http://172.16.106.237:8082"

[root@vm-1 ~]sudo ./gor --input-file requests_0.gor --output-http="http://172.16.106.237:8082"2021/08/17 14:42:55 [PPID 12356 and PID 17914] Version:1.3.0[DEBUG][elapsed 985.792µs]: [INPUT-FILE] FileInput: end of file 'requests_0.gor'

我们看到服务二的日志：

[root@vm-2 ~]# java -jar spirng-boot-demo-0.0.1-SNAPSHOT.jar . ____ _ __ _ _ /\ / ___'_ __ _ _(_)_ __ __ _    ( ( )___ | '_ | '_| | '_ / _` |     \/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |___, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.3.8.RELEASE)2021-08-17 14:18:34.814 INFO 12775 --- [ main] com.zuozewei.SpirngbootdemoApplication : Starting SpirngbootdemoApplication v0.0.1-SNAPSHOT on vm-hsyjy-znkzb-vbi-1.novalocal with PID 12775 (/root/spirng-boot-demo-0.0.1-SNAPSHOT.jar started by root in /root)......Hibernate: select person0_.id as id1_0_, person0_.age as age2_0_, person0_.name as name3_0_ from person person0_

日志说明服务正常响应。

4、实时将流量请求打印到终端

通过命令：

sudo ./gor --input-raw :8082 --output-stdout

此命令其记录到标准控制台输出，请注意，默认情况下 GoReplay 不跟踪响应，您可以使用 –output-http-track-response 选项启用它们。

[root@vm-1 ~]sudo ./gor --input-raw :8082 --output-stdoutInterface: eth0 . BPF Filter: ((tcp dst port 8082) and (dst host 172.16.106.149 or dst host fe80::f599:6e65:3dc9:b84a))Interface: lo . BPF Filter: ((tcp dst port 8082) and (dst host 127.0.0.1 or dst host ::1))Interface: virbr0 . BPF Filter: ((tcp dst port 8082) and (dst host 192.168.124.1))2021/08/17 14:55:45 [PPID 18060 and PID 18061] Version:1.3.01 22ee1f92ac106afb594d205a 1629183362175214036 0POST /person/two ***ept: */*Cache-Control: no-cachePostman-Token: 4922ce23-5706-42c5-8a54-ade5d5aee785Host: 172.16.106.149:8082Accept-Encoding: gzip, deflate, brConnection: keep-aliveContent-Length: 0

5、复制流量实时转发到服务器二

运行命令：

sudo ./gor --input-raw :8082 --output-http="http://172.16.106.237:8082"

[root@vm-1 ~]# sudo ./gor --input-raw :8082 --output-http="http://172.16.106.237:8082"Interface: eth0 . BPF Filter: ((tcp dst port 8082) and (dst host 172.16.106.149 or dst host fe80::f599:6e65:3dc9:b84a))Interface: lo . BPF Filter: ((tcp dst port 8082) and (dst host 127.0.0.1 or dst host ::1))Interface: virbr0 . BPF Filter: ((tcp dst port 8082) and (dst host 192.168.124.1))2021/08/17 14:59:49 [PPID 18118 and PID 18119] Version:1.3.0

在服务器二查看日志,.发现流量已经进来了：

Hibernate: select person0_.id as id1_0_, person0_.age as age2_0_, person0_.name as name3_0_ from person person0_Hibernate: select person0_.id as id1_0_, person0_.age as age2_0_, person0_.name as name3_0_ from person person0_Hibernate: select person0_.id as id1_0_, person0_.age as age2_0_, person0_.name as name3_0_ from person person0_ where person0_.age=?

6、压力测试

goreplay支持将捕获到的生产实际请求流量减少或者放大重播以用于测试环境的压力测试.压力测试一般针对 Input 流量减少或者放大。

录制的请求文件：

[root@vm-1 ~]# cat requests.gor 1 040f1f92ac106afb7e8d5679 1629183084044026882 0POST /person/two ***ept: */*Cache-Control: no-cachePostman-Token: a8600dd1-28c1-4825-b6a2-68c42659942fHost: 172.16.106.149:8082Accept-Encoding: gzip, deflate, brConnection: keep-aliveContent-Length: 0[root@vm-jmeter ~]# cat requests.gor 1 040f1f92ac106afb7e8d5679 1629183084044026882 0POST /person/two ***ept: */*Cache-Control: no-cachePostman-Token: a8600dd1-28c1-4825-b6a2-68c42659942fHost: 172.16.106.149:8082Accept-Encoding: gzip, deflate, brConnection: keep-aliveContent-Length: 0

我们可以看到主要是两个请求。

运行以下命令，将流量从文件回放到服务器二，并放大两倍：

[root@vm-1 ~]sudo ./gor --input-file "requests.gor|200%" --output-http="http://172.16.106.237:8082"2021/08/17 15:03:58 [PPID 12356 and PID 18187] Version:1.3.0[DEBUG][elapsed 1.361742ms]: [INPUT-FILE] FileInput: end of file 'requests.gor'

我们查看下服务二日志的处理情况：

Hibernate: select next_val as id_val from hibernate_sequence for updateHibernate: update hibernate_sequence set next_val= ? where next_val=?Hibernate: select next_val as id_val from hibernate_sequence for updateHibernate: update hibernate_sequence set next_val= ? where next_val=?Hibernate: insert into person (age, name, id) values (?, ?, ?)Hibernate: insert into person (age, name, id) values (?, ?, ?)

我们可以看到服务处理了四个请求。

注意，当然也支持请求流量 10%,20% 等缩小。

五、总结

今天简单介绍 Gor的常见玩法，事实上远不止上述提到的这些，更多的玩法有时间再写。

文件源码：