pinpoint 插件开发

发表于 | 分类于

Pinpoint插件开发

一、技术概述

1. 架构组成

pinpoint-architecture]
pinpoint 主要由 3 个组件外加 Hbase 数据库构成，三个组件分别为：pinpoint-Agent、pinpoint-Collector 和 pinpoint-Web。

2. 系统特色

  1. 分布式追踪，追踪分布式系统中穿梭的消息
  2. 自动侦测应用程序拓扑，以帮助指明应用程序的配置
  3. 横向扩展以支持大规模的服务器组
  4. 提供代码级别的可见性，以方便识别故障点和瓶颈
  5. 使用字节码注入技术，无需修改代码就可以添加新功能

3. 运行 pinpoint系统

开源APM工具pinpoint线上部署 ,当然文章是生环部署,本地缩减系就可以。

二、 开发插件

1.目标

pinpoint插件只拦截指定应用，该插实现拦任意class中所有方法（动态配置）。

2. 创建maven项目

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.navercorp.pinpoint</groupId>
  <artifactId>pinpoint-plugin-apps</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <name>pinpoint-plugin-apps</name>
  <description>pinpoint-plugin-apps</description>
  <properties>
        <encoding>UTF-8</encoding>
        <jdk.version>1.6</jdk.version>
        <jdk.home>${env.JAVA_6_HOME}</jdk.home>
       <!--   <pinpoint.version>1.7.0-SNAPSHOT</pinpoint.version>-->
         <pinpoint.version>1.7.1</pinpoint.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.navercorp.pinpoint</groupId>
            <artifactId>pinpoint-bootstrap</artifactId>
            <version>${pinpoint.version}</version>
        </dependency>

        <!-- for test -->
        <dependency>
            <groupId>com.navercorp.pinpoint</groupId>
            <artifactId>pinpoint-test</artifactId>
            <version>${pinpoint.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- for integration test -->
        <dependency>
            <groupId>com.navercorp.pinpoint</groupId>
            <artifactId>pinpoint-profiler</artifactId>
            <version>${pinpoint.version}</version>
            <scope>runtime</scope>
        </dependency>
        <!-- <dependency>
            <groupId>com.navercorp.pinpoint</groupId>
            <artifactId>pinpoint-profiler-optional</artifactId>
            <version>${pinpoint.version}</version>
            <scope>runtime</scope>
        </dependency> -->
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.0</version>
                <configuration>
                    <source>${jdk.version}</source>
                    <target>${jdk.version}</target>
                    <fork>true</fork>
                    <debug>true</debug>
                    <optimize>true</optimize>
                    <encoding>${encoding}</encoding>
                    <showDeprecation>true</showDeprecation>
                    <compilerVersion>${jdk.version}</compilerVersion>
                    <executable>${jdk.home}/bin/javac</executable>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.17</version>
                <configuration>
                    <excludes>
                        <exclude>**/Mock*</exclude>
                        <exclude>**/Abstract*</exclude>
                        <exclude>**/*Helper</exclude>
                        <exclude>**/*$*</exclude>
                    </excludes>
                    <argLine>-Dfile.encoding=${encoding}</argLine>
                    <jvm>${jdk.home}/bin/java</jvm>
                    <forkMode>once</forkMode>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.5.3</version>
                <configuration>
                    <descriptors>
                        <descriptor>src/integration-test/assembly/assembly.xml</descriptor>
                    </descriptors>
                    <outputDirectory>${project.build.directory}</outputDirectory>
                    <appendAssemblyId>false</appendAssemblyId>
                </configuration>
                <executions>
                    <execution>
                        <id>integration-test-agent</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>            
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>2.18.1</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <!-- AbstractPinpointPluginTestSuite needs this to resolve path of required jars -->
                    <useSystemClassLoader>false</useSystemClassLoader>
                    <failIfNoTests>true</failIfNoTests>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

3. 插件工作原理

插件原理：在启动的时候扫描webapp目录所有的class然后正则匹配，然后获取class所有方法注册到agent中。

4. 定义spi实现类

pinpiont使用标准spi，需要在META-INF 添加如下文件

1
2
META-INF/services/com.navercorp.pinpoint.bootstrap.plugin.ProfilerPlugin
META-INF/services/com.navercorp.pinpoint.common.trace.TraceMetadataProvider

5. 实现ProfilerPlugin

核心代码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
//com.navercorp.pinpoint.plugin.apps.AppsPlugin
/**
 * @author byzy
 */
public class AppsPlugin implements ProfilerPlugin, TransformTemplateAware {
    private final PLogger logger = PLoggerFactory.getLogger(this.getClass());
    private TransformTemplate transformTemplate;

    @Override
    public void setTransformTemplate(TransformTemplate transformTemplate) {
        this.transformTemplate = transformTemplate;
    }
    /*
     *
     * @see com.navercorp.pinpoint.bootstrap.plugin.ProfilerPlugin#setUp(com.navercorp.pinpoint.bootstrap.plugin.ProfilerPluginSetupContext)
     */
    @Override
    public void setup(ProfilerPluginSetupContext context) {
        final AppsConfig config = new AppsConfig(context.getConfig());
        if (logger.isInfoEnabled()) {
            logger.info("TomcatAppsPlugin config:{}", config);
        }
        if (!config.isTomcatEnable()) {
            logger.info("TomcatAppsPlugin disabled");
            return;
        }
        if(null==config.getSearchClassNamePattern()) {
            logger.error("search ClassName is null, skip this plugin");
            return;
        }
        addWebApps(config);

   }
   private void addWebApps(final AppsConfig config) {
        List<String> allClassName = getAllClassName(config);
        if (null == allClassName || 0 == allClassName.size()) {
            logger.warn("search ClassName not find target ,skip this plugin");
            return;
        }
        final String interceptorName = "com.navercorp.pinpoint.plugin.apps.interceptor.AppsInterceptor";
        for (String calssName : allClassName) {
            if (logger.isDebugEnabled()) {
                logger.debug("process class name:{} ", calssName);
            }
            TransformCallback transformer = new TransformCallback() {
                @Override
                public byte[] doInTransform(Instrumentor instrumentor, ClassLoader classLoader, String className, Class<?> classBeingRedefined, ProtectionDomain protectionDomain, byte[] classfileBuffer) throws InstrumentException {
                    InstrumentClass target = instrumentor.getInstrumentClass(classLoader, className, classfileBuffer);
                    List<InstrumentMethod> declaredMethods = target.getDeclaredMethods();
                    for (InstrumentMethod m : declaredMethods) {
                        if (logger.isDebugEnabled()) {
                            logger.debug("register class:{} metnName:{}", target.getName(), m.getName());
                        }
                        m.addScopedInterceptor(interceptorName, target.getName());
                    }

                    return target.toBytecode();
                }

            };

            transformTemplate.transform(calssName, transformer);
        }

    }
}

文件META-INF/services/com.navercorp.pinpoint.bootstrap.plugin.ProfilerPlugin添上class全名com.navercorp.pinpoint.plugin.apps.AppsPlugin
扫描文件目录核心代码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
//com.navercorp.pinpoint.plugin.apps.AppsPlugin
private List<String>  getAllClassName(final AppsConfig config){
       List<String> ret=new ArrayList<String>();
       List<String> webappsPaths = config.getWebappsPaths();
       if(null==webappsPaths || webappsPaths.size()==0) {
           String catHome = SystemProperty.INSTANCE.getProperty(AppConstants.CATALINA_HOME);
           String webappPath=catHome+"/webapps/";
           logger.debug("cat_home:{}  webappPath:{}", catHome,webappPath);
           File file = new File(webappPath);
           File[] files = file.listFiles();
           for (File f : files) {
               if (f.isDirectory()) {
                   File app =new File(f.getPath()+"/WEB-INF/classes/");
                   String libPath=f.getPath()+"/WEB-INF/lib/";
                   if(app.exists()) {
                       SearchClass search=new SearchClass.Builder()                        //
                                                           .classRootPath(app.getPath())       //
                                                           .calssPattern(config.getSearchClassNamePattern())       //
                                                           .jarsPaths(Arrays.asList(libPath))                  //
                                                           .build();                               //
                       try {
                           List<String> searchClassName = search.searchClassName();
                           List<String> searchJarClassName = search.searchJarClassName();
                           ret.addAll(searchClassName);
                           ret.addAll(searchJarClassName);
                       }
                       catch (Exception e) {
                          logger.error("serach class error",e);
                       }
                   }
               }
           }

       }else {
           //TODO-------------
           logger.warn(" not supper webapps param");

       }
       return ret;
   }

6. 添加元数据类AppsTypeProvider

核心代码

1
2
3
4
5
6
7
8
9
10
11
12
//com.navercorp.pinpoint.plugin.apps.AppsTypeProvider
/**
 * @author byzy
 *
 */
public class AppsTypeProvider implements TraceMetadataProvider {
    @Override
    public void setup(TraceMetadataSetupContext context) {
        context.addServiceType(AppConstants.APPS_PLUGIN_TYPE,AnnotationKeyMatchers.exact(AppConstants.APPS_ANNOTATION_KEY_VAL_INFO));

    }
}

添加文件META-INF/services/com.navercorp.pinpoint.common.trace.TraceMetadataProvider内容为com.navercorp.pinpoint.plugin.apps.AppsTypeProvider

7. 编写拦截功能

核心代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
//com.navercorp.pinpoint.plugin.apps.interceptor.AppsInterceptor
public class AppsInterceptor implements AroundInterceptor {
    private final PLogger logger = PLoggerFactory.getLogger(this.getClass());

    private final TraceContext traceContext;
    private final MethodDescriptor descriptor;

    /**  
    *
    *
    * @param traceContext
    * @param descriptor  
    */
    public AppsInterceptor(TraceContext traceContext, MethodDescriptor descriptor) {
        super();
        this.traceContext = traceContext;
        this.descriptor = descriptor;
    }
    @Override
    public void before(Object target, Object[] args) {

    }

    /*
    * <p> after</p>
    *
    * @param target
    * @param args
    * @param result
    * @param throwable  
    * @see com.navercorp.pinpoint.bootstrap.interceptor.AroundInterceptor#after(java.lang.Object, java.lang.Object[], java.lang.Object, java.lang.Throwable)  
    */

    @Override
    public void after(Object target, Object[] args, Object result, Throwable throwable) {

        if (logger.isDebugEnabled()) {
            logger.afterInterceptor(target, args);
        }
        if (logger.isDebugEnabled()) {
            logger.debug("after meth:{}", descriptor.getMethodName());
        }
        Trace trace = traceContext.currentTraceObject();
        if (trace == null) {
            return;
        }
        try {
            StringBuilder values = new StringBuilder();
            if (null != args) {
                int index = 0;
                for (Object arg : args) {
                    if (null != arg) {
                        values.append("[arge_");
                        values.append(index);
                        values.append("]:[");
                        values.append(arg.toString());
                        values.append("]");
                    }
                    index++;
                }
            }
            final SpanEventRecorder recorder = trace.currentSpanEventRecorder();
            recorder.recordServiceType(AppConstants.APPS_PLUGIN_TYPE);
            recorder.recordApi(descriptor);
            recorder.recordAttribute(AppConstants.APPS_ANNOTATION_KEY_VAL_INFO, values.toString());
            recorder.recordException(throwable);
        }
        finally {
            trace.traceBlockEnd();
        }
    }
}

8. 编写配置类常量

核心代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
//com.navercorp.pinpoint.plugin.apps.AppConstants
private AppConstants() {
  }

  /*
  ServiceType Code 全部范围

  类型  范围
  Internal Use    0 ~ 999
  Server  1000 ~ 1999
  DB Client   2000 ~ 2999
  Cache Client    8000 ~ 8999
  RPC Client  9000 ~ 9999
  Others  5000 ~ 7999
  ServiceType Code 私有区域范围
  类型  范围
  Server  1900 ~ 1999
  DB Client   2900 ~ 2999
  Cache Client    8900 ~ 8999
  RPC Client  9900 ~ 9999
  Others  7500 ~ 7999
  AnnotationKey:
  you may pick a value between 900 to 999 safely to use as AnnotationKey code.
  */
  //TERMINAL
  public static final ServiceType APPS_PLUGIN_TYPE = ServiceTypeFactory.of(7501, "APPS-Plugin");
  public static final com.navercorp.pinpoint.common.trace.AnnotationKey APPS_ANNOTATION_KEY_VAL_INFO =com.navercorp.pinpoint.common.trace.AnnotationKeyFactory.of(901, "apps.args");
  // app__scope
  public static  final String   APPS_SCOPE="apps_scope";
  public static final String CATALINA_HOME = "catalina.home";
}

//com.navercorp.pinpoint.plugin.apps.AppsConfig

/**
 * @author byzy
 */
public class AppsConfig {

    private final boolean      tomcatEnable;
    private final List<String> tomcatBootstrapMains;

    private final String       searchClassNamePattern;

    private final List<String>       webappsPaths;

    private final List<String> classLibJars;

    public AppsConfig(ProfilerConfig config) {
        if (config == null) {
            throw new NullPointerException("config must not be null");
        }

        // plugin
        this.tomcatEnable = config.readBoolean("profiler.apps.enable", true);
        this.tomcatBootstrapMains = config.readList("profiler.apps.bootstrap.main");

        this.searchClassNamePattern = config.readString("profiler.apps.searchClassNamePattern", null);

        this.webappsPaths = config.readList("profiler.apps.webappsPaths");

        this.classLibJars = config.readList("profiler.apps.bootstrap.classLibJars");

    }

  // get set .....
}

9. 搜索类SearchClass

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
//com.navercorp.pinpoint.plugin.apps.search.SearchClass
/**  
 *
* @author byzy   
*/
public class SearchClass {



    public final static ClassFilter classFilter=new ClassFilter();

    public final static JarClassFilter jarClassFilter=new JarClassFilter();

    public final static String CLASS_SUFIX=".class";

    public  final static  int CLASS_SUFIX_LEN=CLASS_SUFIX.length();

    private final  Pattern calssPattern;

    /**
     *  class root path
     */
    private final  String classRootPath;

    /**
     * jars paths
     */
    private final  List<String> jarsPaths;


    /**
     * 搜索匿名类
     */
    private  final boolean anonymousClass;


    public static class Builder{

        private   String calssPattern;

        /**
         *  class root path
         */
        private   String classRootPath;

        /**
         * jars paths
         */
        private   List<String> jarsPaths;


        /**
         * 搜索匿名类
         */
        private  boolean anonymousClass=false;


        /**
         * java 正则
         *
         * @param calssPattern
         * @return
         */
        public Builder calssPattern(String calssPattern) {

            this.calssPattern = calssPattern;
            return this;
        }

        public Builder classRootPath(String classRootPath) {
            this.classRootPath = classRootPath;
            return this;
        }

        public Builder  jarsPaths(List<String> jarsPaths) {
            this.jarsPaths = jarsPaths;
            return this;
        }

        public Builder anonymousClass(boolean anonymousClass) {

            this.anonymousClass=anonymousClass;
            return this;
        }

        public SearchClass build() {
            return new SearchClass(this);
        }



    }

    private SearchClass(Builder builder) {

        if(null==builder.calssPattern || "".equals(builder.calssPattern)) {
            this.calssPattern =null;
        }else {
            this.calssPattern =  Pattern.compile(builder.calssPattern);
        }


        this.classRootPath =  builder.classRootPath;
        this.jarsPaths =  builder.jarsPaths;
        this.anonymousClass=builder.anonymousClass;

    }


    private  boolean checkPackages(final String fullClassNames) {

        if(null==calssPattern) {
            return false;
        }
        Matcher matcher = calssPattern.matcher(fullClassNames);

        return matcher.find();

    }
    public List<String> searchJarClassName() throws Exception {

        List<String> ret = new ArrayList<String>();
        try {

            if(null==jarsPaths) {

                return ret ;
            }

            JarFile jar = null;
            Enumeration<JarEntry> entries = null;
            JarEntry jarEntry = null;

            String className = null;
            String fullClassName = null;

            File file=null;
            final String rep = getSystemFileSeparator();
            File[] tempList=null;
            for (String currentDir : jarsPaths) {

                file= new File(currentDir);
                tempList= file.listFiles(jarClassFilter);

                for (File f : tempList) {

                    if (f.getName().endsWith(".jar")) {

                        jar = new JarFile(f);

                        entries = jar.entries();

                        while (entries.hasMoreElements()) {
                            jarEntry = entries.nextElement();

                            if (jarEntry.isDirectory()) {
                                continue;
                            }

                            String jarClassname = jarEntry.getName();
                            if (!jarClassname.endsWith(CLASS_SUFIX)) {
                                continue;
                            }

                            className = jarClassname.substring(0, jarClassname.length() - CLASS_SUFIX_LEN);
                            fullClassName = className.replaceAll(rep, ".");

                            if(skipClassName(fullClassName)) {

                                continue;
                            }

                            if (checkPackages(fullClassName)) {
                                ret.add(fullClassName);
                            }

                        }

                        // jar
                    }
                }

            }
        }
        catch (Exception e) {

            throw e;
        }

        return ret;
    }
    public List<String> searchClassName() throws Exception {

        List<String> ret = new ArrayList<String>();


        try {

            List<String> dirs = getDirAndSubDirPath(classRootPath);

            final int indexRoot = classRootPath.length();
            final String rep = getSystemFileSeparator();

            String path = null;
            String className = null;
            String fullClassName = null;
            File file =null;
            File[] tempList=null;

            for (String currentDir : dirs) {

                file=  new File(currentDir);
                tempList = file.listFiles(classFilter);

                for (File f : tempList) {

                    if (f.getName().endsWith(CLASS_SUFIX)) {

                        // class
                        path = f.getPath();
                        className = path.substring(indexRoot, path.length() - CLASS_SUFIX_LEN);
                        fullClassName = className.replaceAll(rep, ".");
                        if(fullClassName.startsWith(".")) {
                            fullClassName=fullClassName.substring(1, fullClassName.length());
                        }

                        if(skipClassName(fullClassName)) {

                            continue;
                        }

                        if (checkPackages(fullClassName)) {
                            ret.add(fullClassName);
                        }

                    }

                }
            }

        }
        catch (Exception e) {

            throw e;
        }

        return ret;
    }
    private  List<String> getDirAndSubDirPath(String path) {

        List<String> result = new ArrayList<String>();
        File file = new File(path);
        File[] files = file.listFiles();
        for (File f : files) {

            if (f.isDirectory()) {
                result.add(f.getAbsolutePath());
                List<String> dirSubDir = getDirAndSubDirPath(f.getAbsolutePath());
                if (null != dirSubDir) {
                    result.addAll(dirSubDir);
                }

            }

        }
        return result;
    }


    /**
     *
     *      获取系统文件分割符
     * @return
     */
    private  String getSystemFileSeparator(){

        return System.getProperties().getProperty("file.separator");
    }

    private  boolean skipClassName(final String  fullClassName) {

        if(!anonymousClass) {

           return   fullClassName.indexOf("$") ==-1 ? false:true;
        }

        return false;
    }

}

class ClassFilter implements FilenameFilter {

    @Override
    public boolean accept(File dir, String name) {


        if (name.endsWith(".class")) {

            return true;
        }

        return false;

    }

}

class JarClassFilter implements FilenameFilter {
    @Override
    public boolean accept(File dir, String name) {
        if (name.endsWith(".jar")) {

            return true;
        }
        return false;
    }
}

10. 打包生成插件

1
mvn clean -Dmaven.test.skip package

然后生成的pinpoint-plugin-apps-0.0.1-SNAPSHOT.jar 分别放到Agent的plugin中,然后在pinpoint.config中添加如下配置信息:

1
2
3
4
5
6
###########################################################
# APPS
###########################################################
profiler.apps.enable=true
#search class name
profiler.apps.searchClassNamePattern=com.iqarr.test.[\\s\\S]*

分别将pinpoint-plugin-apps-0.0.1-SNAPSHOT.jar放到pinpoint Collector和pinpoint Web的lib目录下。

11. 测试

重启web，Collector，Agent，创建需要测试的类，com.iqarr.test下的类都添加监控中去，然后去pinpoint上看一下插件结果。