Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

大量配置项场景下应用启动非常慢 #3800

Closed
3 tasks done
Shawyeok opened this issue Jul 1, 2021 · 6 comments · Fixed by #3816
Closed
3 tasks done

大量配置项场景下应用启动非常慢 #3800

Shawyeok opened this issue Jul 1, 2021 · 6 comments · Fixed by #3816

Comments

@Shawyeok
Copy link
Contributor

Shawyeok commented Jul 1, 2021 •
edited
Loading

描述bug
项目中创建了多个namespace（均为properties），由于服务是高度配置化的，因而配置项较多，大约有5000行配置，应用启动时加载配置文件到Bean中很慢，拉长了整个启动过程，等待的过程令人沮丧。

配置项大致长下面这个样子，层级说多不算多，说少也不算少

xxxx.dp.caption.converter[0].convertRules[0].castTo=BYTESTOSTRING
xxxx.dp.caption.converter[0].convertRules[0].sourceField=city_id
xxxx.dp.caption.converter[0].convertRules[0].targetField=cityId
xxxx.dp.caption.converter[0].convertRules[1].castTo=BYTESTOSTRING
xxxx.dp.caption.converter[0].convertRules[1].sourceField=city_name
xxxx.dp.caption.converter[0].convertRules[1].targetField=cityName

项目启动巨慢，给程序员正当理由摸鱼，这样太不好了，启动时抓取了threaddump，如下：

"main@1" prio=5 tid=0x1 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at java.util.HashMap.resize(HashMap.java:735)
	  at java.util.HashMap.putVal(HashMap.java:663)
	  at java.util.HashMap.put(HashMap.java:612)
	  at com.ctrip.framework.apollo.internals.DefaultConfig.stringPropertyNames(DefaultConfig.java:115)
	  at com.ctrip.framework.apollo.internals.DefaultConfig.getPropertyNames(DefaultConfig.java:105)
	  at com.ctrip.framework.apollo.spring.config.ConfigPropertySource.getPropertyNames(ConfigPropertySource.java:24)
	  at org.springframework.core.env.CompositePropertySource.getPropertyNames(CompositePropertySource.java:87)
	  at org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource$CacheKey.get(SpringIterableConfigurationPropertySource.java:214)
	  at org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource.getCache(SpringIterableConfigurationPropertySource.java:134)
	  at org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource.getConfigurationProperty(SpringIterableConfigurationPropertySource.java:75)
	  at org.springframework.boot.context.properties.bind.IndexedElementsBinder.getKnownIndexedChildren(IndexedElementsBinder.java:135)
	  at org.springframework.boot.context.properties.bind.IndexedElementsBinder.bindIndexed(IndexedElementsBinder.java:108)
	  at org.springframework.boot.context.properties.bind.IndexedElementsBinder.bindIndexed(IndexedElementsBinder.java:89)
	  at org.springframework.boot.context.properties.bind.IndexedElementsBinder.bindIndexed(IndexedElementsBinder.java:71)
	  at org.springframework.boot.context.properties.bind.CollectionBinder.bindAggregate(CollectionBinder.java:50)
	  at org.springframework.boot.context.properties.bind.AggregateBinder.bind(AggregateBinder.java:58)
	  at org.springframework.boot.context.properties.bind.Binder.lambda$bindAggregate$2(Binder.java:304)
	  at org.springframework.boot.context.properties.bind.Binder$$Lambda$206.1403495948.get(Unknown Source:-1)
	  at org.springframework.boot.context.properties.bind.Binder$Context.withIncreasedDepth(Binder.java:448)
	  at org.springframework.boot.context.properties.bind.Binder$Context.access$100(Binder.java:388)
	  at org.springframework.boot.context.properties.bind.Binder.bindAggregate(Binder.java:303)
	  at org.springframework.boot.context.properties.bind.Binder.bindObject(Binder.java:261)
	  at org.springframework.boot.context.properties.bind.Binder.bind(Binder.java:220)
	  at org.springframework.boot.context.properties.bind.Binder.lambda$bindBean$3(Binder.java:336)
	  at org.springframework.boot.context.properties.bind.Binder$$Lambda$62.533956291.bindProperty(Unknown Source:-1)
	  at org.springframework.boot.context.properties.bind.JavaBeanBinder.bind(JavaBeanBinder.java:83)
	  at org.springframework.boot.context.properties.bind.JavaBeanBinder.bind(JavaBeanBinder.java:72)
	  at org.springframework.boot.context.properties.bind.JavaBeanBinder.bind(JavaBeanBinder.java:54)
	  at org.springframework.boot.context.properties.bind.Binder.lambda$null$4(Binder.java:344)
	  at org.springframework.boot.context.properties.bind.Binder$$Lambda$64.43856716.apply(Unknown Source:-1)
	  at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	  at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359)
	  at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
	  at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
	  at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
	  at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	  at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
	  at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	  at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464)
	  at org.springframework.boot.context.properties.bind.Binder.lambda$bindBean$5(Binder.java:345)
	  at org.springframework.boot.context.properties.bind.Binder$$Lambda$63.249515830.get(Unknown Source:-1)
	  at org.springframework.boot.context.properties.bind.Binder$Context.withIncreasedDepth(Binder.java:448)
	  at org.springframework.boot.context.properties.bind.Binder$Context.withBean(Binder.java:434)
	  at org.springframework.boot.context.properties.bind.Binder$Context.access$400(Binder.java:388)
	  at org.springframework.boot.context.properties.bind.Binder.bindBean(Binder.java:342)
	  at org.springframework.boot.context.properties.bind.Binder.bindObject(Binder.java:277)
	  at org.springframework.boot.context.properties.bind.Binder.bind(Binder.java:220)
	  at org.springframework.boot.context.properties.bind.Binder.lambda$null$0(Binder.java:299)
	  at org.springframework.boot.context.properties.bind.Binder$$Lambda$215.1916303325.get(Unknown Source:-1)
	  at org.springframework.boot.context.properties.bind.Binder$Context.withSource(Binder.java:424)
	  at org.springframework.boot.context.properties.bind.Binder$Context.access$900(Binder.java:388)
	  at org.springframework.boot.context.properties.bind.Binder.lambda$bindAggregate$1(Binder.java:301)
	  at org.springframework.boot.context.properties.bind.Binder$$Lambda$205.1238905282.bind(Unknown Source:-1)
	  at org.springframework.boot.context.properties.bind.IndexedElementsBinder.bindIndexed(IndexedElementsBinder.java:113)
	  at org.springframework.boot.context.properties.bind.IndexedElementsBinder.bindIndexed(IndexedElementsBinder.java:89)
	  at org.springframework.boot.context.properties.bind.IndexedElementsBinder.bindIndexed(IndexedElementsBinder.java:71)
	  at org.springframework.boot.context.properties.bind.CollectionBinder.bindAggregate(CollectionBinder.java:50)
	  at org.springframework.boot.context.properties.bind.AggregateBinder.bind(AggregateBinder.java:58)
	  at org.springframework.boot.context.properties.bind.Binder.lambda$bindAggregate$2(Binder.java:304)
	  at org.springframework.boot.context.properties.bind.Binder$$Lambda$206.1403495948.get(Unknown Source:-1)
	  at org.springframework.boot.context.properties.bind.Binder$Context.withIncreasedDepth(Binder.java:448)
	  at org.springframework.boot.context.properties.bind.Binder$Context.access$100(Binder.java:388)
	  at org.springframework.boot.context.properties.bind.Binder.bindAggregate(Binder.java:303)
	  at org.springframework.boot.context.properties.bind.Binder.bindObject(Binder.java:261)
	  at org.springframework.boot.context.properties.bind.Binder.bind(Binder.java:220)
	  at org.springframework.boot.context.properties.bind.Binder.lambda$bindBean$3(Binder.java:336)
	  at org.springframework.boot.context.properties.bind.Binder$$Lambda$62.533956291.bindProperty(Unknown Source:-1)
	  at org.springframework.boot.context.properties.bind.JavaBeanBinder.bind(JavaBeanBinder.java:83)
	  at org.springframework.boot.context.properties.bind.JavaBeanBinder.bind(JavaBeanBinder.java:72)
	  at org.springframework.boot.context.properties.bind.JavaBeanBinder.bind(JavaBeanBinder.java:54)
	  at org.springframework.boot.context.properties.bind.Binder.lambda$null$4(Binder.java:344)
	  at org.springframework.boot.context.properties.bind.Binder$$Lambda$64.43856716.apply(Unknown Source:-1)
	  at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	  at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359)
	  at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
	  at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
	  at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
	  at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	  at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
	  at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	  at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464)
	  at org.springframework.boot.context.properties.bind.Binder.lambda$bindBean$5(Binder.java:345)
	  at org.springframework.boot.context.properties.bind.Binder$$Lambda$63.249515830.get(Unknown Source:-1)
	  at org.springframework.boot.context.properties.bind.Binder$Context.withIncreasedDepth(Binder.java:448)
	  at org.springframework.boot.context.properties.bind.Binder$Context.withBean(Binder.java:434)
	  at org.springframework.boot.context.properties.bind.Binder$Context.access$400(Binder.java:388)
	  at org.springframework.boot.context.properties.bind.Binder.bindBean(Binder.java:342)
	  at org.springframework.boot.context.properties.bind.Binder.bindObject(Binder.java:277)
	  at org.springframework.boot.context.properties.bind.Binder.bind(Binder.java:220)
	  at org.springframework.boot.context.properties.bind.Binder.bind(Binder.java:208)
	  at org.springframework.boot.context.properties.bind.Binder.bind(Binder.java:190)
	  at org.springframework.boot.context.properties.ConfigurationPropertiesBinder.bind(ConfigurationPropertiesBinder.java:83)
	  at org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor.bind(ConfigurationPropertiesBindingPostProcessor.java:107)
	  at org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor.postProcessBeforeInitialization(ConfigurationPropertiesBindingPostProcessor.java:93)
	  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:419)
	  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1737)
	  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:576)
	  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)
	  at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:356)
	  at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$358.1799040052.getObject(Unknown Source:-1)
	  at org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:390)

上面可以定位到Apollo的这个方法：com.ctrip.framework.apollo.spring.config.ConfigPropertySource.getPropertyNames

  public String[] getPropertyNames() {
    Set<String> propertyNames = this.source.getPropertyNames();
    if (propertyNames.isEmpty()) {
      return EMPTY_ARRAY;
    }
    return propertyNames.toArray(new String[propertyNames.size()]);
  }

再看内部的com.ctrip.framework.apollo.internals.DefaultConfig#getPropertyNames

  public Set<String> getPropertyNames() {
    Properties properties = m_configProperties.get();
    if (properties == null) {
      return Collections.emptySet();
    }

    return stringPropertyNames(properties);
  }

  private Set<String> stringPropertyNames(Properties properties) {
    //jdk9以下版本Properties#enumerateStringProperties方法存在性能问题，keys() + get(k) 重复迭代, jdk9之后改为entrySet遍历.
    Map<String, String> h = new LinkedHashMap<>();
    for (Map.Entry<Object, Object> e : properties.entrySet()) {
      Object k = e.getKey();
      Object v = e.getValue();
      if (k instanceof String && v instanceof String) {
        h.put((String) k, (String) v);
      }
    }
    return h.keySet();
  }

看代码不难发现这个方法时间和空间复杂度都是线性的，和PropertySource中的配置项数量正相关。

再看ConfigPropertySource这个类继承了org.springframework.core.env.EnumerablePropertySource，后者覆盖了PropertySource#containsProperty方法

    public boolean containsProperty(String name) {
        return ObjectUtils.containsElement(this.getPropertyNames(), name);
    }

这个方法也是线性的，但是这个方法消耗就很恐怖了，因为PropertySource#containsProperty方法会在启动过程中被Binder.bind调用很多次。
再通过对启动过程profiling验证一下上面的想法：
image
看到上面这个截图首先可能会有这个问题：为啥最终getPropertyNames的时间消耗只占了22%，另外的大头哪去了？因为这个项目用的是SpringBoot 2.1.x，其内部处理逻辑对于包含大量配置项的项目的效率也很慢，具体可见：#16474。

可以先忽略耗时占比，看Apollo部分的时间消耗也有接近20s了，对于5000行配置来讲，这个时间似乎太慢了。
于是斗胆修改了一下Apollo这部分的实现，基本想法就是缓存propertyNames，再使其和properties保持一致。另外考虑其他组件依赖老版本com.ctrip.framework.apollo.Config#getPropertyNames方法（返回一个Set），为保持兼容性新增了一个方法getPropertyNameArray()（返回一个数组）。

diff --git a/client/src/main/java/com/ctrip/framework/apollo/Config.java b/client/src/main/java/com/ctrip/framework/apollo/Config.java
index 7eaff8e..adecf1c 100644
--- a/client/src/main/java/com/ctrip/framework/apollo/Config.java
+++ b/client/src/main/java/com/ctrip/framework/apollo/Config.java
@@ -171,7 +171,20 @@ public interface Config {
   /**
    * Return a set of the property names
    *
+   * @see Config#getPropertyNameArray()
+   * @deprecated Use Config#getPropertyNameArray instead
    * @return the property names
    */
+  @Deprecated
   public Set<String> getPropertyNames();
+
+  /**
+   * Return a set of the property names
+   * <p>
+   * 当配置文件有很多条目时，<code>getPropertyNames</code>方法会带来不容忽略的性能消耗，故添加此方法，时间复杂度为O(1)
+   *
+   * @see Config#getPropertyNames()
+   * @return the property names
+   */
+  public String[] getPropertyNameArray();
 }
diff --git a/client/src/main/java/com/ctrip/framework/apollo/internals/DefaultConfig.java b/client/src/main/java/com/ctrip/framework/apollo/internals/DefaultConfig.java
index f6cdd8a..36fe223 100644
--- a/client/src/main/java/com/ctrip/framework/apollo/internals/DefaultConfig.java
+++ b/client/src/main/java/com/ctrip/framework/apollo/internals/DefaultConfig.java
@@ -1,19 +1,5 @@
 package com.ctrip.framework.apollo.internals;

-import java.io.IOException;
-import java.io.InputStream;
-import java.util.Collections;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.Properties;
-import java.util.Set;
-import java.util.HashMap;
-import java.util.concurrent.atomic.AtomicReference;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
 import com.ctrip.framework.apollo.core.utils.ClassLoaderUtil;
 import com.ctrip.framework.apollo.enums.PropertyChangeType;
 import com.ctrip.framework.apollo.model.ConfigChange;
@@ -21,17 +7,30 @@ import com.ctrip.framework.apollo.model.ConfigChangeEvent;
 import com.ctrip.framework.apollo.tracer.Tracer;
 import com.ctrip.framework.apollo.util.ExceptionUtil;
 import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Sets;
 import com.google.common.util.concurrent.RateLimiter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.*;
+import java.util.concurrent.atomic.AtomicReference;


 /**
  * @author Jason Song([email protected])
  */
 public class DefaultConfig extends AbstractConfig implements RepositoryChangeListener {
+
   private static final Logger logger = LoggerFactory.getLogger(DefaultConfig.class);
+
+  private static final String[] EMPTY_STRING_ARRAY = new String[0];
+
   private final String m_namespace;
   private Properties m_resourceProperties;
   private AtomicReference<Properties> m_configProperties;
+  private String[] m_propertyNames;
   private ConfigRepository m_configRepository;
   private RateLimiter m_warnLogRateLimiter;

@@ -46,13 +45,14 @@ public class DefaultConfig extends AbstractConfig implements RepositoryChangeLis
     m_resourceProperties = loadFromResource(m_namespace);
     m_configRepository = configRepository;
     m_configProperties = new AtomicReference<>();
+    m_propertyNames = EMPTY_STRING_ARRAY;
     m_warnLogRateLimiter = RateLimiter.create(0.017); // 1 warning log output per minute
     initialize();
   }

   private void initialize() {
     try {
-      m_configProperties.set(m_configRepository.getConfig());
+      updateProperties(m_configRepository.getConfig());
     } catch (Throwable ex) {
       Tracer.logError(ex);
       logger.warn("Init Apollo Local Config failed - namespace: {}, reason: {}.",
@@ -64,6 +64,15 @@ public class DefaultConfig extends AbstractConfig implements RepositoryChangeLis
     }
   }

+  private void updateProperties(Properties properties) {
+    m_configProperties.set(properties);
+    if (properties == null) {
+      m_propertyNames = EMPTY_STRING_ARRAY;
+    } else {
+      m_propertyNames = stringPropertyNames(properties);
+    }
+  }
+
   @Override
   public String getProperty(String key, String defaultValue) {
     // step 1: check system properties, i.e. -Dkey=value
@@ -97,25 +106,25 @@ public class DefaultConfig extends AbstractConfig implements RepositoryChangeLis

   @Override
   public Set<String> getPropertyNames() {
-    Properties properties = m_configProperties.get();
-    if (properties == null) {
-      return Collections.emptySet();
-    }
+    return Sets.newHashSet(m_propertyNames);
+  }

-    return stringPropertyNames(properties);
+  @Override
+  public String[] getPropertyNameArray() {
+    return m_propertyNames;
   }

-  private Set<String> stringPropertyNames(Properties properties) {
+  private String[] stringPropertyNames(Properties properties) {
     //jdk9以下版本Properties#enumerateStringProperties方法存在性能问题，keys() + get(k) 重复迭代, jdk9之后改为entrySet遍历.
-    Map<String, String> h = new HashMap<>();
+    List<String> names = new ArrayList<>(properties.size());
     for (Map.Entry<Object, Object> e : properties.entrySet()) {
       Object k = e.getKey();
       Object v = e.getValue();
       if (k instanceof String && v instanceof String) {
-        h.put((String) k, (String) v);
+        names.add((String) k);
       }
     }
-    return h.keySet();
+    return names.toArray(EMPTY_STRING_ARRAY);
   }

   @Override
@@ -153,7 +162,7 @@ public class DefaultConfig extends AbstractConfig implements RepositoryChangeLis
     }

     //2. update m_configProperties
-    m_configProperties.set(newConfigProperties);
+    updateProperties(newConfigProperties);
     clearConfigCache();

     //3. use getProperty to update configChange's new value and calc the final changes
diff --git a/client/src/main/java/com/ctrip/framework/apollo/internals/SimpleConfig.java b/client/src/main/java/com/ctrip/framework/apollo/internals/SimpleConfig.java
index 23633f5..2dae275 100644
--- a/client/src/main/java/com/ctrip/framework/apollo/internals/SimpleConfig.java
+++ b/client/src/main/java/com/ctrip/framework/apollo/internals/SimpleConfig.java
@@ -1,26 +1,25 @@
 package com.ctrip.framework.apollo.internals;

-import java.util.Collections;
-import java.util.List;
-import java.util.Map;
-import java.util.Properties;
-import java.util.Set;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
 import com.ctrip.framework.apollo.model.ConfigChange;
 import com.ctrip.framework.apollo.model.ConfigChangeEvent;
 import com.ctrip.framework.apollo.tracer.Tracer;
 import com.ctrip.framework.apollo.util.ExceptionUtil;
 import com.google.common.base.Function;
 import com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.*;

 /**
  * @author Jason Song([email protected])
  */
 public class SimpleConfig extends AbstractConfig implements RepositoryChangeListener {
+
   private static final Logger logger = LoggerFactory.getLogger(SimpleConfig.class);
+
+  public static final String[] EMPTRY_STRING_ARRAY = new String[0];
+
   private final String m_namespace;
   private final ConfigRepository m_configRepository;
   private volatile Properties m_configProperties;
@@ -69,6 +68,14 @@ public class SimpleConfig extends AbstractConfig implements RepositoryChangeList
     return m_configProperties.stringPropertyNames();
   }

+  @Override
+  public String[] getPropertyNameArray() {
+    if (m_configProperties == null) {
+      return EMPTRY_STRING_ARRAY;
+    }
+    return m_configProperties.stringPropertyNames().toArray(EMPTRY_STRING_ARRAY);
+  }
+
   @Override
   public synchronized void onRepositoryChange(String namespace, Properties newProperties) {
     if (newProperties.equals(m_configProperties)) {
diff --git a/client/src/main/java/com/ctrip/framework/apollo/spring/config/ConfigPropertySource.java b/client/src/main/java/com/ctrip/framework/apollo/spring/config/ConfigPropertySource.java
index d81a608..52fe4b3 100644
--- a/client/src/main/java/com/ctrip/framework/apollo/spring/config/ConfigPropertySource.java
+++ b/client/src/main/java/com/ctrip/framework/apollo/spring/config/ConfigPropertySource.java
@@ -1,19 +1,15 @@
 package com.ctrip.framework.apollo.spring.config;

+import com.ctrip.framework.apollo.Config;
 import com.ctrip.framework.apollo.ConfigChangeListener;
-import java.util.Set;
-
 import org.springframework.core.env.EnumerablePropertySource;

-import com.ctrip.framework.apollo.Config;
-
 /**
  * Property source wrapper for Config
  *
  * @author Jason Song([email protected])
  */
 public class ConfigPropertySource extends EnumerablePropertySource<Config> {
-  private static final String[] EMPTY_ARRAY = new String[0];

   ConfigPropertySource(String name, Config source) {
     super(name, source);
@@ -21,11 +17,17 @@ public class ConfigPropertySource extends EnumerablePropertySource<Config> {

   @Override
   public String[] getPropertyNames() {
-    Set<String> propertyNames = this.source.getPropertyNames();
-    if (propertyNames.isEmpty()) {
-      return EMPTY_ARRAY;
+    String[] names = this.source.getPropertyNameArray();
+    // For mock source getPropertyNameArray() returns null
+    if (names == null) {
+      return new String[0];
     }
-    return propertyNames.toArray(new String[propertyNames.size()]);
+    return names;
+  }
+
+  @Override
+  public boolean containsProperty(String name) {
+    return this.source.getProperty(name, null) != null;
   }

   @Override

复现

通过如下步骤可以复现：

  1. 创建多个类型为properties的namespace
  2. 给namespace中生成一些配置，例如上面给出的
  3. 启动项目，启动过程抓取threadump

期望

天下武功，唯快不破！项目启动还是快一点的好。

如果可以我希望能提交一个PR，因为我司打算放弃自己的Fork了，投入到社区的温暖怀抱中。

截图

如果可以，附上截图来描述你的问题

额外的细节和日志

  • 版本：我司内部目前用的apollo 1.0.0的一个Fork
  • 错误日志：不适用
  • 配置：
  • 平台和操作系统：在本地macOS和服务器Linux中都可复现
@Shawyeok
Copy link
Contributor Author

Shawyeok commented Jul 2, 2021

@nobodyiam Could you please take a look about this?

@nobodyiam
Copy link
Member

Thanks for the proposal, here are some comments:

  1. How about we move the getPropertyNames method implementation to AbstractConfig so that we don't need to implement twice?
  2. We may use the same way to cache/invalidate the property names as other caches in AbstractConfig
  3. I'd prefer not to add a new interface public String[] getPropertyNameArray();. I assume the execution times of getPropertyNames is not that many?

@Shawyeok
Copy link
Contributor Author

Shawyeok commented Jul 5, 2021 •
edited by Anilople
Loading

I assume the execution times of getPropertyNames is not that many?

I've made a test with 13 namespaces and 4911 property items total, the method com.ctrip.framework.apollo.spring.config.ConfigPropertySource#getPropertyNames is called 318276 times.

Test environment:

  • Spring Boot: v2.2.13.RELEASE

It's caused by the result of getPropertyNames is used in spring internal as cache key.
But it has been changed in latest version of spring boot, so further test is needed, I'll fill that later.

@Shawyeok
Copy link
Contributor Author

Shawyeok commented Jul 8, 2021

it has been changed in latest version of spring boot, so further test is needed, I'll fill that later.

In the latest version of spring boot, the method ConfigPropertySource#getPropertyNames is also called 189475 times.

Test environment:

  • Spring Boot: 2.5.2
  • Apollo Client: 1.8.0

Here is my test details:

  1. Create a empty project with spring boot and apollo dependency
  2. Copy ConfigurationProperties and corresponding properties files from my business project.
@RefreshScope
@Configuration
@ConfigurationProperties(prefix = "xxx.abc.caption")
public class ComponentProperties {

    private List<CaptionProperties> caption;

    private List<CacheProperties> cache;

    private List<RedisProperties> redis;

    <other fields>
}

In the bootstrap process, I'll load config and initialize some components.

@Autowired
ComponentProperties componentProperties;

@PostConstruct
void init() {
    long start = System.currentTimeMillis();
    // this line will fire spring lazy load, read local config cache into properties beans
    List<CaptionProperties> captionList = componentProperties.getCaption();
    long duration = System.currentTimeMillis() - start;
    System.out.printf("duration: %dms\n", duration);
    // business logic
}
  1. Start application with apollo Local mode

Test result (duration output):

Duration
  11717ms
  12755ms
  12849ms
  11482ms
  11903ms
Stdev 622ms
Aveage 12141ms

Shawyeok added a commit to Shawyeok/apollo that referenced this issue Jul 8, 2021
@Shawyeok
Speedup startup process via cache propertyNames (apolloconfig#3800)
69ca882
@Shawyeok Shawyeok mentioned this issue Jul 8, 2021
Merged
5 tasks
Shawyeok added a commit to Shawyeok/apollo that referenced this issue Jul 8, 2021
@Shawyeok
Reduce bootstrap time in the situation with large properties (apolloc…
389f967 
…onfig#3800)
Shawyeok added a commit to Shawyeok/apollo that referenced this issue Jul 8, 2021
@Shawyeok
Reduce bootstrap time in the situation with large properties (apolloc…
04898e9 
…onfig#3800)
Shawyeok added a commit to Shawyeok/apollo that referenced this issue Jul 9, 2021
@Shawyeok
Reduce bootstrap time in the situation with large properties (apolloc…
2e89f18 
…onfig#3800)
Shawyeok added a commit to Shawyeok/apollo that referenced this issue Jul 11, 2021
@Shawyeok
Reduce bootstrap time in the situation with large properties (apolloc…
796d55f 
…onfig#3800)
Shawyeok added a commit to Shawyeok/apollo that referenced this issue Jul 11, 2021
@Shawyeok
Reduce bootstrap time in the situation with large properties (apolloc…
542951e 
…onfig#3800)
Shawyeok added a commit to Shawyeok/apollo that referenced this issue Jul 11, 2021
@Shawyeok
Reduce bootstrap time in the situation with large properties (apolloc…
f262156 
…onfig#3800)
Shawyeok added a commit to Shawyeok/apollo that referenced this issue Jul 15, 2021
@Shawyeok
Reduce bootstrap time in the situation with large properties (apolloc…
b37687c 
…onfig#3800)
Shawyeok added a commit to Shawyeok/apollo that referenced this issue Jul 16, 2021
@Shawyeok
Reduce bootstrap time in the situation with large properties (apolloc…
9979023 
…onfig#3800)
Shawyeok added a commit to Shawyeok/apollo that referenced this issue Jul 18, 2021
@Shawyeok
Reduce bootstrap time in the situation with large properties (apolloc…
c8e839d 
…onfig#3800)
nobodyiam pushed a commit that referenced this issue Jul 19, 2021
@Shawyeok @nobodyiam
Reduce bootstrap time in the situation with large properties (#3800)
6d50ca8
@lonre lonre mentioned this issue Jul 31, 2021
Merged
5 tasks
@zhenglzh
Copy link

@Shawyeok 你好请问下 你那个看启动耗时的工具是什么？

@Shawyeok
Copy link
Contributor Author

@Shawyeok 你好请问下 你那个看启动耗时的工具是什么？

@zhenglzh JProfiler

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Reduce bootstrap time in the situation with large properties Shawyeok/apollo
3 participants
@nobodyiam @Shawyeok @zhenglzh