Directory structure with the new platform

[kimjoar]

With the addition of the new platform, we need to adjust our directory structure to support both legacy and new code simultaneously. The directory structure should as clearly as possible separate out concerns and distinguish between new and old.

Latest proposal: #12466 (comment)

Original description

We need to decide on a directory structure for the new platform.

@epixa created a suggestion in https://gist.github.com/epixa/5af7474d3205f9558113bb6b30248050

Discussions

• 8 Nov 2017
  • talked about philosophy of dir structure
  • decided we should have tenets
  • notes on tenets: https://gist.github.com/archanid/8f9075816be79a72af4817bc59b72cdf

[spalger]

I feel that we don't need to separate dev and tools. I'd even wonder if test should be a part of dev...

[epixa]

The dev directory in my example is a place for input/output artifacts from development, essentially. Unless we wanted to pre-seed the development setup, the directory wouldn't even need to be checked in. When you start up a dev run of Kibana, it'd write things like the pid, data directory, logs, etc. to this directory. On the other hand, tools and tests are permanent, checked-in files. The former could be things like a backporting tool or a version bumping tool, whereas the latter would be for functional test files.

[spalger]

Ah, I thought the dev directory was kind of a "dev tool source" directory, makes sense

[rhoboat]

Added to the issue description to include notes from today's discussion

[epixa]

@kjbekkelund You're getting close to merging the initial platform work into master. To unblock the directory structure work, it would be helpful if you could outline specific concerns you have for the directory structure related to the new platform itself. Like where to put core plugins, how we separate server and client code in a plugin, etc.

[epixa]

I've taken a more pragmatic swing at this. We've learned a lot about how we plan to roll out the new platform since this issue was opened, and I think a complete overhaul of our entire directory structure is not necessary right now. Instead, I focused on how the legacy code and new platform code would live by side by.

I moved the new platform code into src, and then I rearranged some directories to more clearly separate out legacy functionality from the new. All existing src directories are represented here, though I've tweaked some names across the board.

src
  build
    babel-register
    optimize
  core (new platform core)
    cli
    public
    server
      config
      logging
      legacy_compat
    test_helpers
  dev
    build
    docs
    ci_setup
    eslint
    ...
  plugins (new platform plugins)
    timepicker
      public
      server
    ...
  legacy
    cli
    cli_keystore
    cli_plugin
    deprecation
    core_plugins
      console
      dev_mode
      elasticsearch
      ...
    plugin_discovery
    server
    ui
    utils
  test
    es_archiver
    fixtures
    functional_test_runner
    utils
x-pack
  dev
    build_chromium
    gulp_helpers
    dev-tools
  docs
  legacy
    common
    plugins
      apm
      cloud
      console_extensions
      ...
    server
    webpackShims
  plugins (new platform plugins)
    license
      public
      server
    security
      public
      server
    ...
  scripts
  test

One key thing I want to point out here is that a plugin is either all legacy or all new platform, at least from a technical standpoint. So for legacy plugins that are doing a gradual migration over to the new platform, they are technically migrating to a new plugin. Given recent conversations I've had with @spalger about how we might treat legacy plugin code in the new platform (at least in the UI), I think this makes a lot of sense. It also gives us a clear indication of what is and is not new platform stuff, and we can choose to enforce different linting rules and such between old and new.

• Edit 8 May 2018: browser -> public
• Edit 8 May 2018: x-pack directories
• Edit 22 May 2018: xpack_core -> license

[azasypkin]

Looks good to me, I only have mixed feelings about browser, why not just good old client (not ui since technically client code in plugin or core may not provide any user interface)?

[epixa]

@azasypkin @spalger brought up the browser name as well, and ultimately we decided to use public for now so we don't have to overhaul a ton of glob references in our build system for client-side code. Down the line, we can always rename to client or browser if we think it's important, but I suspect we won't care much.