diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000..69fe55e --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,19 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line. +SPHINXOPTS = +SPHINXBUILD = sphinx-build +SOURCEDIR = source +BUILDDIR = build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) \ No newline at end of file diff --git a/docs/make.bat b/docs/make.bat new file mode 100644 index 0000000..543c6b1 --- /dev/null +++ b/docs/make.bat @@ -0,0 +1,35 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=source +set BUILDDIR=build + +if "%1" == "" goto help + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.http://sphinx-doc.org/ + exit /b 1 +) + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% + +:end +popd diff --git a/docs/source/architecture.rst b/docs/source/architecture.rst new file mode 100644 index 0000000..66660be --- /dev/null +++ b/docs/source/architecture.rst @@ -0,0 +1,52 @@ +Architecture View +================= + +Agent. Executes active stimulus using extensible interfaces to probers to benchmark and +collect network and system performance metrics. While a single Agent is capable of perform- +ing localized benchmarks in execution environments (e.g., stress tests on CPU, memory, disk +I/O), the interaction among distributed Agents enable the generation and collection of VNF +end-to-end metrics (e.g., frame loss rate, latency). In a benchmarking setup, one Agent can +create the stimuli and the other end be the VNF itself where, for example, one-way latency is +evaluated. An Agent can be defined by a physical or virtual network function. Agents expose +modular APIs for flexible extensibility (e.g., new probers). Agents receive instructions from +a Manager defining sets of actions to consistently configure and run prober instances, parse +the results, and send back snapshots containing output evaluations of the probers’ actions. + +Prober – defines a software/hardware-based tool able to generate stimulus traffic +specific to a VNF (e.g., sipp) or generic to multiple VNFs (e.g., pktgen). A prober must +provide programmable interfaces for its life cycle management workflows, e.g., configuration +of operational parameters, execution of stimuli, parsing of extracted metrics, and debugging +options. Specific probers might be developed to abstract and to realize the description of +particular VNF benchmarking methodologies. + +Monitor. When possible, it is instantiated inside the target VNF or NFVI PoP (e.g., as +a plug-in process in a virtualized environment) to perform passive monitoring/instrumenta- +tion, using listeners, for metrics collection based on benchmark tests evaluated according to +Agents’ stimuli. Different from the active approach of Agents that can be seen as generic +benchmarking VNFs, Monitors observe particular properties according to NFVI PoPs and +VNFs capabilities. A Monitor can be defined as a virtual network function. Similarly to the +Agent, Monitors interact with the Manager by receiving instructions and replying with snap- +shots. Different from the generic VNF prober approach of the Agent, Monitors may listen to +particular metrics according to capabilities offered by VNFs and their respective execution +environment (e.g. CPU cycles of DPDK-enabled processors). + +Listener – defines one or more software interfaces for the extraction of particular +metrics monitored in a target VNF and/or execution environment. A Listener must provide +programmable interfaces for its life cycle management workflows, e.g., configuration of op- +erational parameters, execution of monitoring captures, parsing of extracted metrics, and +debugging options. White-box benchmarking approaches must be carefully analyzed, as var- +ied methods of performance monitoring might be coded as a Listener, possibly impacting the +VNF and/or execution environment performance results. + +Manager. Responsible for (i) keeping a coherent state and consistent coordination and +synchronization of Agents and Monitors, their features and activities; (ii) interacting with +the Player to receive tasks and decompose them into a concrete set of instructions; and (iii) +processing snapshots along proper aggregation tasks into reports back to the Player. + +Player. Defines a set of user-oriented, north-bound interfaces abstracting the calls needed +to manage, operate, and build a VNF-BR. Player can store different VNF Benchmarking +Descriptors, and trigger their execution when receiving a testing Layout request, that might +reference one or more parametrized VNF-BDs, which are decomposed into a set of tasks or- +chestrated by Managers to obtain their respective reports. Interfaces are provided for storage +options (e.g., database, spreadsheets) and visualization of the extracted reports into VNF- +PPs. \ No newline at end of file diff --git a/docs/source/conf.py b/docs/source/conf.py new file mode 100644 index 0000000..855b2d6 --- /dev/null +++ b/docs/source/conf.py @@ -0,0 +1,173 @@ +# -*- coding: utf-8 -*- +# +# Configuration file for the Sphinx documentation builder. +# +# This file does only contain a selection of the most common options. For a +# full list see the documentation: +# http://www.sphinx-doc.org/en/master/config + +# -- Path setup -------------------------------------------------------------- + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) + + +# -- Project information ----------------------------------------------------- + +project = u'Gym' +copyright = u'2019, Raphael Vicente Rosa' +author = u'Raphael Vicente Rosa' + +# The short X.Y version +version = u'' +# The full version, including alpha/beta/rc tags +release = u'0.1' + + +# -- General configuration --------------------------------------------------- + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The master toctree document. +master_doc = 'index' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = [] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = None + + +# -- Options for HTML output ------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'alabaster' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] + +# Custom sidebar templates, must be a dictionary that maps document names +# to template names. +# +# The default sidebars (for documents that don't match any pattern) are +# defined by theme itself. Builtin themes are using these templates by +# default: ``['localtoc.html', 'relations.html', 'sourcelink.html', +# 'searchbox.html']``. +# +# html_sidebars = {} + + +# -- Options for HTMLHelp output --------------------------------------------- + +# Output file base name for HTML help builder. +htmlhelp_basename = 'Gymdoc' + + +# -- Options for LaTeX output ------------------------------------------------ + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, 'Gym.tex', u'Gym Documentation', + u'Raphael Vicente Rosa', 'manual'), +] + + +# -- Options for manual page output ------------------------------------------ + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, 'gym', u'Gym Documentation', + [author], 1) +] + + +# -- Options for Texinfo output ---------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, 'Gym', u'Gym Documentation', + author, 'Gym', 'One line description of project.', + 'Miscellaneous'), +] + + +# -- Options for Epub output ------------------------------------------------- + +# Bibliographic Dublin Core info. +epub_title = project + +# The unique identifier of the text. This can be a ISBN number +# or the project homepage. +# +# epub_identifier = '' + +# A unique identification for the text. +# +# epub_uid = '' + +# A list of files that should not be packed into the epub file. +epub_exclude_files = ['search.html'] diff --git a/docs/source/index.rst b/docs/source/index.rst new file mode 100644 index 0000000..d878bc4 --- /dev/null +++ b/docs/source/index.rst @@ -0,0 +1,26 @@ +.. Gym documentation master file, created by + sphinx-quickstart on Tue Jul 30 21:37:08 2019. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +Welcome to Gym's documentation! +=============================== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + intro.rst + tests.rst + architecture.rst + workflow.rst + internals.rst + + + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/docs/source/internals.rst b/docs/source/internals.rst new file mode 100644 index 0000000..d863c4d --- /dev/null +++ b/docs/source/internals.rst @@ -0,0 +1,75 @@ +Gym - Under the Hood +==================== + +After installed, to start any Gym component, just specify an id and url. +E.g.: gym-agent --id 1 --url http://127.0.0.1:8989 + +So the Agent started above will have an identity "1" and operating in the management interface "http://127.0.0.1:8989" via REST. The field --debug will enable debugging messages of the component. +Another option to be added to initial operations of an script is the field "--contacts". +This field can specify any list of contacts the component would be able to reach when started. For instance, + + ```bash + $ gym-agent --id 1 --url http://127.0.0.1:8989 + + $ gym-agent --id 2 --url http://127.0.0.1:8988 + ``` + +if two agents are started as the commands above, an manager can be started to reach those agents using the following command: + + + ```bash + $ gym-manager --id 3 --url http://127.0.0.1:8990 --contacts http://127.0.0.1:8989 http://127.0.0.1:8988 + ``` +Thus, when started, manager is going to reach the agents via the management urls specified by the contacts field. + + ```bash + $ gym-player --id 4 --url http://127.0.0.1:8991 --contacts http://127.0.0.1:8990 + ``` + +Likewise, a player when started is going to reach the manager component via the specified url by the contacts field. + + +## Gym Handshake: Hello and Info + +When a component reaches another via the specified url(s) by the contacts field, it is going to send a Hello message containing its reachability information. I.e., such message defines the source node following fields: uuid, url, prefix, role. +In the case a manager component is started with the following command: +gym-manager --id 3 --url http://127.0.0.1:8990 --contacts http://127.0.0.1:8989 + +It's going to send a message to the contact with url "http://127.0.0.1:8989" specifying it has {uuid:3, url:http://127.0.0.1:8990, prefix: random (e.g., 1998), role: manager}. + +The prefix field is going to be randomly generated when the contact is created by the manager component. Such prefix defines that the manager component is always going to reach the other component via that prefix. I.e., using the url "http://127.0.0.1:8989/1988". + +When receiving a Hello message, a component is going to create the contact using the information contained in the message. And it is going to reply with an Info message containing the exact information it has about itself (same data as contained in the Hello message). + +Notice: every component must specify a url prefix to reach another component. + + +Agent/Monitor +Both components load their probers/listeners when started. Each prober/listener is specified by the component source folder. +E.g., in the source folder (path gym/agent/probers) each file named prober_* (e.g., prober_ping.py). Similarly, the same applies for the monitor component and its listeners. +Each prober/listerner contains an unique identifier, its execution parameters and its output metrics. +This way, an Agent/Monitor will have all its operational features (i.e., probers/listeners available for tests) and envrionmental features (i.e., information regarding the allocated resources for the component, such as cpu, memory, disk, operating system version). + + +Manager +It realizes the discovery of features from and the coordination activities of Agents/Monitors. +Therefore, it waits for contacts from Agents/Monitors or Player components. + + +Player +Players starts with contacts specified, reaching then a Manager component, which reaches all the "subcontacts" specified by Player, if not yet already reached. Thus Player detains the whole information of Manager and its incumbent Agent(s)/Monitor(s). + + +Messaging +Messages are all exchanged in the JSON format, in an RPC-like format. +Future work consists in evolvint the messaging system to an actual RPC system through well-known frameworks (e.g., gRPC) or even utilize distributed storage/streamming platforms (e.g., etcd, kafka, zeromq). + +REST-(like) APIs are simply defined by POST, so not yet full REST. Ideally, Gym components will evolve to have full REST APIs, together with streamming platforms, optional for users determine their needs. + +Storage +Gym aims to support different storage options of its output results (VNF-BRs) in the Player component. Currently gym supports storage in disk and in the elasticsearch database. + + +Debugging +Providing the option --debug when starting any gym component, all the debug messages are shown in the terminal. +Later, gym aims to create granular debugging methods for its components. \ No newline at end of file diff --git a/docs/source/intro.rst b/docs/source/intro.rst new file mode 100644 index 0000000..49de1f2 --- /dev/null +++ b/docs/source/intro.rst @@ -0,0 +1,55 @@ + +Gym +=== + +Gym is a reference implementation of the ongoing draft in the Benchmarking Methodology Working Group (BMWG) in Internet Engineering Task Force (IETF), named https://datatracker.ietf.org/doc/draft-rosa-bmwg-vnfbench/ + +If you want to cite this work, please use: + +ROSA, R. V.; BERTOLDO, C.; ROTHENBERG, C. E. [**Take your vnf to the gym: A testing framework for automated nfv performance benchmarking**](https://ieeexplore.ieee.org/document/8030496). IEEE Communications Magazine, v. 55, n. 9, p. 110–117, 2017. ISSN 0163-6804. + +Bibtex: + +@ARTICLE{Rosa:2017:Gym, +author={R. V. {Rosa} and C. {Bertoldo} and C. E. {Rothenberg}}, +journal={IEEE Communications Magazine}, +title={Take Your VNF to the Gym: A Testing Framework for Automated NFV Performance Benchmarking}, +year={2017}, +volume={55}, +number={9}, +pages={110-117}, +keywords={program testing;virtualisation;VNF;automated NFV performance benchmarking;software entity;testing framework;vIMS scenario;network functions virtualization;Benchmark testing;Measurement;Monitoring;Software testing;Visualization;Network function virtualization}, +doi={10.1109/MCOM.2017.1700127}, +ISSN={0163-6804}, +month={Sep.}, +} + + +Gym was built to receive high-level test descriptors and execute them +to extract VNFs profiles, containing measurements of performance +metrics - especially to associate resources allocation (e.g., vCPU) +with packet processing metrics (e.g., throughput) of VNFs. From the +original research ideas, such output profiles might be used +by orchestrator functions to perform VNF lifecycle tasks (e.g., +deployment, maintenance, tear-down). + +The proposed guiding principles to design +and build Gym can be composed in multiple practical ways for +different VNF testing purposes: + +* Comparability: Output of tests shall be simple to understand and + process, in a human-read able format, coherent, and easily + reusable (e.g., inputs for analytic applications). + +* Repeatability: Test setup shall be comprehensively defined through + a flexible design model that can be interpreted and executed by + the testing platform repeatedly but supporting customization. + +* Configurability: Open interfaces and extensible messaging models + shall be available between components for flexible composition of + test descriptors and platform configurations. + +* Interoperability: Tests shall be ported to different environments + using lightweight components. + + diff --git a/docs/source/tests.rst b/docs/source/tests.rst new file mode 100644 index 0000000..fd95aeb --- /dev/null +++ b/docs/source/tests.rst @@ -0,0 +1,14 @@ +Tests Description +================= + +Test 0 +^^^^^^ + +Test 1 +^^^^^^ + +Test 2 +^^^^^^ + +Test 3 +^^^^^^ \ No newline at end of file diff --git a/docs/source/workflow.rst b/docs/source/workflow.rst new file mode 100644 index 0000000..375eaba --- /dev/null +++ b/docs/source/workflow.rst @@ -0,0 +1,58 @@ +How Gym Works +============= + +Gym Steps +^^^^^^^^^ + +Gym core components communicate through a flexible REpresentational State Transfer + (REST) Application Programming Interface (API) using generic RPC calls with custom/extensible + JavaScript Object Notation (JSON) message formats. In the following, we +describe a generic workflow based on request-reply message exchanges and pairwise component + interactions represented as numbered (1 to 7). + +1. The first step consists of a user defining the composition of the VNF testing VNF-BD +containing the structural and functional requirements to express target performance +metrics to generate a VNF-PP. + +2. The Player processes the parametrized VNF-BD considering the features offered by +the associated Manager(s). The output is a workflow of tasks, in sequence or parallel, +submitted to a selected Manager that satisfies (i.e. controls a matching set of Agents/- +Monitors) the VNF-BD requirements. Based on input variables, a VNF-BD can be +decomposed into different sets of tasks with the corresponding high-level probers/lis- +teners parameters. + +3. The Manager decomposes tasks into a coherent sequence of instructions to be sent to +Agents and/or Monitors. Inside each instruction, sets of actions define parametrized +execution procedures of probers/listeners. Sequential or parallel tasks may include prop- +erties to be decomposed into different sets of instructions, for instance, when sampling +cycles might define their repeated execution. + +4. By interpreting action into a prober/listener execution, an Agent or Monitor performs +an active or passive measurement to output metrics via a pluggable tool. A VNF +developer can freely create a customized prober or listener to interface her tests and +extract particular metrics. An interface of such a tool is automatically discovered by +an Agent/Monitor and exposed as available to Managers and Players along with its +corresponding execution parameters and output metrics. + +5. After computing the required metrics, a set of evaluations (i.e., parsed action outputs) +integrate a so-called snapshot sent from an Agent/Monitor to the Manager. A snap- +shot associated to a specific task is received from the Agent/Monitor that received +the corresponding instruction. An evaluation contains timestamps and identifiers of +the originating prober/listener, whereas a snapshot receives an Agent/Monitor unique +identifier along the host name information from where it was extracted. + +6. After processing all the instructions related tree of snapshots, the Manager composes +a report, as a reply to each task requested by the Player. The Manager can sample +snapshots in a diverse set of programmable methods. For instance, a task may require +cycles of repetition, so the correspondent snapshots can be parsed and aggregated in a +report through statistical operations (e.g., mean, variance, confidence intervals). + +7. Finally, the Player processes the report following the VNF-PP metrics definition, as +established initially during the VNF-BD decomposition. While the VNF-PP contains +filtered evaluation metrics and parameters, snapshots can be aggregated/sampled into +a report. Results can be exported in different file formats (e.g., formats CSV, JSON, +YAML) or saved into a database for further analysis and visualization. For instance, +in our current Gym prototype we integrate two popular open source components, the +Elasticsearch database and the Kibana visualization platform 2 —tools providing high +flexibility in querying, filtering and creation of different visual representations of the +extracted VNF-PPs. \ No newline at end of file