This part contains information about installing and maintaining Flopsar environments.

1. Installation¶

Each component is installed separately but there is a common initial installation procedure for both manager and database. This procedure must be performed before any further steps can be made.

In order to run the Flopsar environment successfully, you must prepare your network so that it allows all the required connections (Fig. 5.2) to be established:

outgoing TCP connections from agent to database instances.
outgoing TCP connections from database instances to manager.
outgoing TCP connections from workstation instances to manager and database instances.

1.1. Manager and Database Binaries Installation¶

There are two packages available for installation: flopsar-2.2-Linux.deb and flopsar-2.2-Linux.rpm. For Debian based distributions you can install it by executing the following command:

$ dpkg -i flopsar-2.2-Linux.deb

For Red Hat based distributions you can install it by executing the following command:

$ rpm -ih flopsar-2.2-Linux.rpm

or

$ zypper install flopsar-2.2-Linux.rpm

Flopsar installs several files in your system. You can inspect these files by querying contents of the Flopsar package:

$ rpm -qlp flopsar-2.2-Linux.rpm
$ dpkg -c flopsar-2.2-Linux.deb

1.1.1. Flopsar Dependencies¶

Flopsar base configuration (without plugins) should be able to run on your system successfully. However, if your installation cannot run due to missing libraries, use ldd tool to check which libraries are missing. Below, a list of known Flopsar dependencies is specified:

Flopsar base

libncurses

SMTP plugin

libcurl

1.1.2. Manager¶

make sure you have completed the binaries installation procedure.
make sure your destination machine has the same IP address as the one specified in your license file. Otherwise, manager will not run.
initialize a new environment working directory <manager_home> by executing the following command:
$ fs2mgr init <manager_home>
put your license file into the <manager_home>/lic directory.
configure the manager
start manager by executing:
$ fs2mgr start <manager_home>

Note

There can be no multiple instances of manager running on the same machine and using the same licence file.

1.1.3. Database¶

make sure you have completed the binaries installation procedure.
initialize a new database working directory <database_home> by executing the following command:
$ fs2db init <database_home>
configure the database
start database by executing:
$ fs2db start <database_home>

1.2. Agent¶

In order to install agent, follow carefully the instructions below:

copy the agent file flopsar-agent-2.2.jar to your server machine.
configure the agent.
optionally add your formatters.
restart your application.

When the agent starts for the first time it does not have any configuration deployed. It just registers itself to the manager, after successful connection, and does not instrument anything.

Tip

You can verify if the agent is registered to the manager by executing the following command:

$ fs2mgr get --target agents <manager_home>

2. Logging¶

Each Flopsar module (application) uses logging levels. The levels are defined as follows:

finest: the most detailed logging. It should not be used during normal, production operation since it can degrade the application performance significantly. Use only in cases when you suspect some problems with the application, want to trace how it processes requests, requested by the Flopsar Technology Support to help investigate some problem, or any other problems. It prints lots of diagnostic information, which might not be covered by this manual.
finer: a less detailed logging than finest. Used to log some additional information about the application operation and data processing.
fine: a more detailed logging than info. Used to log some additional information about the application operation and data processing.
info: a default level used in applications. Used only to log some runtime information to verify configuration and the like. You should use this level on normal, production operation.
warning: used in cases when some error occurs but with no impact on the applilcation stability and data integrity.
error: normally, there should be no error messages in the log. The level is used in cases when some error occurs. These errors impact the application operation stability and data integrity but not all of them. The messages are informative, so you should know whether you can ignore it or not. Any system calls and external libraries errors are also logged in this level.
fatal/severe: there should be no such messages in the log. Messages of this level are only printed in cases when some fatal error occurred and the application cannot run any longer. For example, some misconfigured options can result in printing fatal messages in the log. Please, read them and fix your configuration. The messages are usually very informative.

3. Agent¶

agent is a core component. It runs on a JVM and its main goal is to collect data from applications running on the JVM and the JVM itself.

3.1. Configuration¶

Agents are configured by their runtime options. The following options are mandatory:

-Dflopsar.id: Primary identifier of a server instance. It should consist of only word characters and have no more than 62 characters. If you provide a name consisting of more characters, the agent will truncate the name.
-javaagent: Path to your agent file flopsar-agent-2.2.jar.
-Dflopsar.manager: Socket address of the manager TCP server. Agents initiate connections to this server and the connections are maintained until the agents are shutdown.
-Dflopsar.home: Path to the Flopsar working directory. Log files are stored in this directory.

The above options must by set, otherwise the agent will not work. For example, your configuration could look like this one:

-Dflopsar.id=MyAgent -javaagent:/path/to/flopsar-agent.jar -Dflopsar.manager=172.16.0.1:9000 -Dflopsar.home=/path/to/some/dir

There are some additional options, which are not mandatory:

-Dflopsar.id.dyn: Secondary identifier of a server instance. If you use this option, you must set its value V to only another JVM option, i.e. the agent will take the value of this option and execute the method java.lang.System.getProperty(V,null). The output of the method will be the effective value of the secondary identifier. If you use both identifiers, primary and secondary, the agent will combine them into a single, composite identifier. They will be concatenated by an underscore character. The composite identifier’s length is up to 62 characters.
-Dflopsar.debug: By default source code line numbers in exceptions stack traces are omitted. If you want the agent to preserve these line numbers set this option value to true.
-Dflopsar.logsize: Total maximum size in MB of log files. By default this value is set to 100. By setting this option you specify how much space log files can use.
-Dflopsar.qsize: Total maximum size of the agent outbound queue size in MB. By setting this option you specify how much memory the agent can use.
-Dflopsar.qratio: Ratio of agent outbound queue size to the maximum JVM memory size in percentages. By default this value is set to 20. By setting this option you specify how much memory the agent can use. If you specify both this option and -Dflopsar.qsize, the latter takes precedence.

For example, your extended configuration could look like this one:

-Dflopsar.id=MyAgent -javaagent:/path/to/flopsar-agent-VER.jar -Dflopsar.manager=172.16.0.1:9000 -Dflopsar.home=/path/to/some/dir -Dflopsar.logsize=200  -Dflopsar.qsize=400 -Dflopsar.debug=true

There are various places where you can add your custom JVM options, it depends on the Java software you use. Please, refer to your Java software documentation for more details about adding extra JVM options.

Important

Please, make sure that every agent uses its own, unique identifier. If there are multiple agents with the same identifiers and they connect to the same manager, their connections will be dropped by the manager everytime a new connection attempt appears from any of the agents.

3.2. Formatters¶

agent can be used standalone or it can be extended by means of Flopsar Extensions. Flopsar agent is a powerful engine, which is extensible by formatters. The extensions are some extra jar files, which consist of user defined POJO classes. The jars can be then included in agent deployment.

Formatter is a key feature. It enables to process data inside the instrumented methods. In order to take advantage of this feature, all you have to do is setup a configuration, create a simple POJO class and implement a single method (called formatter):

public static java.lang.String myFormatter(java.lang.Object[] args)

Warning

You can give any name to the above method but the signature of the method must be preserved. Otherwise, it will not work.

The argument of the formatter is an array of the instrumented method arguments delivered to your formatter. These are your original application objects, not clones. When you set up the configuration correctly, agent will inject your formatter code into the instrumented methods during the instrumentation. It can be injected in one of the two places, i.e. at the beginning or the end of the method. Note, your formatter code is executed inside your instrumented method, so make sure your formatter code generates minimal overhead.

Warning

If you crash your application because of badly written formatters, do not blame us. Formatters code can be written by anyone, that is why they are not supported by Flopsar Technology.

In fact, it is up to you what extra features agent will have. You can even extend your agent installation to cover a distributed transactions tracing.

Finding the best place for formatters jars can be a bit tricky and it strongly depends on your Java enviroment. You can find some tips and tricks on our GitHub page.

3.3. Data Collecting Considerations¶

Although, agent is designed to collect as much data as possible, we are not able to guarantee that all the instrumented methods calls data will be collected. It is because agent priority is not to crash the application it monitors. That has some serious implications on the agent design and operation. We just favor stability and reliability over data completeness.

Each call of an instrumented method, symbol or metric value is represented as a serialized entity called packet. As soon as the packet is created it is ready to be sent to a database. In order the packet to be sent it must be first put to the outbound queue. This is the point where agent decides whether put it or not. The packet is put to the queue if and only if the connection to the database is established and there is enough space in the outbound queue (the size of the outbound queue can be customized). Otherwise, the packet is dropped. This scenerio holds for each packet.

3.4. Instrumentation¶

The fundamental feature of agent is instrumentation. This feature makes use of java.lang.instrument framework. A user configuration describes what and where to instrument (see Managing Agent Configurations for details). However, there are some classes which are always excluded from instrumentation:

com.flopsar.*
org.objectweb.*
javassist.*
org.apache.bcel.*
java.*
javax.*
jdk.*
$
com.sun.*
sun.*

When agent receives a configuration to load, it first parses it and then sorts its instrumentation rules. If there are any changes in the configuration that requires configuration reloading, agent reloads those classes that have been instrumented earlier and those which are required by this new configuration. This classes flow is passed through the configuration filters (Fig. 3.1). The are a few filters operating on two levels: class and method ones. The first filter excludes all the classes from packages, which match the internal exclusion set. Next, it goes through the second, user defined, exclusion class filter. At this point, we have all the classes we want to instrument. In the next step, all the methods from each, selected class are passed through the method filters. At this point, a class can also be excluded from configuration if any of its methods does not satisfy the filters conditions. At the end, we obtain a set of methods which are selected for instrumentation.

Fig. 3.1 Instrumentation Filter

The methods instrumentation is performed in a specific order (Fig. 3.2). First, rules with custom formatters are applied, then rules with standard formatters and finally rules without formatters at the end. For example, if there are two method rules, one instrumenting with a custom formatter and the other one instrumenting only public methods and our method is both public and its signature matches the one specified in the first rule, then the rule with the formatter will be applied.

Fig. 3.2 Instrumentation Order

3.5. Hot Methods¶

This feature enables you to record instrumented methods executions. The Hot methods feature is very useful, when you tune your configuration for minimal performance impact. In this way, you can easily detect the most frequently executed methods and exclude them from instrumentation, unless you really need to instrument them.

When agent starts, this feature is enabled by default. agent simply registers the number of instrumented methods executions. The Hot Methods statistics are dumped every 30 seconds to a log file (see the example below), provided that the logger level is set to at least fine level. This is done for 30 cycles and then is disabled. You can enable it later from workstation (see Hot Methods for details).

>>>>>>>>>>>>>>>>>>>>>>> Hot Methods Report [Cycles left 27] <<<<<<<<<<<<<<<<<<<<<<<<
HM [206633] -> [com/mysql/jdbc/ConnectionImpl][checkClosed\^\_()V]
HM [166145] -> [com/mysql/jdbc/ConnectionImpl][versionMeetsMinimum\^\_(III)Z]
HM [145591] -> [com/konakart/db/KKCriteria][getAsColumns\^\_()Ljava/util/Map;]
HM [109807] -> [com/konakart/db/KKCriteria][getDbName\^\_()Ljava/lang/String;]
...........

The values in the parentheses denote the number of times the corresponding method was called.

3.6. Logging¶

The logging levels are defined in Logging. The log files size is 100 MB in total by default but it can be changed by setting flopsar.logsize option. If the log file reaches its maximum size a new file is created. The logging framework maintains up to 5 log files.

Note

By default, agent always starts with finest logger level and after 2 minutes it is switched to info automatically. You can always change the level in runtime using workstation.

3.6.1. Logger Messages¶

Each logger message entry has the following structure:

DATE [LEVEL]<THREAD> Message content

where:

Message content:
DATE:	current date.
LEVEL:	current logger level.
THREAD:	current Java thread.
	is self-explanatory.