This part contains information about installing and maintaining Flopsar environments.

1. Installation¶

Each component is installed separately but there is a common initial installation procedure for both manager and database. This procedure must be performed before any further steps can be made.

In order to run the Flopsar environment successfully, you must prepare your network so that it allows all the required connections (Fig. 5.2) to be established:

outgoing TCP connections from agent to database instances.
outgoing TCP connections from database instances to manager.
outgoing TCP connections from workstation instances to manager and database instances.

1.1. Manager and Database Binaries Installation¶

There are two packages available for installation: flopsar-2.4-Linux.deb and flopsar-2.4-Linux.rpm. For Debian based distributions you can install it by executing the following command:

$ dpkg -i flopsar-2.4-Linux.deb

For Red Hat based distributions you can install it by executing one of the following commands:

$ rpm -ih flopsar-2.4-Linux.rpm
$ zypper install flopsar-2.4-Linux.rpm
$ yum install flopsar-2.4-Linux.rpm

Flopsar installs several files in your system. You can inspect these files by querying contents of the Flopsar package:

$ rpm -qlp flopsar-2.4-Linux.rpm
$ dpkg -c flopsar-2.4-Linux.deb

Important

You will not be able to run Flopsar unless you accept the Flopsar End User License Agreement (EULA). In order to accept the license you must set the environment variable FLOPSAR_EULA_ACCEPT to YES.

1.1.1. Flopsar Dependencies¶

Flopsar base configuration (without plugins) should be able to run on your system successfully. However, if your installation cannot run due to missing libraries, use ldd tool to check which libraries are missing. You can also check all dependencies by executing the following command:

$ rpm -qp --requires flopsar-2.4-Linux.rpm

1.1.2. Manager¶

make sure you have completed the binaries installation procedure.
make sure your destination machine has the same IP address as the one specified in your license file. Otherwise, manager will not run.
initialize a new environment working directory <manager_home> by executing the following command:
$ fs2mgr init <manager_home>
put your license file into the <manager_home>/lic directory.
configure the manager
start manager by executing:
$ fs2mgr start <manager_home>

Note

There can be no multiple instances of manager running on the same machine and using the same licence file.

1.1.3. Database¶

make sure you have completed the binaries installation procedure.
initialize a new database working directory <database_home> by executing the following command:
$ fs2db init <database_home>
configure the database
start database by executing:
$ fs2db start <database_home>

1.2. Agent¶

In order to install agent, follow carefully the instructions below:

copy the agent file flopsar-agent-2.4.jar to your server machine.
create a directory and ext subdirectory for the agent data.
configure the agent.
optionally put all your formatters into the ext directory.
restart your application.

When the agent starts for the first time it does not have any configuration deployed. It just registers itself to the manager, after successful connection, and does not instrument anything.

Tip

You can verify if the agent is registered to the manager by executing the following command:

$ fs2mgr get --target agents <manager_home>

Important

If you plan to install agent in OSGi environments, make sure to add com.flopsar classes to the OSGi container classpath. Please, refer to your OSGi environment documentation for details.

2. Logging¶

Each Flopsar module (application) uses logging levels. The levels are defined as follows:

finest: the most detailed logging. It should not be used during normal, production operation since it can degrade the application performance significantly. Use only in cases when you suspect some problems with the application, want to trace how it processes requests, requested by the Flopsar Technology Support to help investigate some problem, or any other problems. It prints lots of diagnostic information, which might not be covered by this manual.
finer: a less detailed logging than finest. Used to log some additional information about the application operation and data processing.
fine: a more detailed logging than info. Used to log some additional information about the application operation and data processing.
info: a default level used in applications. Used only to log some runtime information to verify configuration and the like. You should use this level on normal, production operation.
warning: used in cases when some error occurs but with no impact on the applilcation stability and data integrity.
error: normally, there should be no error messages in the log. The level is used in cases when some error occurs. These errors impact the application operation stability and data integrity but not all of them. The messages are informative, so you should know whether you can ignore it or not. Any system calls and external libraries errors are also logged in this level.
fatal/severe: there should be no such messages in the log. Messages of this level are only printed in cases when some fatal error occurred and the application cannot run any longer. For example, some misconfigured options can result in printing fatal messages in the log. Please, read them and fix your configuration. The messages are usually very informative.

3. Agent¶

agent is a core component. It runs on a JVM and its main goal is to collect data from applications running on the JVM and the JVM itself.

3.1. Configuration¶

Agents are configured by their runtime options. The following options are mandatory:

-Dflopsar.id: Primary identifier of a server instance. It should consist of only word characters and have no more than 62 characters. If you provide a name consisting of more characters, the agent will truncate the name.
-javaagent: Path to your agent file flopsar-agent-2.4.jar.
-Dflopsar.manager: Socket address of the manager TCP server. Agents initiate connections to this server and the connections are maintained until the agents are shutdown.
-Dflopsar.home: Path to the Flopsar working directory. Log files are stored in this directory.

Important

This directory must contain ext subdirectory. This is the place you put all your formatter classes.

The above options must by set, otherwise the agent will not work. For example, your configuration could look like this one:

-Dflopsar.id=MyAgent -javaagent:/path/to/flopsar-agent.jar -Dflopsar.manager=172.16.0.1:9000 -Dflopsar.home=/path/to/some/dir

There are some additional options, which are not mandatory:

-Dflopsar.ext.packages: Comma-separated list of packages, which all formatter classes come from. If not specified or empty, it contains only org.flopsar.ext package by default. Classes from these packages will be loaded by the agent from the ext subdirectory.

Note

No class coming from any of the specified packages will be instrumented, since the agent assumes these are formatter classes.

Important

Do not put your formatter classes into packages, which have the same root package as your instrumented application, e.g. if all your application classes belong to com.foo.bar.* package, make sure your formatter classes do not.
-Dflopsar.cl: Comma-separated list of class loader classes, which should be instrumented. You can use this option is you have instrument some classloader, which is not automatically instrumented by agent. This can be useful if the agent cannot load some classes by already instrumented classloaders. This option is empty by default.
-Dflopsar.id.dyn: Secondary identifier of a server instance. If you use this option, you must set its value V to only another JVM option, i.e. the agent will take the value of this option and execute the method java.lang.System.getProperty(V,null). The output of the method will be the effective value of the secondary identifier. If you use both identifiers, primary and secondary, the agent will combine them into a single, composite identifier. They will be concatenated by an underscore character. The composite identifier’s length is up to 62 characters.
-Dflopsar.errline: If you want the agent to skip line numbers (in stack traces) in the instrumented methods, set this option value to false. The default value is true.
-Dflopsar.logsize: Total maximum size in MB of log files. By default this value is set to 100. By setting this option you specify how much space log files can use.
-Dflopsar.qsize: Total maximum size of the agent outbound queue size in MB. By setting this option you specify how much memory the agent can use.
-Dflopsar.qratio: Ratio of agent outbound queue size to the maximum JVM memory size in percentages. By default this value is set to 20. By setting this option you specify how much memory the agent can use. If you specify both this option and -Dflopsar.qsize, the latter takes precedence.

For example, your extended configuration could look like this one:

-Dflopsar.id=MyAgent -javaagent:/path/to/flopsar-agent-VER.jar -Dflopsar.manager=172.16.0.1:9000 -Dflopsar.home=/path/to/some/dir -Dflopsar.logsize=200  -Dflopsar.qsize=400 -Dflopsar.debug=true

There are various places where you can add your custom JVM options, it depends on the Java software you use. Please, refer to your Java software documentation for more details about adding extra JVM options.

Important

Please, make sure that every agent uses its own, unique identifier. If there are multiple agents with the same identifiers and they connect to the same manager, their connections will be dropped by the manager everytime a new connection attempt appears from any of the agents.

3.2. Formatters¶

agent can be used standalone or it can be extended by means of Flopsar Extensions. Flopsar agent is a powerful engine, which is extensible by formatters. The extensions are some extra jar files, which consist of user defined POJO classes. The jars can be then included in agent deployment.

Formatter is a key feature. It enables to process data inside the instrumented methods. In order to take advantage of this feature, all you have to do is setup a configuration, create a simple POJO class and implement a single method (called formatter):

public static java.lang.String myFormatter(java.lang.Object[] args)

Warning

You can give any name to the above method but the signature of the method must be preserved. Otherwise, it will not work.

The argument args of the formatter is an array of the following objects (in order of appearance):

args[0]: If the instrumented method is static this element is null, otherwise it is a reference to a class instance the instrumented method is executed on.
args[1..N]: List of all arguments of the instrumented method in the same order as they appear in the method signature. These are the references to your original application objects, not some clones.
args[N+1]: If the instrumented method returns void this element is null, otherwise it is a reference to the object instance this method returns.

Important

Please note, this argument is passed to the formatter if and only if the instrumentation is performed at the end of the method.

where N is the number of the instrumented method arguments.

When you set up the configuration correctly, agent will inject your formatter code into the instrumented methods during the instrumentation. It can be injected in one of the two places, i.e. at the beginning or the end of the method. Note, your formatter code is executed inside your instrumented method, so make sure your formatter code generates minimal overhead.

Warning

If you crash your application because of badly written formatters, do not blame us. Formatters code can be written by anyone, that is why they are not supported by Flopsar Technology.

In fact, it is up to you what extra features agent will have. You can even extend your agent installation to cover a distributed transactions tracing.

3.3. Data Collecting Considerations¶

Although, agent is designed to collect as much data as possible, we are not able to guarantee that all the instrumented method calls data will be collected. It is because agent priority is not to crash the application it runs on. That has some serious implications on the agent design and operation. We just favor stability and reliability over data completeness.

Each call of an instrumented method, symbol or metric value is represented as a serialized entity called packet. As soon as the packet is created it is ready to be sent to a database. In order the packet to be sent it must be first put to the outbound queue. This is the point where agent decides whether put it or not. The packet is put to the queue if and only if the connection to the database is established and there is enough space in the outbound queue (the size of the outbound queue can be customized). Otherwise, the packet is dropped. This scenerio holds for each packet.

3.4. Instrumentation¶

The fundamental feature of agent is instrumentation. This feature makes use of java.lang.instrument framework. A user configuration describes what and where to instrument (see Configurations for details). However, there are some classes which are always excluded from instrumentation:

com.flopsar.*
org.objectweb.*
javassist.*
org.apache.bcel.*
jdk.internal.org.objectweb.asm.*
$*

Important

All exception classes are ignored and not instrumented.

When agent receives a configuration to load, it first parses it and then sorts its instrumentation rules. If there are any changes in the configuration that requires configuration reloading, agent reloads those classes that have been instrumented earlier and those which are required by this new configuration. This classes flow is passed through the configuration filters (Fig. 3.1). The are a few filters operating on two levels: class and method ones. The first filter excludes all the classes from packages, which match the internal exclusion set. Next, it goes through the second, user defined, exclusion class filter. At this point, we have all the classes we want to instrument. In the next step, all the methods from each, selected class are passed through the method filters. At this point, a class can also be excluded from configuration if any of its methods does not satisfy the filters conditions. At the end, we obtain a set of methods which are selected for instrumentation.

Fig. 3.1 Instrumentation Filter

The methods instrumentation is performed in a specific order (Fig. 3.2). First, rules with custom formatters are applied, then rules with standard formatters and finally rules without formatters at the end. For example, if there are two method rules, one instrumenting with a custom formatter and the other one instrumenting only public methods and our method is both public and its signature matches the one specified in the first rule, then the rule with the formatter will be applied.

Fig. 3.2 Instrumentation Order

3.5. Hot Methods¶

This feature enables you to record instrumented methods executions. The Hot methods feature is very useful, when you tune your configuration for minimal performance impact. In this way, you can easily detect the most frequently executed methods and exclude them from instrumentation, unless you really need to instrument them.

When agent starts, this feature is enabled by default. agent simply registers the number of instrumented methods executions. The Hot Methods statistics are dumped every 30 seconds to a log file (see the example below), provided that the logger level is set to at least fine level. This is done for 30 cycles and then is disabled. You can enable it later from workstation (see Hot Methods for details).

>>>>>>>>>>>>>>>>>>>>>>> Hot Methods Report [Cycles left 27] <<<<<<<<<<<<<<<<<<<<<<<<
HM [206633] -> [com/mysql/jdbc/ConnectionImpl][checkClosed\^\_()V]
HM [166145] -> [com/mysql/jdbc/ConnectionImpl][versionMeetsMinimum\^\_(III)Z]
HM [145591] -> [com/konakart/db/KKCriteria][getAsColumns\^\_()Ljava/util/Map;]
HM [109807] -> [com/konakart/db/KKCriteria][getDbName\^\_()Ljava/lang/String;]
...........

The values in the parentheses denote the number of times the corresponding method was called.

3.6. Logging¶

The logging levels are defined in Logging. The log files size is 100 MB in total by default but it can be changed by setting flopsar.logsize option. If the log file reaches its maximum size a new file is created. The logging framework maintains up to 5 log files.

Note

By default, agent always starts with finest logger level and after 2 minutes it is switched to info automatically. You can always change the level in runtime using workstation.

3.6.1. Logger Messages¶

Each logger message entry has the following structure:

DATE [LEVEL]<THREAD> Message content

where:

Message content:
DATE:	current date.
LEVEL:	current logger level.
THREAD:	current Java thread.
	is self-explanatory.