Marcus Hirt's Homepage
Marcus is one of the founders of Appeal Virtual Machines, the company that created the Java Virtual Machine JRockit. He is currently working as Engineering Manager for the JRockit Mission Control team. In his spare time he enjoys coding on one of his many pet projects, and composing music.
The next version of JRockit Mission Control just went live! Feature-wise it is only a minor upgrade, I will go into detail about the most exciting one in a bit, but there are some other changes to this release that are quite profound.
The, in my humble opinion, greatest change is that all JRockit Mission Control features are available by default. There is no need to dabble with licenses. Everything is free to use for evaluation, which among other things means that you can now try out the latency analyzer properly.
JRockit Mission Control has also been relocated to a new homepage at OTN, located at http://www.oracle.com/technology/products/jrockit/missioncontrol/index.html. The next few days a lot of new JRockit Mission Control related material will be posted there.
The update site has moved as well, and can now be found at http://www.oracle.com/technology/software/products/jrockit/missioncontrol/updates/eclipse-3.3/jrmc/.
As you will undoubtedly notice, everything has also been re-branded.
Now, the cool new feature is actually a JRockit JVM side upgrade, that will be automatically picked up by the JRockit Mission Control client. It is now possible to do very cheap memory allocation profiling using the JRockit Latency Analyzer.
There is a new type of latency events available in the latency recordings. They are triggered whenever the JRockit JVM needs to acquire a new Thread Local Area or whenever the JRockit JVM needs to allocate a new large object. These events provides a good approximation to what is causing the most pressure on the memory system. This means that it is quite easy to find out where to start optimizing allocation behaviour using JRMC LAT.
Here is an example:
Reducing the pressure on the memory system leads to less frequent garbage collections and thus a more responsive system.
For more information on the JRockit JVM R27.6 release, please have a look at the release notes.
It's finally here! Mission Control is now available as a set of plug-ins for Eclipse! There is more information on this and what's new in my previous blog entry.
The link to the update site is https://dev2devclub.bea.com/updates/eclipse-3.3/jrmc/.
Have fun!
Within the next few days, Mission Control will be available for download directly into your IDE from an update site. To fully take advantage of this, however, you need to run Eclipse on a JRockit. This blog is an update to an old blog of mine, which explains how and has a few very specific tips for the insanely rich or the ones working for BEA. ;)
Despite JRockit being an outspoken server side JVM, many long running client applications, like IDE's, actually run very well on JRockit. Running Eclipse on JRockit is no exception.
The easiest way to change the JVM on which to start Eclipse is by modifying the eclipse.ini file. You can also set the -vm on the command line used to start Eclipse. In windows you'd typically do that in the shortcut you use to launch Eclipse. Just using JRockit out of the box yields pretty good performance. Here is an example ini file:
-showsplash
org.eclipse.platform
-vm
D:\jrockits\R27.5.0_R27.5.0-111_1.5.0\bin\javaw.exe
You can usually get better performance by setting the initial heap size and the max heapsize to the same, thus bypassing the dynamic growing/shrinking of the heap:
-showsplash
org.eclipse.platform
-vm
D:\jrockits\R27.5.0_R27.5.0-110_1.5.0\bin\javaw.exe
-vmargs
-Xms384m
-Xmx384m
The insanely rich (or BEA employees) can run Eclipse on WLRT/deterministic GC (some would call this overkill) for a very smooth experience. This is the eclipse.ini I use myself:
-showsplash
org.eclipse.platform
-vm
D:\jrockits\R27.5.0_R27.5.0-110_1.5.0\bin\javaw.exe
-vmargs
-Xms384m
-Xmx384m
-XgcPrio:deterministic
-XpauseTarget:20
-XXcompactratio:1
One common pitfall is trying to use Sun specific flags, which would cause the JRockit launcher to exit right away, like for example -Xincgc or -Xbatch.
For more information:
Mission Control just got even better! Some improvements will be quite obvious and very visible. Other improvements are subtle usability fixes that simply makes Mission Control more pleasant to use. In this blog I'll briefly list some of the new features, in no particular order, as well as some new freely available services.
For the first time we're making Mission Control available as a set of Eclipse Plug-ins! This means you will have the power of Mission Control readily available from within your IDE. It also means you will be able to update Mission Control from within the IDE.
(The publishing of the update site has been delayed due to errors in the license agreement text. It should be published within the next few weeks. Details on how to access the update site will be published on the Mission Control Home Page, when available.)
The Eclipse integration comes with basic JDT integration, such as the ability to jump to source from stacktraces, classes and methods being displayed in Mission Control.
Running Mission Control withing Eclipse requires a 3.3 version of Eclipse or above. Also note that, to get the most out of Mission Control running in Eclipse, you should run Eclipse on JRockit. More on interesting ways of doing that in an upcoming blog. :)
There is now a public bugzilla for Mission Control available at bugzilla.bea.com. It is monitored by the Mission Control developers and provides an efficient means for posting change requests and suggestions for enhancements.
A lot of GUI changes have been made to make Mission Control play nice in an Eclipse environment. The meny layouts have changed. The JRA wizard has been improved greatly. JRA recordings can now be stored in an Eclipse project as a resource and launched by just double clicking (starting the default editor) on the resource. The memory consumption has been lowered and performance improved (yes, we doggy fed Mission Control to itself ;) ). Even more performance improvements will be available in the next major version of Mission Control.
You can now export charts from the Console as images. You can synchronize the charts X-ranges. You can add and remove charts from the Overview in the Console. You can, after freezing the updates of a chart, hover over data points to get exact readings. There are filters available to more easily find specific MBeans. And much, much more.
The Dial concept is back! Some of you may remember them from the old days of the Management Console. They are configurable and can show any attribute you want. And, if you do not like them, you can simply remove them altogether from the overview.
There is now more data available in Mission Control. In JRA we show the nursery size before and after a young space collection. We can record lazy unlocking profiling data. In the latency recorder we're recording more information in the events (for instance thread transition information for park/unpark).
And we have of course fixed problems found, for instance Memleak not properly updating the trend table under certain conditions. Please see the release notes for further information.
There is a new Mission Control out with new features and better community support. Try it out and let us know what you think!
For more information:
One of the parts of Mission Control is the JRA (the JRockit Runtime Analyzer). The JRA is a powerful profiler that can tell you a lot about the performance bottlenecks in your application whilst keeping the overhead barely noticable.
There are new options added to the JRA recordings almost every release. In recent releases of Mission Control the JRA wizard will check what capabilities the JRockit you are attempting to start a recording on has, and present you with configurable options accordingly. The problem is that the legacy protocol used when connecting to a 1.4 version of JRockit unfortunately does not support querying for the available parameters. Do not despair; there are still means of starting JRA recordings that contain all the new goodies. This is how:
There is a neat little command line tool called jrcmd that can be found in JROCKIT_HOME/bin. The jrcmd tool can be used to invoke commands on a locally running JRockit process. It works like this:
JROCKIT_HOME/bin/jrcmd to list the JRockit processes on your machine. JROCKIT_HOME/bin/jrcmd <pid> help JROCKIT_HOME/bin/jrcmd <pid> help startjrarecording In this case, the command we want to use is startjrarecording, and what we want to know is what parameters we can use on a particular JRockit when issuing the startjrarecording command.
On the windows box I'm currently on, it looks like this:
D:\jrockits\R27.3.0_R27.3.0-106_1.5.0>bin\jrcmd.exe 2640 com.jrockit.mc.rcp.start.MCMain 5012 jrockit.tools.jrcmd.JrCmd
Notice how it lists itself. In this case I'm interested in knowing the available commands on the JRockit currently running my Mission Control (com.jrockit.mc.rcp.start.MCMain), so I do:
D:\jrockits\R27.3.0_R27.3.0-106_1.5.0>bin\jrcmd.exe 2640 help
2640:
The following commands are available:
kill_rmp_server
start_rmp_server
kill_management_server
start_management_server
checkjrarecording
stopjrarecording
startjrarecording
print_object_summary
memleakserver
print_class_summary
print_codegenlist
run_optfile
dump_codelayout
dump_codelist
dump_codemap
print_utf8pool
print_properties
print_threads
datadump_request
runsystemgc
runfinalization
heap_diagnostics
oom_diagnostics
print_exceptions
version
timestamp
command_line
memprof
sanity
verbosity
set_filename
help
print_memusage
For more information about a specific command use 'help '.
Parameters to commands are optional unless otherwise stated.
To find out what parameters are available to me when starting a jrarecording I do:
D:\jrockits\R27.3.0_R27.3.0-106_1.5.0>bin\jrcmd.exe 2640 help startjrarecording
2640:
Starts a JRA recording.
filename - name of the file to store JRA recording to
(string, jrarecording.xml)
recordingtime - length of the recording in seconds (int, 60)
delay - delay before starting recording in seconds (int,
0)
methodsampling - enable method sampling (bool, true)
gcsampling - enable gc information (bool, true)
heapstats - include heap statistics (bool, true)
nativesamples - include native code in sampling (bool, false)
methodtraces - include stack traces (bool, true)
tracedepth - depth of stack traces (int, 16)
sampletime - time between samples in milliseconds (int, 30)
zip - zip the recording (bool, true)
hwsampling - use hardware sampling if possible (bool, false)
threaddump - do full threaddumps at start and end of recordning
(bool, true)
threaddumpinterval - also do threaddumps every 'n' interval (can be
specified as x[ns|ms|s]) (time, 0s)
latency - include latency analysis (bool, false)
latencythreshold - do not record events shorter than this number (can
be specified as x[ns|ms|s]) (time, 20ms)
cpusamples - sample cpu usage during the recording (bool,
true)
cpusampleinterval - cpu sample interval (can be specified as
x[ns|ms|s]) (time, 1s)
In this case I found out that I can indeed start up a recordings with new features such as latency analysis data on this particular JVM. To actually start such a recording I could for example write:
D:\jrockits\R27.3.0_R27.3.0-106_1.5.0>bin\jrcmd.exe 2640 startjrarecording laten cy=true latencythreshold=20ms recordingtime=120s 2640: JRA recording started.
The JRockit JVM incorporates technology (the JRockit Deterministic Garbage Collector) which can guarantee an upper bound for worst case pause times for garbage collections. This greatly helps in reducing unexpected latencies. When writing Java applications, however, it is still possible to introduce latencies through non-optimal use of Java Synchronization, I/O and so on. These latencies can cause poor perfomance and/or response times in your application and can be quite hard to find. The latency analyser provides a simple way to analyse where such latencies are caused.
The JRockit Latency Analyser is built into JRA (JRockit Runtime Analyzer) in Mission Control 3.0. When starting a JRA recording it is now possible to select a Latency Recording Profile.
When such a profile is selected, or Enable Latency Recording is selected from the advanced options, the recorded profile will contain latency information. When a recording contains such information, three additional tabs (Latency Log, Latency Graph, Latency Traces), two views (Event Types, Properties) and a new Latency Perspective will be at your disposal.
The Latency Log tab includes a log which simply lists all the events in a table and lets you filter them.
The Latency Graph tab includes a graph, showing for each thread which latency events that have happened during the duration of the recording.
The Latency Traces tab includes a tree table view showing you stacktraces that have been involved in causing these latencies, and how much latency has been caused by the traces leading to a certain node.
In the example below we have found the response times of our latency sensitive application somewhat disappointing. The CPU isn't saturated, so we've decided to make a JRA recording with latency information in it. We're now viewing the recording in the latency graph (below). At the top of each view is the range selection tool, where you can zoom and pan to filter the events over time. I've zoomed in on an arbitrary part of the recording to show how the range selector looks when zoomed in. This particular recording shows the same problem all over the recording, so panning to another part would yield a very similar image.
From the graph we can see that we seem to do quite a lot Java Synchronization. In the log view (screen shot below) we can look at each of the individual events. In the screen shot I've filtered on the Java Synchronization Events and added the first few of them to the Operative Set. They are the ones highlighted in the Range Selector. Seems a good portion of them are indeed the Java Synchronization Events. Another way to select only the Java Synchronization Events would be to deselect the other Event Types in the Event Type view.
If we switch to the trace view, we can see that almost all the latency is caused from calls to a java.util.logging Logger.
Now, if we remove those logging calls, almost no Java Synchronization events remain, and thus the latency caused by those events have been eliminated. See below for a picture of a recording with the logging calls eliminated.
JRockit Mission Control 3.0 is included with the JRockit R27.3 JDK, which was recently made generally available. For more information about the features in Mission Control 3.0, please see the R27.3 release notes and the Mission Control Documentation.
For information about what was introduced with Mission Control 2.0, please see my previous blog entry - Mission Control 2.0 Sneak Peak
For information on how to obtain a free evaluation/developer license, please visit the Mission Control homepage.
The JRockit Memory Leak Detection tool (henceforth abbreviated Memleak) is part of JRockit Mission Control 2.0, the JRockit tools suite. It has served many people quite well in hunting for memory leaks in Java, i.e. hunting for the cause to why objects that shouldn't be in use still have references to them.
How to hunt for these memory leaks using the Memleak has been discussed to death, so this blog entry is going to show you other fun filled uses of the tool. ;)
First off we'll explore how Memleak can help us look at how any given type is being used within the JVM.
Start Mission Control and hook it up to a JRockit doing pretty much nothing but sleeping. In my case I'm connecting it to a JRockit on the network that was discovered using the JRockit Discovery Protocol.
You may want to pause the updating of the trend table by pressing the pause button in the toolbar above the table, especially if JRockit is very busy.
After pausing the updating of the table, locate a class for which you want to explore the dependencies, say Hashtable entries (java.util.Hashtable$Entry), and select Show Referring Types from the context menues to see what is pointing to that particular type.
You'll find that, not surprisingly, the entries are pointed to by hashtable entry arrays. Click the plus sign to work your way backwards, and you'll find out that, again not very surprisingly, various hashtable clones are pointing to these arrays. Click Hashtable to expand who is using Hashtables. Woooah!? We get a funny arrow directly to Hashtable from our Entry, thus completing a cycle in our graph.
Hmmm. Someone is obviously putting Hashtables into Hashtables. Now, who does that? Let's look at the instances that are taking part in that particular relationship. Bring up the context menu on the java.util.Hashtable$Entry, and select show instances. A dialog will pop up, prompting you to select for what relationship you want to list the instances. We select Hashtable, since we want to find out what entries that are pointing to hashtables. We get a list of entries, in my case 5 candidates.
As we can see, there are four of them keeping alive more than 100 000 bytes of data. Let's pick the top one - bring up the context menu and select to show who's pointing to that particular instance (Show Referring Instances).
Working our way backwards we see that it's the com.sun.jmx.mbeanserver.RepositorySupport holding on to a Hashtable that contains Hastables. We can now select the RepostiorySupport instance and verify that it indeed holds Hastables by bringing up the context menu and selecting "Inspect Instance".
Summary:
In this example we used the Memory Leak Detector in Mission Control for looking at relations between various types and instances in our running JVM. In this particular case we followed up on where people are putting hashtables into hashtables.
This is an updated version of my JRCMD goes remote blog entry. In version R27 JRockit had a major overhaul of the MBeans used to expose JRockit specific manageability features. One major difference is that the JRockit specific features are not loaded into the platform MBean server by default. To load them, a specific MBean must first be created, which in turn will load the rest of the MBeans exposing the API. The MBeans also moved to a new domain: "bea.jrockit.management".
The MBean to load is the bea.jrockit.management:JRockitConsole MBean. This can be done by simply using the MBeanServerConnection, for example like this:
MBeanServerConnection#createMBean( "bea.jrockit.management.JRockitConsole", null);
The following example is an altered version of my old JRCMD Goes Remote blog that, just like the JRCMD, allows the performance counters and ctrl-break handlers to be read/executed - but remotely from another machine.
