Metagnostic
home

JNIPort for Dolphin Smalltalk

Overview

Contents

Players

Layers

Examples

Configuration

InFAQ

Changes

Licence


Back to Goodies

JNIPort Players

These two sections introduce the main players in the community of JNIPort objects and try to explain something of how they interact.

This presentation is about the objects, which is in contrast to the alternative presentation as a breakdown into the three layers of the software. Here I mostly ignore anything below layer two, since it's only used in the implementation of these concepts. I also ignore the distinction between layers two and three; the topmost layer just provides ways of automating what layer two does without materially changing the cast of objects.

Another approach to understanding the basic concepts of JNIPort is this overview of the classes involved.

Please note: this description has become slightly out of date. It is still accurate as far as it goes, but it should be updated to discuss the JVM's role as a <javaClassFinder>, and how the new “supplementary classloaders” fit into the picture. The section on the class lifecycle should be updated to mention the additional possibility of being converted into a “lazy ghost”.

The JVM Object

The central actor in JNIPort is the is the JVM object, an instance of class JVM (or a subclass). This represents one running Java VM in the Smalltalk space (the running VM is actually a DLL supplied by Sun, IBM, or whoever). The job of the JVM is mostly coordination, it is the one object that all other objects in the system know about, and it allows them to find each other. The JVM also manages the process of starting up the Java VM and the JNIPort runtime machinery.

For a user of JNIPort, the main function of the JVM is that it knows how to find the objects representing Java classes — the “class statics”, of which more below. It also coordinates the class registry, the class index, and the callback registry, these are important parts of the implementation of JNIPort; but are (to varying degrees) less relevant to JNIPort users.

JNIPort is designed to allow several JVMs to exist at once, since that is the way the JNI API is structured. In practice, there is little need for this (in fact, the current crop of Java implementations don't support having more than one in the lifetime of a process). It is possible to use JNIPort as if the JMV were a Singleton, and the JVM class has a few methods like #current (which answers “the” running instance if there is one) and #default (which is the same, but will also start up a default instance if necessary), to help with this.

Class Statics

There is a great deal of truth in the observation that “in Java, classes are not objects”. The “static” fields and methods of a Java class are not associated with any object, they float in a sort of global space with the Java class serving only to delimit a namespace and define accessibility. Of course, for Smalltalk, that Just Won't Do…

JNIPort represents Java classes and interfaces by class statics. These objects (instances of subclasses of JavaStatic) stand in a 1-to-1 relationship with Java classes. They have similar roles to Smalltalk classes (though they are not kinds of Behavior), they:

  • Know how to invoke the static methods of the Java class.
  • Know how to read and write the static fields of the Java class or interface.
  • Know how to create instances of the Java class.
  • Know their place in the Java class hierarchy.
  • Know how to do reflection.

In addition, they can be specialised to a particular Java class, so they may also:

  • Have methods corresponding to the static methods of the Java class.
  • Have getter and setter methods for the static fields of the Java class or interface.
  • Have factory methods corresponding to the constructors of the Java class.

More strictly, the 1-to-1 relationship is that within the context of a JVM (since each JVM has its own collection of class statics) there is a class static for each known Java class, interface, or type. For instance java.lang.String (a class), java.util.Iterator (an interface), java.lang.String[] (an array class), and byte (a primitive type), all have corresponding class statics. The class statics for the array classes and primitive types are mainly used by the internal machinery of JNIPort, although the array class statics do act as factories for creating new Java arrays.

Each class static is paired with a Java instance wrapping a reference to a java.lang.Class. The class instance is the #classObject of the class static, and the class static is the #classStatic of the class object.

Whenever a new Java class becomes known to JNIPort, it is assigned a corresponding new class static. JNIPort can become aware of classes in several ways, including:

  • Calling JVM>>findClass:.
  • Seeing an instance of a class.
  • Seeing a subclass, or an instance of a subclass.
  • Using Java's introspection methods such as java.lang.Class.forName().

The known class statics are held in the class registry, where they normally stay for the lifetime of the JVM, although there are exceptions during bootstrapping. A later version of JNIPort will provide some means of “retiring” class statics that are no longer needed or used (thus allowing the Java class itself to be garbage collected), but this version has no public API for doing so.

Class statics are specialised by subclassing. You can create a subclass of JavaStatic and arrange that some Java class, and all its subclasses, will have class statics that are instances of your subclass. This is set up by the class registry during bootstrapping. These specialised subclasses are the class-side wrapper classes, and they are where the wrapper methods live; you can also add whatever helper methods you want or need. You can create wrapper classes by hand, although it's tedious (as I know — I hand-wrote many of the wrapper classes supplied as part of JNIPort), or you can generate them automatically with the help of the wrapper generation wizard, or you can have them generated dynamically by using ghost classes.

Class statics have another vital role that is internal to JNIPort: they know how to wrap references to Java objects in Java Instances of the right kind. Each class static knows a Smalltalk class to use to wrap references to instances of its Java class, its instance class. The instance class is normally the same as the instance class for the Java superclass, but that can be changed during bootstrapping, and when ghost classes are used. This is discussed in the section about the lifecycle of an instance, below.

One particular class static corresponds to the Java class java.lang.Class, and that object has a special role to play in the mechanism of JNIPort. This is described in the section about the lifecycle of a class static, below.

Java Instances

JNIPort represents references to Java objects by instances of (subclasses of) JavaClassInstance. Each instance has a reference to its (shared) class static.

It is important to realise that these wrap references to Java objects, and that there can be more than one reference to the same object. The #= and #hash methods are defined so that they test for the Java identity of the referred-to objects, so #= is true of two references when they refer to the same Java object, even though they may not be #== to each other. (By the way, the methods #equals: and #hashCode are the equivalents of the java.lang.Object.Equals() and java.lang.Object.hashCode() methods, respectively).

In some cases, it is desirable or necessary to ensure that references are unique. JNIPort includes a facility for making a reference to a Java object canonical; and when this is used, any subsequent references to that object will be mapped to the canonical instance. This is available on a per-instance or per-class basis. It is not the default for most classes, since the “interning” is expensive and it is not normally needed.

Internally to JNIPort, instance wrappers have the important role of managing the lifetime of the object reference. The Java VM needs to be told when references are discarded, or else it can't do garbage collection properly, so instance wrappers are finalisable objects that explicitly release the JNI reference when they die. Some of the code refers to this, and the rest of the mechanism of setting up wrapper objects, as managed objects.

For the user of JNIPort, the important thing about these objects is that they act as proxies for the underlying object:

  • Know how to invoke the methods of the Java object.
  • Know how to read and write the fields of the Java object.
  • Know what class static they “belong to”.

In addition, they can be specialised to a particular Java class, so they may also:

  • Have methods corresponding to the methods of the Java object.
  • Have getter and setter methods for the fields of the Java object.

As with class statics, proxies are specialised by subclassing. In this case the custom instance-side wrapper classes are subclasses of JavaClassInstance rather than JavaClassStatic, but the idea is otherwise the same.

It might be worth explaining that instances can be either local or global; this is a JNI concept. JNI has two different kinds of references to objects, and they are used and managed in slightly different ways. Local references ought to be more efficient, but need to be managed more carefully. It is possible that a future version of JNIPort will not use local references at all. In any case, for user code, there should be no difference between local and global.

Other Players

There are some other notable objects in the JNIPort community. These are mostly concerned with the implementation of JNIPort, but they are intended to be visible to the JNIPort user.

The Class Registry

There is one class registry for each JVM object. It holds the list of class statics that JNIPort knows about.

Its role is mostly to ensure that every Java class and type is wrapped in exactly one class static (for more details see the class lifecycle section, below). However it is also able to provide lists of all the known classes, all known subclasses of some superclass, and so on.

The Class Index

There is one class index for each JVM object. It holds the mapping from the names of classes and types (represented as Symbols, in fact) to the corresponding class statics. Classes may appear under more than one name (because of the weird JNI naming rules, which are not relevant here — fortunately); also not all known classes will appear in the index, only those that have been looked-up by name. The index is effectively a cache to allow JVM>>findClass: to avoid going back to the Java runtime for known classes (which would be slow).

(Design note: The class index is actually a minor player, but it has a nasty habit of trying to vanish and get its role absorbed into that of some other object — which it then complicates unnecessarily. So I've dragged it out into the light of day as a public player to try to stem the evil tendency.)

The Callback Registry

There is one callback registry for each JVM object that supports callbacks. It holds references to the objects that “tag” callbacks (see the section on callbacks), and is responsible for dispatching the incoming callback requests or notifications to the correct handler for each tag.

The Ghost Class Maker

There may be one ghost class maker for each JVM object. If there is, then its role is to wait for new Java classes to be registered and then drive the creation of corresponding new ghost classes. This is discussed a bit more under class lifecycle, below.

The ghost class maker is really only a public object because it has to be configured.


The Play

This section is about the interactions between the main objects described in the first section. This is almost all about the implementation of JNIPort — user-defined objects have no special role — but an overview of how the system works may be helpful.

The Lifecycle of an Instance

The process starts with a raw JNI reference to some Java object; it may have been returned from a Java method call, for instance. Before JNIPort can do anything with the reference it must be wrapped in a Smalltalk object. The process normally starts with the JVM object…

The JVM object's role in life to put other objects in touch with each other. In this case the raw JNI reference needs to find a class static that will generate a wrapper/proxy for it.

The JVM object starts by using low-level JNI features to get a reference to the Java object's class. This will answer a raw JNI pointer to a class object, and the JVM will find a corresponding class static for it as described in the next section. The JVM will then ask the newly discovered class static to wrap the original raw JNI reference.

The class static may have canonical instances, if so then it must first check to see if it already has a proxy for the same Java object. It checks its object registry (essentially just a weak set), if it has one, to see if there is a match. If so then it releases the raw JNI pointer immediately, and returns the pre-existing proxy to the caller.

Otherwise it creates a new proxy object. This will be an instance of the class static's “instance class”, it will hold the raw JNI reference and know that it belongs to this class static.

If the particular class static is supposed make all instances canonical, then it enters the new instance in its object registry. Finally it returns the new instance to the JVM.

An optimisation to this process is possible when the correct class static is known (with certainty) before the raw reference is created. In such cases there is no need to ask the JVM to wrap the reference, the class static can be asked directly (which saves an expensive class lookup). Some cases where this optimisation can be applied are:

  • When a JNI method is called that is defined to return a reference to a class object
  • When a constructor is called (constructors always return an object of the exact class; they can never answer an instance of some subclass).
  • When an element of an array of a “final” class is read.
  • When a method that returns an instance of a “final” class is called.
  • When a field holding an instance of a “final” class is read.

The first three cases are easy to code for and JNIPort applies them wherever possible. The last two cases need significantly more messy code, so JNIPort only uses those optimisations in ghost methods.

This process (of looking up a class static then creating a wrapper object) may sound slow, and in fact it is. On my P3 650Mz laptop, JNIPort can wrap a maximum of between 10K and 20K object references per second, depending on which of the optimisations are applicable.

The Lifecycle of a Class Static

The previous sub-section and this one are quite closely intertwined; the lifecycle of an instance is strongly affected by its “onwing” class static, but the class static is paired with an instance of java.lang.Class. This makes it difficult to describe one separately from the other, it's a chicken-and-egg situation. The same problem occurs in the code too, and requires special handling as part of bootstrapping.

The lifecycle of a class static starts when the JVM object is asked to find a wrapper for a raw JNI reference to an instance of java.lang.Class. This is a special case of wrapping any Java reference, and the previous section talked about that in some detail, so it should suffice here to say that the JVM will find the class static corresponding to java.lang.Class and then ask it to create a proxy for the Java object.

We'll call the specific class static for java.lang.Class the Class class static (for lack of a snappier name). The Class class static now consults the class registry to see if it already has a proxy for the same object. If it has, then it then it releases the raw JNI object reference and gives the pre-existing wrapper back to the caller. This is an extension of the mechanism for making objects canonical; all instances of java.lang.Class are automatically interned, and kept as strong references in the class registry.

If the class registry does not find a pre-existing wrapper, then the Class class static creates an instance of the correct proxy class for java.lang.Class, in fact a JavaLangClass, and adds it to the registry before passing it back to the JVM.

So far this hardly differs from the normal processing for wrapping an instance of any class with canonical instances (there are some differences in the code, but those are mostly about efficiency, e.g. not using Mutexes unnecessarily). What is different in this case is that the Class class static also creates a new class static and pairs it with the new class object before registering it.

To create the new class static, the Class class static must first choose a Smalltalk class for it (I'm sorry if this is getting confusing — it goes with the territory, like the “metaclass knot” in Smalltalk). To do this it tries to find the most applicable subclass of JavaStatic according to the following rules:

  • If the class is an array class then use JavaArrayClassStatic.
  • If the class is an interface then use JavaInterfaceStatic.
  • Otherwise the class must be a subclass of java.lang.Object; use the same class static class as is used for the target's Java superclass. That may involve re-entering this procedure to create a class static for the superclass, but the recursion must eventually end with java.lang.Object, if not before.

You may notice that these rules seem to imply that every normal class will be given the same kind of class static (and you may also notice that Java's primitive types aren't handled at all by the rules). The answer is that special arrangements are made during bootstrapping to ensure that selected Java classes are given the “right” class statics. For instance the class static for java.lang.ClassLoader is an instance of the specialised subclass, StaticJavaLangClassLoader (yeah, I know, the names are getting longer and longer…). As a result, once JNIPort is up and running, if it sees any new subclass of ClassLoader, then it will give it a class static that is also a StaticJavaLangClassLoader.

The new class static and the new class object are now married together and initialised. They will stay associated for the rest of the lifetime of this JVM, and any future references to the Java class will be mapped into the existing class static. The class is now considered to be registered.

The instance class of the new class static is set to be the same as that for the Java superclass. This seems to imply that the same kind of proxy is used for all Java references. The answer is roughly the same as for class static classes: the instance class can be changed during bootstrapping. For instance, during bootstrapping the class static for java.lang.String has its instance class set so that any references to Java strings will be wrapped in instances of JavaLangString. (So too would subclasses of java.lang.String, if it had any.)

One final flourish is worth mentioning: if ghost classes are in use, then at this point the ghost class maker will be informed that a new class has been registered. Before the class static is returned to the JVM (remember, this all started with the JVM being asked to wrap a JNI reference?), the ghost class maker will convert it into a ghost class. Details are here but, briefly, it will:

  1. Generate new ephemeral subclasses of the class static class and the instance class.
  2. Use Java introspection to discover what fields and methods the Java class has.
  3. Populate the ghost classes with methods that use the low-level JNI APIs to call those methods or get/set those fields.
  4. Replace (using #become:) the class static by an instance of the new ghost static class, and tell it that it should use the new ghost instance class.

The configuration controls exactly which classes are converted into ghosts and which members of those classes are exposed via ghost methods.

Bootstrapping

JNIPort, being fairly complicated internally, has a correspondingly elaborate startup sequence. Most of it is routine, even dull, and is in any case is not relevant to JNIPort users. One aspect of it, however, is relevant: the process of bootstrapping. This process initialises the class registry and sets up the mechanism for wrapping Java classes and objects.

I call it “bootstrapping” because that mechanism is managed largely by the class static for java.lang.Class, but that object (a class static like any other) depends on the machinery. So there is some slightly tricky code for avoiding the circularity while setting up the initial objects in the class registry. The class statics for java.lang.Class and java.lang.Object and instance classes for java.lang.Class, java.lang.Object, and java.lang.String all refer to, and depend on, each other. Once that knot of mutually-referential objects has been created, the class registry enters the second phase of bootstrapping.

The second phase is where custom subclasses of JavaInstance and JavaStatic are registered as the preferred wrapper classes for corresponding Java classes. The mechanism might strike you as a bit of a hack (it should, because it is a hack), but it is reasonably simple, and it is normally only used during bootstrapping.

Each subclass of JavaInstance or JavaStatic is given a chance to register with the JVM object that is starting up. The class is sent the #registerWrapperWith: message (passing the new JVM as a parameter). Superclasses are always asked to register before their subclasses, but there is no defined ordering other than that. The default handing is as follows:

  • If the class does not think that it is a wrapper class (it may be an abstract superclass of concrete wrappers) then it ignores the message.

  • Otherwise, the class will do a #findClass: on the name of the Java class that it wraps. That will answer a class static in the normal way, but because the class registry is not yet fully initialised, the new class static will not reflect the intended custom wrapper classes. So that has to be fixed.

  • If the Smalltalk class is an instance-side wrapper, let's say the Smalltalk class is JavaIoReader which is intended to wrap instances of java.io.Reader, then the instance class of the new class static will be wrong. It'll probably be JavaLangObject — the default. In this case JavaIoReader can register itself as the proper instance class by changing the instance class reference held by the class static.

  • If the Smalltalk class is a class-side wrapper, for example StaticJavaLangSystem corresponding to java.lang.System, the new class static will be an instance of the wrong class (it'll be a StaticJavaLangObject). So StaticJavaLangSystem creates a new instance of itself, initialised with the data from the “wrong” class static, and uses #become: to replace the old with the new.

  • A small subtlety here is that after the class static has been modified or replaced, the class registry must be updated by making corresponding changes to any other class static that have “inherited” its settings. In the current implementation this is unnecessary, since there won't be any, but that cannot be relied on for future implementations. The nett effect is that any particular class static may change several times during bootstrapping.

By the time bootstrapping is finished, each Java class that has a custom wrapper class (on class or instance side) will have a class static in the registry, and that static will reflect the intended wrappers. The class static and instance class should now be stable for the rest of the life of the owning JVM. The exceptions are when ghost classes are used (in which case the ghost classes will be installed later using much the same mechanism) or when you manually register a new wrapper class (say, by using the wrapper generation wizard).

You may notice that instance wrappers and class-side wrappers are installed independently — a Java class may have an instance wrapper, or a class wrapper, or both, and if it has both then the two do not need to know about each other. Also there is no requirement that the class hierarchy of the wrapper classes follows the Java class hierarchy, though it would be confusing if the two were too far out of synch. For instance, the JNIPort wrappers for java.lang.String and the various Java array classes all share an abstract superclass, JavaAggregate, even though there is no formal relationship between them in Java.


Copyright © Chris Uppal, 2003-2005

Java, JNI (probably), JVM (possibly), and God knows what else, are trademarks of Sun Microsystems, Inc.