I've mentioned previously that we have spent a lot of time on ASN.1 in this release cycle, so I better say something about it. However this article won't be an introduction to ASN.1, I want to focus on the improvements in our implementation, but there are some easy introductions available and even free books for the gory details.

I figured that the best way to demonstrate the framework is to show how it's used in an application, and our most interesting application so far is the X.509 framework, so let's take a look at that. The X.509 framework is structurally fairly simple, you have a hierarchy of X509Objects representing various components of an X.509 Certificate and few supporting classes like X509Registry or specific exception classes. The job of the ASN.1 framework is to turn X509Object instances into DER encoded bytes and back. To be able to do that the framework needs a structural description of the encoded bytes. It needs to know that an encoded certificate starts with an encoded TBSCertificate, then an identifier of the algorithm used to sign the contents of the TBSCertificate (TBS stands for 'to be signed' here) and finally the bytes of the signature itself. ASN.1 describes this structure using a C-ish notation. The Certificate definition looks as follows:

	Certificate  ::=  SEQUENCE  {
		tbsCertificate       TBSCertificate,
		signatureAlgorithm   AlgorithmIdentifier,
		signatureValue       BIT STRING  }

A SEQUENCE is like a struct in C and the elements show name first and type second. Our framework represents this information with a structure of ASN1.Type objects, closely following the ASN.1 expressions. In this case it would be something like

	(SEQUENCE name: #Certificate)
		addElement: #tbsCertificate type: TBSCertificate;
		addElement: #signatureAlgorithm type: AlgorithmIdentifier;
		addElement: #signatureValue type: BIT_STRING;
		yourself

This expression will not work correctly though, because the #type: arguments would have to be other ASN1.Type objects. If you build the type objects in the right order, making sure all the component types are created before the containing types, you might be able build the structure by hand, however that would be very inconvenient to maintain. It is OK for relatively simple structures, like

"	RSAPublicKey ::= SEQUENCE {
		modulus           INTEGER,  -- n
		publicExponent    INTEGER   -- e }
"
	(SEQUENCE name: #RSAPublicKey)
		addElement: #modulus type: INTEGER;
		addElement: #publicExponent type: INTEGER;
		yourself

For the more complex cases the framework provides a more convenient mechanism, a Module. An ASN.1 Module is a container for a set of related Type definitions and consequently provides a context for lookup of types by name. As soon as you put a SEQUENCE into a Module, you can define its elements using type names instead of full instances and it also takes care of resolving forward references, i.e. you can define types in any order you wish, their mutual references will be resolved properly as the type definitions get added. The X509 framework maintains its module in a shared class variable X509Object.ASN1Module. Using a module the type definition for Certificate can look as follows:

	module := Module new: #X509.
	tCertificate :=
		(module SEQUENCE: #Certificate)
			addElement: #tbsCertificate type: #TBSCertificate;
			addElement: #signatureAlgorithm type: #AlgorithmIdentifier;
			addElement: #signatureValue type: #BIT_STRING;
			yourself

Once we have the type definitions in place the marshaling framework knows enough about the encoded bytes, however in order to be able to map objects to bytes, it needs to know how the types correspond to classes. For most of the "simple" types there's predefined correspondence, i.e. BOOLEAN maps to Booleans, INTEGER to Integers, etc. However SEQUENCE and SET types default to instances of ASN1.Struct which is kind of like a Dictionary but with few convenience gimmicks, like you can use the element names as accessors and such. But we don't want a Struct instance for Certificates, we want instances of Certificate class. That's why the SEQUENCE and SET types have an optional 'mapping' attribute. You can tell it to map to a given Smalltalk class. It is responsibility of the developer to make sure that the class provides all the expected accessor methods. The Certificate class does that of course, so all that needs to be done is to tell the type about it:

	tCertificate mapping: Certificate

The last type feature to discuss is encoding retention. X.509 has a fairly tortured history and the practical outcome of it is the rule to never ever re-encode a certificate. Therefore it is desirable for a certificate imported from outside to retain its DER encoding, in case it needs to be exported again. The retained encoding also serves as a cache, so writing out an object with retained encoding can simply dump the retained bits, instead of going through the encoding process.

Any Type can be told to retain its encoding. An encoding is captured in an instance of ASN1.Encoding pointing to the relevant bytes. The framework will pass the Encoding to the corresponding object using #_encoding:type: message. The default implementation in Object will wrap the object in a TypeWrapper which has a slot to capture the encoding, however it is expected that most applications will simply allocate a slot directly in the objects that will retain encoding and therefore will most likely override the method to store the Encoding there (see Certificate>>#_encoding:type: for example).

Note also that the encoding retention behavior was factored out of the marshaling machinery into a standalone EncodingPolicy object and is therefore completely customizable. An interesting side-effect of this is that with customized EncodingPolicy you get a chance to intervene with the marshaling at interesting points in the process. One possible exploitation of this, that was very handy for us while trying to figure out bugs in the marshaling process was the PrettyPrinter policy which produces a map of the bytes on a text stream while marshaling.

Here's an example. Let's take something simpler than Certificate, for example the Name used for issuer or subject fields of Certificate. The full ASN.1 definition of Name is somewhat complex, but this is the part relevant to our example:

"	Name ::= CHOICE { RDNSequence }
	RDNSequence ::= SEQUENCE OF RelativeDistinguishedName
	RelativeDistinguishedName ::= SET OF AttributeTypeAndValue
	AttributeTypeAndValue ::= SEQUENCE {
		type     AttributeType,
		value    AttributeValue }
	AttributeType ::= OBJECT IDENTIFIER
	AttributeValue ::= ANY DEFINED BY AttributeType
"
	(module CHOICE: #Name)
		addElement: nil type: #RDNSequence;
		retainEncoding: true.
	module SEQUENCE: #RDNSequence OF: #RelativeDistinguishedName.
	module SET: #RelativeDistinguishedName OF: #AttributeTypeAndValue.
	(module SEQUENCE: #AttributeTypeAndValue)
		addElement: #type type: #AttributeType;
		addElement: #value type: #AttributeValue.
	module OBJECT_IDENTIFIER: #AttributeType.
	module ANY: #AttributeValue.

Basically a Name is somewhat nested collection of attributes, where attribute has a type and a value. Now let's unmarshal an encoded Name using the PrettyPrinter policy.

 	bytes :=   16r3068310B3009060355040613025553311330110603550407130A43696E63696E6E61746931173015060
355040A130E43696E636F6D2053797374656D7331193017060355040B131043696E636F6D20536D616C6
C74616C6B3110300E0603550403130754657374204341 asBigEndianByteArray.
	marshaler := DERStream with: bytes.
	output := String new writeStream.
	marshaler encodingPolicy: (PrettyPrinter on: output).
	marshaler reset.
	name := marshaler unmarshalObjectType: module Name.
	output contents

If all goes well the result of the above code should look as follows:

0	Name
0		RDNSequence
2			RelativeDistinguishedName
4				AttributeTypeAndValue
6					AttributeType
11					ObjectIdentifier(2.5.4.6)
11					AttributeValue
15					'US'
15				AttributeTypeAndValue {type ObjectIdentifier(2.5.4.6), value 'US'}
15			OrderedCollection (AttributeTypeAndValue {type ObjectIdentifier(2.5.4.6), value 'US'})
15			RelativeDistinguishedName
17				AttributeTypeAndValue
19					AttributeType
24					ObjectIdentifier(2.5.4.7)
24					AttributeValue
36					'Cincinnati'
36				AttributeTypeAndValue {type ObjectIdentifier(2.5.4.7), value 'Cincinnati'}
36			OrderedCollection (AttributeTypeAndValue {type ObjectIdentifier(2.5.4.7), value 'Cincinnati'})
36			RelativeDistinguishedName
38				AttributeTypeAndValue
40					AttributeType
45					ObjectIdentifier(2.5.4.10)
45					AttributeValue
61					'Cincom Systems'
61				AttributeTypeAndValue {type ObjectIdentifier(2.5.4.10), value 'Cincom Systems'}
61			OrderedCollection (AttributeTypeAndValue {type ObjectIdentifier(2.5.4.10), value 'Cincom Systems'})
61			RelativeDistinguishedName
63				AttributeTypeAndValue
65					AttributeType
70					ObjectIdentifier(2.5.4.11)
70					AttributeValue
88					'Cincom Smalltalk'
88				AttributeTypeAndValue {type ObjectIdentifier(2.5.4.11), value 'Cincom Smalltalk'}
88			OrderedCollection (AttributeTypeAndValue {type ObjectIdentifier(2.5.4.11), value 'Cincom Smalltalk'})
88			RelativeDistinguishedName
90				AttributeTypeAndValue
92					AttributeType
97					ObjectIdentifier(2.5.4.3)
97					AttributeValue
106					'Test CA'
106				AttributeTypeAndValue {type ObjectIdentifier(2.5.4.3), value 'Test CA'}
106			OrderedCollection (AttributeTypeAndValue {type ObjectIdentifier(2.5.4.3), value 'Test CA'})
106		OrderedCollection (OrderedCollection (AttributeTypeAndValue {type ObjectIdentifier(2.5.4.6), value 'US'})
OrderedCollection (AttributeTypeAndValue {type ObjectIdentifier(2.5.4.7), value 'Cincinnati'}) OrderedCollection 
(AttributeTypeAndValue {type ObjectIdentifier(2.5.4.10), value 'Cincom Systems'}) OrderedCollection
(AttributeTypeAndValue {type ObjectIdentifier(2.5.4.11), value 'Cincom Smalltalk'}) OrderedCollection 
(AttributeTypeAndValue {type ObjectIdentifier(2.5.4.3), value 'Test CA'}))
106	Name<RDNSequence:OrderedCollection/>

The numbers at the beginning of each line show offsets into the source bytes. Each unmarshaled entity has two entries in the output, its type at the offset where it starts and the value printString at the offset where it ends. Simple types will have 2 entries next to each other and constructed types will have their elements indented.

There's much more to talk about in ASN.1, things like tagging, sub-typing, constraints, etc. But this post is again getting long, so I'll pick this up some other time. If the release is out before I get to it, you can find more about ASN.1 in the release notes and in the shiny new ASN.1 chapter of the SecurityGuide.pdf. Until then, thanks for reading this far and Happy Holidays!

Considerable amount of time in this release cycle was spent on improving our ASN.1 support. I don't want to get into details of this rather technical topic (maybe next time), but I'd like to show how much we've gained from it. As it happens ASN.1 is one of the principal building blocks of many security related standards, for example the PKCS suite of standards, CMS and S/MIME and obviously the ITU-T's X-series of recommendations where ASN.1 itself comes from (X.208). As you've probably guessed the X.509 certificate standard is part of the X-series as well but it is also published as RFC 3280. Anyway, enough with the barrage of acronyms, my point is, it's very useful to have it. So let's get back to X.509 certificates.

VisualWorks has supported X.509 certificates for a while, however until now it was the bare minimum necessary to conduct an SSL handshake. To be fair that's still quite a bit of functionality because you need to be able to decode pretty much any certificate, model all aspects of a certificate, validate all kinds of certificate properties and certificate chains etc. But it's still a far cry from full X.509 support. The most glaring limitation was inability to generate and encode certificates. With the new ASN.1 framework, we were able to rip out pretty much all the special purpose marshaling code from the X.509 framework and in exchange get not only the ability to decode but also to encode certificates. So here's how you can generate a self signed certificate with VW7.4.

An X.509 certificate binds a "name" of a subject to a "public key". The certificate itself is signed by the issuer of the certificate and the "name" of the issuer must also be present on the certificate. So to create the certificate we'll need the name of the subject, the name of the issuer and subject's public key. A self signed certificate is one where the issuer and the subject is the same entity, therefore the subject and issuer names are the same. Self-signed certificates are commonly used for certificate authority (CA) certificates, because they conveniently bundle the autority name with their public key. So let's define the name:

	caName := Security.X509.Name new

				add: 'C' -> 'US';

				add: 'L' -> 'Cincinnati';

				add: 'O' -> 'Cincom Systems';

				add: 'OU' -> 'Cincom Smalltalk';

				add: 'CN'-> 'Test Certificate Authority';

				yourself.

A name in X509 is actually a collection of so called AttributeValueAssertions where the keys represent well known aspects of a name. The various attributes are C=Country, L=Location, O=Organization, OU=Organization Unit, CN=Common Name, etc. Yes, the APIs could be nicer, but we're not there yet and I want to present code that actually works, so please, bear with me.

We'll also need keys. Let's say we want the CA to use RSA keys. Here's how to generate some.

	caKeys := Security.RSAKeyGenerator keySize: 1024.

	caKeys publicKey

Note that key generation is actually a fairly expensive process so depending on your hardware this may take a few seconds. I'm purposely not storing the public key, because the generator will cache both keys until it's flushed. At this point I just wanted to trigger key generation so that we're done with it.

Now we are ready to create the Certificate. There are few more required attributes like the serial number and validity dates, but those are self explanatory.

	ca := Security.X509.Certificate new

			serialNumber: 1000;

			issuer: caName;

			subject: caName;

			notBefore: Date today;

			notAfter: (Date today addDays: 7);

			publicKey: caKeys publicKey;

			forCertificateSigning;

			yourself.

The #forCertificateSigning bit is necessary for a CA certificate, but let's not worry about that now. We're almost done. The last missing bit on our shiny new certificate is the signature. The magic spell for that is #signUsing: which takes and instance of a signing algorithm preinitialized with a key as the argument. Of course the key has to be the private key, but you already knew that, right ?

	ca signUsing: (

		Security.RSA new

			useSHA;

			privateKey: caKeys privateKey;

			yourself).

And there it is. If you ask the certificate for its #printOpenSSLString, you should get something like this:

Certificate:

	Data:

		Version: 3 (0x2)

		Serial Number: 

			03:e8

		Signature Algorithm: sha-1WithRSAEncryption

		Issuer: C=US, L=Cincinnati, O=Cincom Systems, OU=Cincom Smalltalk, CN=Test Certificate Authority

		Validity

			Not Before: Dec 13 00:00:00 2005 GMT

			Not After : Dec 20 00:00:00 2005 GMT

		Subject: C=US, L=Cincinnati, O=Cincom Systems, OU=Cincom Smalltalk, CN=Test Certificate Authority

		Subject Public Key Info:

			Public Key Algorithm: rsaEncryption

			RSA Public Key: (1024 bits)

				Modulus (1024 bit):

					00:e5:4e:70:0d:65:7f:11:98:a3:2c:37:5a:0a:6d:

					ab:8f:28:92:fc:f9:db:f7:9c:1a:fa:01:a5:96:95:

					24:da:1c:ad:6b:18:65:cd:96:66:dd:e3:90:c8:2a:

					f6:62:ba:03:04:ec:ed:e0:db:f6:ab:65:93:84:4c:

					ef:94:12:a2:cb:14:b5:f2:15:c1:cf:37:9f:fb:e4:

					3a:ae:5e:3f:fb:9f:21:71:15:de:b8:20:c8:e8:8d:

					59:28:bf:ae:85:35:1a:9b:81:3f:b3:cc:d5:35:a1:

					da:3f:2e:dc:ca:cb:38:5e:33:a5:98:cf:7d:9f:2e:

					3e:99:ce:22:0f:21:26:24:37

				Exponent: 65537 (0x10001)

		X509v3 extensions:

			KeyUsage: critical

			X509v3 Basic Constraints: critical

			CA:TRUE

	Signature Algorithm: sha-1WithRSAEncryption

		97:dc:d6:dd:6f:d1:ce:08:d4:f8:d6:5f:bd:70:f1:ac:6a:7a:

		96:58:8c:b0:29:db:2b:82:43:6b:a3:f5:72:55:f5:c2:42:80:

		2c:4a:da:99:e0:be:e8:dd:55:df:45:69:8c:64:d8:5d:bf:78:

		42:0b:2a:89:19:3b:ae:b8:fa:db:b5:66:f3:1f:84:2a:e8:ab:

		09:5f:e6:48:03:25:60:98:a4:29:42:a1:6d:1e:69:82:b9:81:

		63:d1:3e:23:74:df:a2:cd:e3:c5:ca:a3:d6:da:1a:67:37:8b:

		50:cf:16:47:e6:17:ae:df:2b:a4:56:6e:06:58:58:c2:b4:24:

		ab:37

This gives you a complete certificate expressed as a Smalltalk object, but how can you share it with the rest of the world? That's where our new encoding capability gets involved.

	marshaler := ASN1.DERStream on: (ByteArray new: 100).

	marshaler marshalObject: ca withType: ca asn1Type.

	marshaler contents

The result of this is a byte array representing the DER encoding of the certificate.

To prove that I'm not making all this up, let's see how OpenSSL likes our new certificate. OpenSSL comes with this handy little utility called, surprisingly, openssl. You can usually find OpenSSL pre-installed on most Unix based platforms and if you're using Windows you should have had cygwin installed on it already. So give the following a try as well.

First we need to save the DER bytes into a file. The certificate actually caches its DER encoding so we don't need to invoke the above again, the following will suffice:

	'TestCA.der' asFilename writeStream

		binary;

		nextPutAll: ca encoding source;

		close

After this you should have a file TestCA.der in your image directory. To run it through OpenSSL, execute the following in your favourite shell:

	openssl x509 -inform DER -in TestCA.der -text

With any luck you should get the following in response.

Certificate:

    Data:

        Version: 3 (0x2)

        Serial Number: 1000 (0x3e8)

        Signature Algorithm: sha1WithRSAEncryption

        Issuer: C=US, L=Cincinnati, O=Cincom Systems, OU=Cincom Smalltalk, CN=Test Certificate Authority

        Validity

            Not Before: Dec 13 05:00:00 2005 GMT

            Not After : Dec 20 05:00:00 2005 GMT

        Subject: C=US, L=Cincinnati, O=Cincom Systems, OU=Cincom Smalltalk, CN=Test Certificate Authority

        Subject Public Key Info:

            Public Key Algorithm: rsaEncryption

            RSA Public Key: (1024 bit)

                Modulus (1024 bit):

                    00:e5:4e:70:0d:65:7f:11:98:a3:2c:37:5a:0a:6d:

                    ab:8f:28:92:fc:f9:db:f7:9c:1a:fa:01:a5:96:95:

                    24:da:1c:ad:6b:18:65:cd:96:66:dd:e3:90:c8:2a:

                    f6:62:ba:03:04:ec:ed:e0:db:f6:ab:65:93:84:4c:

                    ef:94:12:a2:cb:14:b5:f2:15:c1:cf:37:9f:fb:e4:

                    3a:ae:5e:3f:fb:9f:21:71:15:de:b8:20:c8:e8:8d:

                    59:28:bf:ae:85:35:1a:9b:81:3f:b3:cc:d5:35:a1:

                    da:3f:2e:dc:ca:cb:38:5e:33:a5:98:cf:7d:9f:2e:

                    3e:99:ce:22:0f:21:26:24:37

                Exponent: 65537 (0x10001)

        X509v3 extensions:

            X509v3 Key Usage: critical

                Digital Signature, Certificate Sign

            X509v3 Basic Constraints: critical

                CA:TRUE

    Signature Algorithm: sha1WithRSAEncryption

        97:dc:d6:dd:6f:d1:ce:08:d4:f8:d6:5f:bd:70:f1:ac:6a:7a:

        96:58:8c:b0:29:db:2b:82:43:6b:a3:f5:72:55:f5:c2:42:80:

        2c:4a:da:99:e0:be:e8:dd:55:df:45:69:8c:64:d8:5d:bf:78:

        42:0b:2a:89:19:3b:ae:b8:fa:db:b5:66:f3:1f:84:2a:e8:ab:

        09:5f:e6:48:03:25:60:98:a4:29:42:a1:6d:1e:69:82:b9:81:

        63:d1:3e:23:74:df:a2:cd:e3:c5:ca:a3:d6:da:1a:67:37:8b:

        50:cf:16:47:e6:17:ae:df:2b:a4:56:6e:06:58:58:c2:b4:24:

        ab:37

-----BEGIN CERTIFICATE-----

MIICjzCCAfigAwIBAgICA+gwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCVVMx

EzARBgNVBAcTCkNpbmNpbm5hdGkxFzAVBgNVBAoTDkNpbmNvbSBTeXN0ZW1zMRkw

FwYDVQQLExBDaW5jb20gU21hbGx0YWxrMSMwIQYDVQQDExpUZXN0IENlcnRpZmlj

YXRlIEF1dGhvcml0eTAeFw0wNTEyMTMwNTAwMDBaFw0wNTEyMjAwNTAwMDBaMHsx

CzAJBgNVBAYTAlVTMRMwEQYDVQQHEwpDaW5jaW5uYXRpMRcwFQYDVQQKEw5DaW5j

b20gU3lzdGVtczEZMBcGA1UECxMQQ2luY29tIFNtYWxsdGFsazEjMCEGA1UEAxMa

VGVzdCBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwgZ0wCwYJKoZIhvcNAQEBA4GNADCB

iQKBgQDlTnANZX8RmKMsN1oKbauPKJL8+dv3nBr6AaWWlSTaHK1rGGXNlmbd45DI

KvZiugME7O3g2/arZZOETO+UEqLLFLXyFcHPN5/75DquXj/7nyFxFd64IMjojVko

v66FNRqbgT+zzNU1odo/LtzKyzheM6WYz32fLj6ZziIPISYkNwIDAQABoyQwIjAP

BgNVHQ8BAf8EBQMDB4QAMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEFBQAD

gYEAl9zW3W/RzgjU+NZfvXDxrGp6lliMsCnbK4JDa6P1clX1wkKALErameC+6N1V

30VpjGTYXb94QgsqiRk7rrj627Vm8x+EKuirCV/mSAMlYJikKUKhbR5pgrmBY9E+

I3Tfos3jxcqj1toaZzeLUM8WR+YXrt8rpFZuBlhYwrQkqzc=

-----END CERTIFICATE-----

So, this is what it takes to generate a certificate in VW7.4. The APIs definitely need more work, but it's fairly useable already. I think that even with a fairly minimalistic UI slapped on top of this it should be possible to run a private PKI hierarchy.

Actually, I guess I should show how to "issue" a certificate. It is nothing more than generating a new certificate for a given subject with subject's public key and signed using issuer's private key. So we need subject name and keys.

	subjectName := Security.X509.Name new add: 'CN' -> 'Test Subject'; yourself.

	subjectKeys := Security.DSAKeyGenerator keySize: 1024.

	subjectKeys publicKey

The certificate is created as previously, note however that it is important to express the usage of the associated key properly. So let's say the keys for this certificates can be used for signing data and not other certificates (you can find more examples in the 'accessing - key usage' protocol on Certificate).

	subject := Security.X509.Certificate new

			serialNumber: 1000;

			issuer: caName;

			subject: subjectName;

			notBefore: Date today;

			notAfter: (Date today addDays: 7);

			publicKey: subjectKeys publicKey;

			forSigning;

			yourself.

And finally the signature.

	subject signUsing: (

		Security.RSA new

			useSHA;

			privateKey: caKeys privateKey;

			yourself).

We'll leave encoding of this new certificate as an exercise for the attentive reader. Thanks for reading this far, I hope you enjoyed the article.

Knock, knock, knock ... is this thing on ? Let's hope.

VW 7.4 is going through its final stages, so I thought this might be a good time to brag about what's coming. Today I'd like to mention one of my pet projects. Well, it's "my" only in the sense of me being fond of it, but the work is actually done by Dave Wallen. He wrote a DLL/CC wrapper for the crypto library used by OpenSSL. It's not finished and it's shipping as a preview with 7.4, but already provides access to ARC4, Blowfish, DES and AES, including all the cipher modes and padding. The main requirements were to have the wrapper behave exactly as the native Smalltalk equivalents, support the same API and preserve the same runtime characteristics like making sure it generates as little garbage in the process as possible. And Dave's done a pretty good job with that. So let's take a look at this puppy.

The API compatibility is great. If you load the OpenSSL parcel you can you can pretty much follow the existing examples from SecurityGuide as they are. The only change is to substitute classes from Security.OpenSSL for the ones in Security namespace. For example

	plaintext := 'This is the end ...' asByteArray.

	key := #[1 2 3 4 5 6 7 8].

	iv := #[8 7 6 5 4 3 2 1].

	alice := Security.OpenSSL.DES newBP_CBC.

	alice setKey: key;

 		setIV: iv copy.

	ciphertext := alice encrypt: plaintext.

 

and to decrypt

	bob := Security.OpenSSL.DES newBP_CBC.

	bob setKey: key;

		setIV: iv copy.

	(bob decrypt: ciphertext) asString

 

Note that it's not just the constructors that are polymorphically compatible, but even the wrapper structures work the same way (where possible).

	(Security.OpenSSL.OutputFeedback on: Security.OpenSSL.Blowfish new)

		setKey: #[1 2 3 4];

		setIV: #[8 7 6 5 4 3 2 1] copy;

		encrypt: 'hello' asByteArray

 

In many cases a switch from one library to the other should be as easy as importing the right namespace into your application code. However there are and will be some incompatibilities. For example tripple DES is implemented as just another cipher in OpenSSL and our generic tripple encryption wrapper approach just doesn't map well to that. We could implement a wrapper that would make you create three instances of DES and then just throw 2 away, but that seems a bit too extreme. Moreover OpenSSL only supports tripple encryption with DES, so a wrapper would just be confusing in terms of where it can be used anyway. So for tripple DES there's only the constructor based API, as in

	Security.OpenSSL.DES newBP_3EDE_CBC

		setKey: '0123456789abcdefghijklmn' asByteArray;

		setIV: #[8 7 6 5 4 3 2 1];

		encrypt: 'Hello World!' asByteArray

 

As I hinted earlier, there's also capability mismatch between the two libraries. For example we can do tripple encryption with any of the supported block ciphers, not just DES. On the other hand OpenSSL can do ciphers that we don't yet and possibly never will (e.g. CAST, IDEA), etc. But for the majority of features that matter the most there should be an overlap.

OK, that's all nice, but you're probably asking by now, why did we do this ? There's actually a number of reasons that in total seemed compelling enough.

First of all it gives our users a choice. Having native Smallltalk implementations is incredibly convenient, you don't have to worry if and where you have an external library installed or if it's even easily available for the platform that you want to run on. However even though the perfomance of the native ciphers is more than adequate for many applications it will be a stretch in others. Symmetric ciphers are actually the worst case in terms of Smalltalk's lag behind a C implementation. Symmetric ciphers are generally taylored for heavily optimized "close to the metal" implementations, with lots of bit shuffling and mixing in a fixed set of registers. Current Smalltalks have little chance to compete with hand crafted assembler there. To give you an idea here's how they compare on this machine, a Pentium4 M/1.7 GHz. The openssl utility clocks its AES perfomance as follows (cygwin installed on WinXP):

$ openssl speed aes-128-cbc

Doing aes-128 cbc for 3s on 16 size blocks: 7744385 aes-128 cbc's in 3.00s

Doing aes-128 cbc for 3s on 64 size blocks: 1939658 aes-128 cbc's in 2.99s

Doing aes-128 cbc for 3s on 256 size blocks: 491623 aes-128 cbc's in 3.00s

Doing aes-128 cbc for 3s on 1024 size blocks: 122641 aes-128 cbc's in 3.00s

Doing aes-128 cbc for 3s on 8192 size blocks: 15343 aes-128 cbc's in 3.00s

OpenSSL 0.9.7e 25 Oct 2004

built on: Sat Dec 11 12:01:27 WEST 2004

options:bn(64,32) md2(int) rc4(idx,int) des(ptr,risc1,16,long) aes(partial) blowfish(idx)

compiler: gcc -DOPENSSL_SYSNAME_CYGWIN32 -DOPENSSL_THREADS -DDSO_WIN32 -DOPENSS L_NO_KRB5 -DOPENSSL_NO_IDEA -DOPENSSL_NO_RC5 -DOPENSSL_NO_MDC2 -DTERMIOS -DL_ENDIAN -fomit-frame-pointer -O3 -march=i486 -Wall -DSHA1_ASM -DMD5_ASM -DRMD160_ASM



available timing options: TIMES TIMEB HZ=1000 [sysconf value]

timing function used: times

The 'numbers' are in 1000s of bytes per second processed.

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes

aes-128 cbc      41248.39k    41462.30k    41895.97k    41791.81k    41840.83k

 

So OpenSSL can generally push through 40 GB/sec in this mode. Now let's take a look at native Smalltalk implementation:

	| aes gigs |

	gigs := 5. "We'll encrypt 5GB of data"

	aes := Security.AES newCBC

		setKey: #[0 1 2 3 4 5 6 7 8 9 10  11 12 13 14 15];

		setIV: #[8 7 6 5 4 3 2 1 1 2 3 4 5 6 7 8] copy;

		yourself.

	#(16 64 256 1024 8192) do: [ :size || reps ms b |

		b := ByteArray new: size.

		reps := gigs*(10**6) // size. 

		ms := Time millisecondsToRun: [ reps timesRepeat: [ aes encryptInPlace: b ] ].

		Transcript cr; print: size; show: ' bytes, '; print: reps; show: ' reps = ';

			print: ms; show: ' ms => '; print: gigs * 1000.0 / ms; show: ' GB/s'  ].



	16 bytes, 312500 reps = 5979 ms => 0.83626 GB/s

	64 bytes, 78125 reps = 6016 ms => 0.831117 GB/s

	256 bytes, 19531 reps = 6021 ms => 0.830427 GB/s

	1024 bytes, 4882 reps = 6022 ms => 0.830289 GB/s

	8192 bytes, 610 reps = 6051 ms => 0.82631 GB/s

 

It is consistently a bit less than 1 GB/s. I hate to post such a comparatively unimpressive number for my beloved platform, but as I said there are pretty good reasons. One, symmetric ciphers and hashes are designed to fit common hardware. Two, Smalltalk pays a very high price for indexed access into byte arrays, performing bounds checks on both read and write for each byte. So these are really the worst cases. For example in my ad hoc tests RSA is only about 3-4 times slower than OpenSSL's speed test was showing. But we'll be able to do better comparison once the wrapper supports those as well. So with all that taken into account I'd say the results aren't all that bad. Certainly within practical limits of many applications.

Anyway, with that off my chest, here are the results of the same code as above, just using Security.OpenSSL.AES.

	16 bytes, 312500 reps = 4988 ms => 1.00241 GB/s

	64 bytes, 78125 reps = 1325 ms => 3.77358 GB/s

	256 bytes, 19531 reps = 404 ms => 12.3762 GB/s

	1024 bytes, 4882 reps = 172 ms => 29.0698 GB/s

	8192 bytes, 610 reps = 109 ms => 45.8716 GB/s

 

It shows nicely that the cost of DLL calls has a significant impact on performance and that for small chunks of data and consequently high frequency of DLL calls it comes quite close to the native implementation. An interesting blip is the largest block size which reports higher throughput than the results of the openssl command. I'm not about to dive into OpenSSL source code to find out why, but my guess is that we don't really know what exactly is being measured by the openssl command. So there's probably some additional activity included in those numbers.

Anyway, I think the most interesting outcome of these benchmarks is that for small blocks (let's say up to 64 bytes) the native implementation doesn't do much worse than calling OpenSSL, beyond that it's something to consider (if the performance of encryption is a concern in your application). But as soon as you get up to 8K size blocks you're pretty much matching the speed of pure C code.

This might be quite interesting if you intend to use VW as an SSL server, for example. Symmetric ciphers (and hashes) are the work horse of SSL connections. The results suggest that with the OpenSSL wrapper you might be able to handle up to 40-times as many concurrent connections. That's of course completely ignoring the cost of any other processing, so in practice the ratio should be much smaller, but still significant.

Speaking of that, here's how you can plug the wrapper into the SSL framework. Basically all it takes is initializing selected SSL "cipher suites" with the right algorithms. Unfortunatelly the currently shipping version of SSL doesn't provide mutators for SSLCipherSuites so we have to do the following for now:

	suite := SSLCipherSuite SSL_RSA_WITH_DES_CBC_SHA.

	suite cipherSpec

		instVarAt: 1	"the first variable, bulkCipher";

		put: Security.OpenSSL.DES newCBC.

	HttpClient new

		sslContext: (SSLContext suites: (Array with: suite));

		get: 'https://www.microsoft.com'

 

Of course, the right way to do this would be extending the SSLCipherSuite and SSLCipherSpec classes with new suite definitions (following the example of the existing ones).

Something to note about this example is that the suite as defined will use OpenSSL for DES-CBC encryption and native Smalltalk for SHA and RSA. So you can mix and match algorithm implementations arbitrarily. This is actually quite useful if your application must make use of specific hardware for some cryptographic operations, like having a smart card perform all RSA computations for security reasons. All you need is to create another version of the RSA algorithm that simply calls out to the card and hook it up through an SSLCipherSuite definition. Actually, I have to admit that it's not yet readilly available with the current implementation, but all it takes is a straightforward extension of the SSLRSAKeyExchange class that is held by the SSLCipherSuite instance. This should all be polished off by the time our OpenSSL wrapper supports public key algorithms as well.

And this brings me to another reason to implement the OpenSSL wrapper, it's an example how to create such a wrapper for other libraries that might be mandated for cryptographic use by your organization. Similarly to the smart card example, various security policies may require your application to exclude any cryptographic operations from your Smalltalk image. Or you may need to use hardware based cryptographic accellerator to speed up these operations on a heavilly loaded SSL server. The wrapper allows us to validate sanity of our APIs and prove that interfacing these external facilities should be possible in a reasonably straightforward manner. It also provides a good starting point for development of your own wrapper.

Finally, it's also a very useful tool for us. So far we were able to test interoperability of our cryptographic library only indirectly, by running SSL communications to external servers. Any fault in our algorithm implementation would very likely manifest itself there. However because of the nature of the beast, it was quite difficult to figure out what exactly is wrong when SSL communication failed. This wrapper will make this process much more efficient, robust and pleasant to work with. So, thanks, Dave! You've just made my life sooo much easier.