See http://www.zeroc.com/doc/screencasts.html for the screencast itself. I hope you enjoy it, and look for more in the future.

… or how long is a piece of string?

From an IceStorm perspective, there are two types of clients:

• Publishers: Clients that publish messages to topics.
• Subscribers: Servers that receive messages from a topic.

The exact number of supported publishers and subscribers depends on the total load imposed by them. Load is composed of several factors.

The first is message throughput, expressed as events per second (EPS). Message throughput is a function of latency and the total number of raw messages sent. By way of example:

• You have 1 publisher publishing 1 EPS to 1 subscriber: throughput is 1 EPS.
• You have 1 publisher publishing 10 EPS to 1 subscriber: throughput is 10 EPS.
• You have 1 publisher publishing 10 EPS to 10 subscribers: throughput is 100 EPS.
• You have 10 publishers publishing 10 EPS to 10 subscribers: throughput is 1000 EPS.

The second factor that influences load is message size. By multiplying EPS with message size, you can roughly determine the amount of bandwidth IceStorm will consume.

Bigger events also mean that IceStorm’s memory consumption will increase. However, memory use is not a primarily a function of the number of subscribers, but a function of the number of publishers and the number of events that are queued for subscribers. A single event published to one subscriber will only consume marginally less memory than a single event that is published to one hundred subscribers. This is because the memory allocated to an event is shared among all subscribers. Each subscriber has a separate event queue that points at the event, so queuing an event for an additional subscriber only consumes memory to store a pointer to the event in the queue, not memory to store an extra copy of the event.

IceStorm provides four quality-of-service modes for event delivery: oneway, twoway, twoway-ordered, and batched. Of these, oneway is fastest, followed by twoway, and then twoway-ordered. In particular, oneway events (if you can live with the limitations imposed by them) provide a big increase in EPS. For high-latency networks, you can also set batch delivery, which increases overall throughput by combining several events into a single physical protocol message. The cost is that events arrive at subscribers in bursts, and the bursts are separated by longer gaps than for unbatched delivery. For low-latency networks, batching does not provide any benefits. (See Ice Manual for more information on batch delivery.)

The best way to determine IceStorm’s maximum raw throughput is to run a latency test for the delivery mode you intend to use. For example, if you intend to use twoway delivery, run the C++ version of the twoway latency demo on your machine (see cpp/demo/Ice/latency). This demo measures the latency of sending twoway messages on a single machine. On one of my dual-core machines, the latency demo measures 0.149ms per message, which works out to 6,716 twoway messages per second. If you have more than two cores, you will achieve higher throughput (but throughput will not increase linearly with the number of cores).

To get a feel for the maximum bandwidth, you can run the C++ throughput demo (cpp/demo/Ice/throughput). Of course, this demo does not take into account the physical bandwidth limitations imposed by your network because client and server communicate over the loopback interface. However, you can also run the throughput client and server on separate machines to see how your network affects bandwidth.

To get optimum throughput from IceStorm for your application, you will need to play with the IceStorm configuration settings. The main setting you will be concerned with is the number of threads in the publishing thread pool. This setting determines how many events can concurrently be received and published. (Note that, if you are publishing messages to IceStorm with a oneway proxy and also require strict event ordering, you cannot configure more than one thread in the IceStorm thread pool.)

A good starting point for the thread pool size is to set the size equivalent to the number of cores on your IceStorm host. For example, assuming the name of the service is IceStorm, and IceStorm runs on a four-core machine, define the following:

IceStorm.Publish.ThreadPool.Size=4

As you can see, the question of how many clients IceStorm can support cannot be answered in isolation. To obtain a realistic estimate, you have to understand the application structure, the load imposed on IceStorm, the network and bandwidth restrictions, as well as the hardware on which IceStorm executes. Running benchmarks in an environment that resembles the deployment configuration as closely as possible is the best way to determine the total possible load that can be supported. The general guidelines I’ve outlined above should point you in the right direction.

Before Ice 3.3, the purpose of Glacier2’s buffered mode was primarily to isolate clients from each other. Because any request could block (asynchronous or otherwise), in buffered mode, Glacier2 spawned two threads per connected client. One thread was used to forward requests from the connected client to the back-end, and the other was used to forward replies from the back-end to the client. In unbuffered mode, any misbehaved application (whether malicious or just faulty) could block an invocation, causing denial of service to all other connected clients.

Because buffered mode was the only way to provide isolation, it was used by virtually all front-facing applications. However, real-world use showed that once several hundred clients were connected, Glacier2 suffered scalability issues from context switching between threads as well as the limited virtual memory space in 32-bit operating systems.

With the introduction of true non-blocking asynchronous invocations in Ice 3.3, we re-architected Glacier2 to take advantage of this feature. Given that this eliminates all the buffering threads, Glacier2 should theoretically realize a big improvement in scalability. I set about testing this during the Ice 3.3 development process.

To test scalability, I wanted to establish many Glacier2 sessions. I modified the subscriber from the demo/IceStorm/clock example to establish multiple sessions with the Glacier2 router, and I modified the publisher to publish a single event per second. (The publisher did not use Glacier2 but connected directly to IceStorm.) The published events were text strings that contained the current time.

Next I set up the following environment:

• Glacier2 hosted on a single-core 3.2GHz CentOS 4 machine with 2GB of memory.
• IceStorm hosted on a CentOS 4 virtual machine, with the VM using one core of a Q6600 machine with 4GB of memory.
• Publisher and subscriber hosted on a MacBook.
• All machines were connected via a gigabit network.
• All machines used Ice 3.3, compiled with optimization.

After starting Glacier2 and IceStorm, I ran the subscriber application and waited for all subscriptions to complete. Initially, I configured the subscriber to establish 1,000 sessions with the Glacier2 router. After starting the publisher, I could see the events arrive at the subscriber. With 1,000 events per second flowing through the Glacier2 router, the system had no problems keeping up.

I gradually pumped up the volume by adding more and more subscribers until, eventually, I topped out at 8,000 subscribers, which corresponds to 8,000 events per second flowing through Glacier2. At that point, the Glacier2 host reached 100% CPU usage. Adding more subscribers confirmed that Glacier2 could no longer keep up with the flood of events; messages were gradually accumulating in the router, and subscribers were no longer receiving timely updates. On the other hand, with 8,000 subscribers, the IceStorm host used only 5% of the CPU, and the machine running the subscribers also showed very low CPU utilization.

Considering that the Glacier2 host (3.2GHz single core) is a slow and quite underpowered machine by modern-day standards, I think my testing shows that Glacier2’s current scalability is very good indeed. This is especially true given that Glacier2 is really only used for WAN-side applications. Although 8,000 events per second may not sound all that fast, consider the picture from a bandwidth point of view: even if these events are very modestly sized at 1KB each, a T3 connection (at 45 MB/s or 5760 KB/s) would be fully saturated with 5,760 events per second. In the end, I think you’ll find that Glacier2 will not be the bottleneck in your project.

In my first article, which appeared in Issue 12, I introduced a call queue to avoid blocking the GUI thread. The idea was to write a class for each remote operation that the application needed to invoke; adding an instance of this class to the queue scheduled an invocation by a separate thread. The following code demonstrates the API:

class Call : public Shared
{
public:
    virtual void execute() = 0;
};
typedef Handle<Call> CallPtr;

class SayHelloCall : public Call
{
public:
    SayHelloCall(const HelloPrx& hello) :
        _hello(hello)
    {
    }

    void execute()
    {
        _hello->sayHello();
    }

private:
    const HelloPrx _hello;
};

CallQueuePtr queue = ...;
HelloPrx hello = ...;
queue->call(new SayHelloCall(hello));

The call queue implementation was relatively simple: a dedicated thread executed the queued calls in order, essentially serializing all of the invocations. A side-effect of this implementation is that it guaranteed strict ordering of twoway requests in the same server.

I enhanced the call queue in Issue 13 to support multiple concurrent requests using two different implementations: one using asynchronous invocations and the other using a thread pool. Both strategies sacrificed the strict ordering guarantee in order to improve throughput. I also introduced the idea of using multiple call queues for messages with different qualities of service (such as oneway, twoway, and batched messages).

I presented an advanced technique in Issue 14 that used the Ice router facility to eliminate the need to define a Call class for each remote operation. With this technique, invocations on Ice objects from the GUI are accomplished using the standard asynchronous API. The resulting code is considerably more straightforward:

class AMI_Hello_sayHelloI : public AMI_Hello_sayHello
{
public:
    void ice_response()
    {
    }

    void ice_exception(const Ice::LocalException&)
    {
    }
};

HelloPrx twoway = // Some twoway proxy.
twoway->sayHello_async(new AMI_Hello_sayHelloI());

The disadvantage of this technique was that, along with the complexity of having to implement an Ice router, oneway invocations were not handled gracefully. Instead, they required the use of a special request context key, similar to the one used by the Glacier2 router.

We were not very happy with this situation, and we went about fixing it in the Ice 3.3 release. To ensure that asynchronous calls never block, Ice 3.3 introduced some major changes to the internal architecture, which Benoit and Mark detailed in their article “Background I/O” in Issue 28 of Connections. The Ice run time now maintains its own queue of outstanding calls. Unlike my call queue implementations, Ice’s queue is hidden from the application and much more efficient because calls only need be queued if they cannot be sent immediately.

Contrary to what you might expect, however, synchronous oneway invocations such as the one shown below may still block:

HelloPrx twoway = // Some twoway proxy.
HelloPrx oneway = HelloPrx::uncheckedCast(twoway->ice_oneway());
oneway->sayHello();

The call to sayHello can block the calling thread during connection establishment, while performing DNS lookups, or because the server is slow or non-responsive.

Why did we decide to preserve the existing semantics of synchronous oneways? It certainly would be possible to make oneway invocations non-blocking, but that would introduce a big problem: how would we report errors back to the application? What should Ice do if, for example, a DNS lookup failed? Since the call to sayHello would not be allowed to block, the DNS lookup would have to occur in a separate thread, so the call to sayHello could return. But once the call to sayHello has returned, there is no obvious way to report the error to the application.

Ice 3.3 solved this problem in a different way by adding support for asynchronous oneways. These work much like asynchronous twoways in that you have to define an AMI callback object. The primary difference is that ice_response is never called because oneway invocations do not have responses. If an error occurs while Ice attempts to send the message, the Ice run time invokes ice_exception on the callback as usual. The code below shows how to make an asynchronous oneway request:

class AMI_Hello_sayHelloI : public AMI_Hello_sayHello
{
public:
    void ice_response()
    {
        assert false;
    }

    void ice_exception(const Ice::LocalException&)
    {
    }
};

HelloPrx twoway = // Some twoway proxy.
HelloPrx oneway = HelloPrx::uncheckedCast(twoway->ice_oneway());
oneway->sayHello_async(new AMI_Hello_sayHelloI());

One of the first applications to take advantage of the new non-blocking oneway semantics was IceStorm. Much effort went into prior versions of IceStorm to ensure that invocations from publishers would never block, a non-trivial task given that there was no simple way to make non-blocking invocations. With the new capabilities in Ice 3.3, I immediately re-architected and simplified the IceStorm internals to take advantage of the new non-blocking semantics. However, once I started performance and stress testing, I noticed that there was a new issue: there was no flow control on the non-blocking invocations. The upshot of this was that flooding IceStorm with events caused the memory usage to climb very quickly. To solve this problem we added a callback that Ice invokes after a queued request is actually sent. This can be used to flow-control oneway events.

To receive this notification, an AMI callback object must implement the Ice::AMISentCallback interface. Calling the asynchronous oneway method returns false if the message was queued, or true if the message was sent immediately. When the queued message is eventually sent, the Ice run time invokes the AMISentCallback::ice_sent method on the callback object:

class AMI_Hello_sayHelloI : public AMI_Hello_sayHello, public Ice::AMISentCallback
{
public:
    void ice_sent()
    {
        /* called when the message is sent*/
    }

    // ice_response, ice_exception implementation here
};

HelloPrx twoway = // Some twoway proxy.
HelloPrx oneway = HelloPrx::uncheckedCast(twoway->ice_oneway());

if(!oneway->sayHello_async(new AMI_Hello_sayHelloI()))
{
    // The request was queued, and ice_sent will be called
    // once the message is sent.
}

The addition of the guaranteed non-blocking semantics renders much of the content of my first three articles obsolete. Developing GUI applications with Ice is now much more straightforward, and I can offer much simpler advice: all remote invocations made from the GUI should be asynchronous!

Earlier I mentioned the issue of strict ordering guarantees for twoway invocations. The call queue implementation in Issue 12 preserved the order of requests, whereas the subsequent strategies did not (and neither does the non-blocking asynchronous solution in Ice 3.3). If your application depends on the order in which requests are dispatched in the server, you must take additional measures such as disabling portions of the UI while a call is in progress.

Finally, since AMI callbacks are invoked by an Ice thread, it may not be safe for your callbacks to manipulate the GUI directly. This is the topic of my article in Issue 15, which is still as relevant as ever.

Does that mean things are always identical? No. There are necessary differences imposed by the target language. For example, for a Slice interface Foo, in C++ you use FooPrx::checkedCast, and in Java and C# you have to use FooPrxHelper.checkedCast. There are also cases where we take advantage of a language-specific feature, such as C# delegates in the new AMI Silverlight mapping. These differences make a mapping more convenient to use, but less regular.

Our general rule is to strive to keep mappings as similar as possible without sacrificing convenience if a language feature would produce a more natural mapping.

Regularity aids in learning and comprehension. As modern-day programmers, we are expected to be multi-lingual. Anything Ice can do to aid comprehension is much appreciated! Unfortunately, the same cannot be said for the Google Protocol Buffers mappings.

The first thing you will notice when using PB is that the generated type name is different in each target language. This is not the case with any of the Slice language mappings, where the target language type name, by default, always matches the Slice type name.

// Slice
module Demo
{
    class Hello
    {
        int id;
    };
}

The generated code for this example always produces a set of types in the Demo namespace. For example, in C++, the types Demo::Hello and Demo::HelloPrx are emitted. With PB, the situation is different.

// Person.proto
package tutorial;
option java_outer_classname = "PersonPB";
message Person
{
    required int32 id = 1;
}

As with the Slice language mappings, the C++ type generated by protoc is tutorial::Person. However, for Java, the type is tutorial.PersonPB.Person, and for Python, it is Person_pb2.Person.

Unlike the Slice-to-Java mapping, where the translator creates a package containing separate classes, the PB-to-Java mapping produces a single file that contains all definitions for a proto file. I don’t think a single file containing nested classes was a very good choice. For the application programmer, this causes rather lengthy and obfuscated class names. It also forces the addition of the option java_outer_classname construct. Why? By default the class file that is emitted by the protoc compiler has the same name as the proto file. In the example above, a single file tutorial/Person.Java would be generated. However, this file wouldn’t compile since Java does not allow a nested class to have the same name as an outer class. Therefore, by necessity the PB developers added the java_outer_classname option to force the protoc compiler to produce a different file name.

The Slice-to-Python mapping, again, produces a package containing all of the necessary classes. The PB-to-Python mapping produces a single file. For example, Person.proto emits a single file Person_pb2.py. The package specified by the protocol definition has no effect on the generated package name. Personally, I find this very unexpected, and not at all desirable. For example, consider a change to the name of the protocol file. Now, at the very least, all of the import statements in your code must change. Far from ideal!

With the Slice language mappings, identifiers and methods contained within an interface, class or struct are always the same. For the example above, a class Hello is emitted containing the member variable id. If you know one Slice mapping, you know them all.

For the PB language mappings, the identifiers emitted by the various mappings are, surprisingly, different for each language. In C++, the variable id maps to the member functions Person::has_id, Person::get_id, and so on. In Java, the methods are named Person.hasId and Person.getId, and in Python the mapping uses Person.id and Person.HasField(”id”). Each of these is different, and worse, for no obvious reason. Once again, knowledge of one mapping does not impart knowledge of any other.

When it comes to creating instances of the various classes, the situation is, again, different in each language. For C++ and Python, you create an instance of the class and populate the members, but Java has a totally different mechanism. The Java mapping uses a factory pattern (called a Builder), where you construct a Builder and then use it to create an immutable message instance. For example:

tutorial.PersonPB.Person p = tutorial.PersonPB.Person.newBuilder().
    setId(1).build();

I’m not saying that the builders are bad—they are not. Because of builders, it is impossible to create an un-initialized message instance, which is a good thing. However, why the different paradigms? What is good in Java would also be good in C++ and Python.

As you can see, the Slice language mappings are regular, and as close to each other as possible. This wasn’t an accident. We are all very aware, as daily users of our own product, that this is important. It is a pity that the PB mappings don’t take the same approach.

The rapid demise of XML as a data store for large amounts of non-human readable structured data, and for RPC (such as SOAP), comes as no surprise to me. XML, although touted as such, is clearly not human readable. On top of that it is both bandwidth and storage intensive, and requires huge amounts of CPU cycles to process. Even in this day and age of fast CPUs, terabytes of storage and gigabit networks, every byte and cycle still counts, especially for huge companies like Google with vast amounts of data. Bandwidth is not free, and neither are cycles (someone has to pay that power bill!).

Now we’ve reached a new milestone. Web services are quickly dying a well-deserved death. WSDL, despite arguments to the contrary, is nothing new (Michi has written about this in the past, see “To Slice or not to Slice” for details). It is nothing more than an exceptionally convoluted, and unreadable, form of an interface definition language. SOAP, WSDL’s partner in crime, is nothing but hype. Finally, the toilet of history has been flushed, and we can watch the last vestiges of the colossal mistake that is web services swirl down the drain.

The lessons of the past are finally being relearned. Slow, fat and bloated is out. Sleek, small and fast is in. Binary encodings and protocols are undergoing a renaissance. All of this is good news for ZeroC and Ice. We understand speed. We live for simplicity.

So where does Google protocol buffers fit in? Other than agreeing with my general world view that binary encodings for non-human readable data are a good thing, Google protocol buffers are only part of a puzzle for many applications. What is missing is a facility to pass a protocol buffer object over the wire. It didn’t take long for some developers to discuss starting such a project. It also didn’t take long for Blair Zajac to point out here and here that Ice would be a good companion.

I shouldn’t have to go on about why using Ice for an RPC mechanism is a good thing. I could trot out all the standard arguments such as speed, flexibility, security, support for both synchronous and asynchronous operations, firewall traversal, quality documentation, and so on and so forth. But I won’t

Using Ice, it is already pretty easy to pass any data that can be encoded to a sequence of bytes, such as the Google protocol buffers.

Lets take a quick look at what this would look like in python.

// .proto
package tutorial;
message Person {
required int32 id = 1;
required string name = 2;
optional string email = 3;
}
// Slice
module Demo
{
sequence<byte> Person;
interface Hello
{
void sayHello(Person p);
};
};


In the client:


# python
hello = Demo.HelloPrx….
p = Person_pb2.Person()
# Fill in details
hello.sayHello(p.SerializeToString());


In the server:

# python
class HelloI(Demo.Hello):
def sayHello(self, s, current=None):
p = Person_pb2.Person()
p.ParseFromString(s)
print "Hello World from %s" % str(p)


Not very complicated, but can we do better? What would be really cool, is if you could pass a Person object as a method parameter, and it would magically appear in the server as a Person object. In other words, automate all that busy work code. Now that would be great!

That is exactly what I’ve spent the past few days doing, and the result is our latest ZeroC Labs project, which you can read about here. As always, source code is readily available.

We’re releasing this labs project as an experiment, to gauge interest. If the community finds it useful, we’ll integrate this more fully into a future release of Ice. It may even be possible to not only support the Google protocol buffers encoding, but also other encodings such as C# and Java serialized types.

Please look over what this integration has to offer, and give us any feedback you might have!

Have fun with Ice, Matthew