Glacier2, the Ice firewall traversal service, has two modes by which it forwards requests and replies between clients and servers—buffered and unbuffered. In unbuffered mode, the thread that receives a request or reply forwards it immediately. In buffered mode, requests and replies are queued and delivered by a separate thread. With Ice 3.3, buffered mode is required only if an application needs request batching and request overriding. In contrast, with all prior versions of Glacier2, buffered mode was essential.

Before Ice 3.3, the purpose of Glacier2’s buffered mode was primarily to isolate clients from each other. Because any request could block (asynchronous or otherwise), in buffered mode, Glacier2 spawned two threads per connected client. One thread was used to forward requests from the connected client to the back-end, and the other was used to forward replies from the back-end to the client. In unbuffered mode, any misbehaved application (whether malicious or just faulty) could block an invocation, causing denial of service to all other connected clients.

Because buffered mode was the only way to provide isolation, it was used by virtually all front-facing applications. However, real-world use showed that once several hundred clients were connected, Glacier2 suffered scalability issues from context switching between threads as well as the limited virtual memory space in 32-bit operating systems.

With the introduction of true non-blocking asynchronous invocations in Ice 3.3, we re-architected Glacier2 to take advantage of this feature. Given that this eliminates all the buffering threads, Glacier2 should theoretically realize a big improvement in scalability. I set about testing this during the Ice 3.3 development process.

To test scalability, I wanted to establish many Glacier2 sessions. I modified the subscriber from the demo/IceStorm/clock example to establish multiple sessions with the Glacier2 router, and I modified the publisher to publish a single event per second. (The publisher did not use Glacier2 but connected directly to IceStorm.) The published events were text strings that contained the current time.

Next I set up the following environment:

• Glacier2 hosted on a single-core 3.2GHz CentOS 4 machine with 2GB of memory.
• IceStorm hosted on a CentOS 4 virtual machine, with the VM using one core of a Q6600 machine with 4GB of memory.
• Publisher and subscriber hosted on a MacBook.
• All machines were connected via a gigabit network.
• All machines used Ice 3.3, compiled with optimization.

After starting Glacier2 and IceStorm, I ran the subscriber application and waited for all subscriptions to complete. Initially, I configured the subscriber to establish 1,000 sessions with the Glacier2 router. After starting the publisher, I could see the events arrive at the subscriber. With 1,000 events per second flowing through the Glacier2 router, the system had no problems keeping up.

I gradually pumped up the volume by adding more and more subscribers until, eventually, I topped out at 8,000 subscribers, which corresponds to 8,000 events per second flowing through Glacier2. At that point, the Glacier2 host reached 100% CPU usage. Adding more subscribers confirmed that Glacier2 could no longer keep up with the flood of events; messages were gradually accumulating in the router, and subscribers were no longer receiving timely updates. On the other hand, with 8,000 subscribers, the IceStorm host used only 5% of the CPU, and the machine running the subscribers also showed very low CPU utilization.

Considering that the Glacier2 host (3.2GHz single core) is a slow and quite underpowered machine by modern-day standards, I think my testing shows that Glacier2’s current scalability is very good indeed. This is especially true given that Glacier2 is really only used for WAN-side applications. Although 8,000 events per second may not sound all that fast, consider the picture from a bandwidth point of view: even if these events are very modestly sized at 1KB each, a T3 connection (at 45 MB/s or 5760 KB/s) would be fully saturated with 5,760 events per second. In the end, I think you’ll find that Glacier2 will not be the bottleneck in your project.