Gerald Kanapathy's Blog

Gerald Kanapathy's Homepage
Gerald Kanapathy is Director of Consulting in BEA's Business Interaction Division, primarily in charge of Architecture Strategy for deploying BID solutions. He has worked with Plumtree and BEA since the beginning of 2000, focusing on customer deployments using AquaLogic User Interaction (formerly Plumtree Portal) and later on AquaLogic BPM (formerly Fuego).

His deep knowledge of the BID products and how they are to be used, plus his background in a variety of enabling technologies and enterprise software often has him seconded into other roles besides his primary one. Read the blog for updates on what he's currently working on.

The actual Oracle OpenWorld opening keynote

Ah, Quinton. You thought you went to the OracleWorld (OpenWorld, whatever dude) opening keynote, but you it seems like you mistakenly assumed that it started on Monday, on a business day during business hours. Nope. In fact, the first keynote took place Sunday night. I saw no other keynotes nor do I expect to, but I'm going to tell you that the Sunday keynote rocked the socks and gloves completely off any of the rest of them. (Unless KITT, the car from Knight Rider, shows up unannounced. Then that one would rock a little bit more.)

Yes, I went to that keynote, because:

• It was free.
• I was already a couple of blocks away in the Union Square shopping district, observing consumer spending behavior and making early estimates of seasonal retail demand. (Also, I needed to buy shoes. I can't shop for those online.)
• Larry Ellison was speaking, and he is going to be  might become is trying to become my new boss. Oh, right, I'm supposed to pretend that everyone isn't thinking that. Bite me.

If you want more detail and can stand to install RealPlayer (or Real Alternative), you can get the full show here, or an mp3 of the audio from the same place. You can probably find some reportage like this around the internet too. Here's my summary:

1. Holy crap, this is a big room. My God, it's full of chairs!
2. That thing where everyone the audience holds up a colored piece of paper found under their seats, so that a long shot spelled out something. (I don't know what. Probably something like "Oracle r001z!!!") They took a picture of it. Whoopee. Reminding everyone present that they are mere pixels in Oracle's HDTV.
3. Opening Sunday Night Live skit, where fake Larry Ellison (okay, ex-Saturday Night Live guy Darrell Hammond, the SNL Bill Clinton) ran through a bunch of flashcards and said whatever came to his mind. Highlights:
   PeopleSoft, Siebel, etc: "Got it", "Got it", etc.
   Google (or maybe it was IBM): "Not yet"
   Microsoft: "Don't even want it"
   Nimitz-class aircraft carrier: "Own one"
   Mount Rushmore: "I would look good up there"
4. Larry Ellison dedicates the night to his co-founder, Bob Miner, who died 13 years ago.
5. Larry tells some cute anecdotes about Oracle's founding and early life as a small amateurish company.
6. Larry rambles on for an hour as the anecdotes get tedious and boring and not nearly as cute. Also, Larry has a goofy laugh.
7. SNL people do some lame CIA security skit.
8. Safra Katz, the CFO and President (I thought Charles Phillips was President) says something I can't remember. I think mostly she was MCing.
9. Someone whose name I didn't catch (Hey I'm not paid for this.) talks about Oracle's community and charitable involvement. Videos about it. Nice
10. SNL people do a skit about "fictional" Oracle acquisition target "Bee-Yee-Hale". Oh, yes they did. Highlights:
    "Take off the stupid gas mask. They never use tear gas in the first offer."
    "I hear they have a great gym!"
11. Some Oracle band plays. They were not terrible at first. Then they started singing, and they were terrible. The only lyrics that mattered were: "Oracle! Always...Rocks!" Which entirely sucked.
12. Some videos from old-time Oraclers telling anecdotes that would have been cute if Larry hadn't worn them out earlier.
13. I left and went to dinner. My free pass didn't let me into the party/reception/disco, so I bolted to another part of town to avoid the crowds and traffic. Besides, I reckon that food prepared for 30,000 isn't likely to be very good.
14. Apparently they kept going and did a Weekend Update style bit at the end, but I missed it because it was a big room and the sound took several minutes to reach my end of the auditorium. I just now listened to the mp3. It's only mildly funny, except that Kevin Nealon (a former SNL Weekend Update anchor) told a fake news story about a mysterious ticking package delivered to Larry's doorstep from Alfred Chuang. Which wouldn't have been that funny, except that he mispronounced Alfred's last name. Which isn't that funny, unless you know how to pronounce Alfred's last name.

Was it better than staying home and watching TV? It doesn't matter. I have a DVR.

Looking at the OpenSocial API

I know, I just finished talking about how OpenSocial isn't that important. It's not, but you know, if Google's going to throw around money to get people's attention, I'm going to catch some of the splatter.

So. It would be more accurately called OpenGadget, because the real valuable part of the social network (the "network" part) isn't opened up by this API. It's another Portlet API, much like ours. Or WSRP without SOAP. I talked about Facebook's API before, and the exact same things hold true about OpenSocial. They have differently-formatted code, but it's still basically HTML with extra stuff as output, and REST-type calls for user data.

That would be it, a pretty short post, but there's one interesting thing I noticed that OpenSocial has that Facebook doesn't, and that's the Persistence Data API. What's that good for? Well, it looks to me like OpenSocial lets you save preferences. Like the ALUI/Plumtree protocol, the platform system can keep track of data on behalf of the Gadget/app/Portlet. (Okay, well, it can't yet, because that API hasn't actually been released yet, and those docs are right now just a "preview". Does Google have anything that's *not* a beta? And I guess it's not fair to point out that Facebook doesn't have this API, when in fact, nobody does. Well, we do, but that's eight-year-old news.) What does that do for you? Well, it dramatically decreases the infrastructural burden of writing many apps. Without the persistence API or prefs, if your app wanted to remember anything about the user or what they did, you'd have to have your own database or file or some other place to store data, and you'd have to come up with your own conventions on keying it, and you'd have to worry about space and data management and test environments and yadda yadda yadda. With a prefs API, your app simply tells the platform/portal to remember things about the user for you.

How is this useful? Well, it makes it a lot easier to build apps like, oh, Zombie or Vampire. The coding isn't much different, but on Facebook, I'd have to design and make my own database, then worry about having a database for a development system, for staging, and for production. With OpenSocial (well, once it exists and once they Persistence API exists) or with ALUI (now) you can just tell the platform/portal to remember that a particular user has bitten 853 other users and is a Imperial Grand Kleagle Zombie of Half Moon Bay. You can do this with just code, no configuration, management, setup, or administration on pretty much and web application server out of the box. I think we all know that throwing down a couple of lines of code is nothing compared to the hassle of setting up and configuring your database. Even if you only do it once.

You might think this should be no big deal in a corporate environment if you're integrating real applications that have their own data in a database anyway. Indeed, that's the case if you're writing portlets/apps from scratch. But if you're writing portlets to integrate in exist applications (say, an ERP or CRM system) your portlet code really doesn't need to store anything *except* small fragments of user contextual data (e.g., "I want my default display to be in reverse date order"). All the rest of the data is in the big enterprise system, but your portlet can't put preferences in there. Without the portal being able to store prefs, you'd have to set up a little database to keep track of that kind of thing, which is, again, a big pain in the butt.

Now, if you're wondering, as of today, we have no plans to support OpenSocial in ALI, at least not in the next release. It wouldn't be that hard to do, but we'll wait and see before we commit any effort to it. We already have something functionality equivalent that people are happy with, remember.

So, OpenSocial: That's a good point you scored over Facebook. Or it will be, once you get it over the goal line.

OpenSocial. Yawn.

Google and a bunch of sites with not much traffic (aka "losers") came out in support of OpenSocial. Is it better? Does that mean the end of Facebook?  My take on it is that this isn't particularly important. While the tech press and spin machines get all excited about the thousands of apps available on Facebook, I don't think the apps matter much to Facebook's success. Oh, I have no doubt they make some marginal contributions to visit duration and pages viewed per user, but I think the overwhelming value of Facebook resides in the set of users, the set of relationships between users, and the small handful of apps (Poke, Wall, Message, Status, maybe Photos/Notes) that let you see what your friends are doing. The sorry part then, about OpenSocial is that it tries to take advantage of the least useful feature of these networks. I guess it's clever of Facebook, Google et al to push and hype it, because hey, it gets other people to put in free (free to the network owners) effort to capture the incremental usage. And an apps API gives bandwagon-jumpers and trend-chasers (San Francisco, Silicon Valley, and the technology business press are poisonous with these) some way to associate themselves with success.

Am I too cynical? Well, let's look at that:

• There are thousands of Facebook apps, but I haven't found a good way to search for or sort by useful ones. Don't kid me with that "viral among your friends" line. Despite what Web 2.0 (wow, doesn't that term already smell like "dot-com"?) rhetoric would have you think, "Viral" doesn't means the results are good or useful. And in the Facebook apps ecosystem, the results look like smallpox. All I get from my friends is Movies, Zombies, Superwall, Funwall, Graffiti, Superpoke, and the usual. I can only conclude from this that no truly useful apps exist, or that the viral mechanism doesn't work that well. Neither conclusion speaks much for the significance of apps.
• Facebook group pages can't even have apps added to them. That doesn't seem to hold them back much. (The biggest thing holding back Facebook group pages is that activity in a group I'm in doesn't make it to my news feed. Hey maybe I'm wrong. Maybe the inability to put apps on a group or network page is the biggest thing holding them back, and Ning will slaughter them for it.)
• MySpace seemed to get by without any apps API. The main thing that hurts them is that their PR department isn't as good, and their user demographic isn't as aligned with technology and business journalists. Also the pages look like crap.

Hey, wait a minute there. I'm supposed to be all about how ALUI Portal (Plumtree, whatever) is great because you can build apps in it, just like on Facebook or OpenSocial. Right? So, why am I undercutting this?

I'm not. I'm not saying that remote applications plugging into a personalizable framework is bad or useless, not at all. What I am saying is that, in the context of a social network, it's not that important. In a social network (especially ones that rely nearly entirely on user-generated content), the applications are pretty well defined, and Facebook's got them covered: poke, individual messages, posting of items and status, and that's pretty much it. You get some specialization of applications for different types of messages or content items (for text, or links, or video, or photos, or documents; or a Zombie attack instead of a message), sure, but there are only a few of those (and sure enough, all the functionality of the "Superwall" and "Funwall" apps were quickly incorporated into the Facebook standard Wall app).

But in an enterprise portal, it's different. There, you do have specialized apps doing specialized things, needing information from various other types of systems and applications. You do in fact need to be able to write portlets (or apps) and you can get useful value from it. That's why JSR-168 and WSRP have been around for years while Facebook and OpenSocial both showed up in the past six months (or six days).

Another way to look at it is that the social sites are basically specialized cases of a portal application. They solve a specific application, namely, to connect users, and let them post messages and share content. For this, there are only a few applications needed. (In Plumtree/ALUI, we've had Collaboration Server applications providing document sharing and message posting for about seven years now.) But an enterprise portal needs the more general app/portlet plugin API, as that's core to getting enterprise functionality.

Of course, it's possible that someone could take OpenSocial or Facebook and write some really useful application for it. For example, a social network about cars might have some application that would take a VIN and list out parts and add-ons and notices about your car or something. (Or, in the real world, on MySpace, bands can put up tour dates, songs, etc.) But as useful as that is, it's not core to the value of a social network like Facebook. The value is simply in the connections to other people. That's not to say that they couldn't orient themselves around specific interests or apps, and thereby make these matter (much as MySpace has done in the specific area for artists, or that Ning is kinda-sorta trying to do) but right now they're both buying and selling the "personal connections" story. (Possibly to distinguish themselves from "community sites" which are pretty Web 1.0.)

I'll have more of this to come, because ALI 6.5 (that's the next portal release due early next year) is going to have some more of the social networking features, and I'll want to talk about how they apply, and more importantly, don't apply, in the enterprise.

Release Announcement: AquaLogic® Interaction WSRP Consumer 1.1

We are proud to announce the release of BEA AquaLogic® Interaction WSRP Consumer 1.1.

Summary

BEA AquaLogic® Interaction WSRP Consumer enables you to deploy portlets on AquaLogic Interaction that conform to the WSRP portlet standard. It is recommended that all new and existing customers install WSRP Consumer 1.1.

Release Highlights

• Support for WSRP portlet modes and window states
• Support for WSRP portlet preferences

More specific information can be found in the Release Notes on http://edocs.bea.com/, or using the links provided below in the Documentation section of this announcement.

Download Instructions

Go to the AquaLogic Interaction WSRP Consumer downloads page. (http://commerce.bea.com/showproduct.jsp?family=ALIWSRP&major=1.1&minor=0)

Documentation

Go to the AquaLogic Interaction WSRP Consumer Documentation page. (http://edocs.bea.com/alui/devtools/wsrp/docs11/index.html)

Full RSS Feeds: Oh, I haven't forgetten

Yes, a while back I mentioned how it sucks that dev2dev doesn't have a full-text RSS feeds. (Oh, for the entire blog community, because, come on, there just aren't that many posts.) . Yeah, it's bad for readers. It's also bad for writers. Thank god that somewhere in the infinite monkeys of the internet, someone will take the time to explain why. And dev2dev doesn't carry advertising.

"The mote out of thy brother's eye"

Sometimes the difference between a term of art and obscurantist jargon is how close you are to a topic. 

Facebook's API and my old familiar

Facebook is hot. Like, so hot that if I were making a table of "hot" vs "not", it would include:

Okay, well, in the Bay Area Web 2.0 tech bubble world that is the case. In normal places, perhaps not so much. But hey, you can read something else if you don't care. I want to look at Facebook Apps. Or really, the Facebook API.

It seems there are over 4,000 Facebook Apps available, and there were over 1,000 in the first month. Now, most of them are completely useless stupid crap, but that is true of everything in the world, and it actually speaks well of the platform that it's so easy to use and deploy that even stupid useless applications manage to get created. And besides, there are actually are some useful and quality ones there.

What I found interesting was the similarity between the Facebook and ALUI mechanisms. No, I understate that. I was actually shocked at the similarities between the ALUI portlet configuration pages and the FB app configuration page, and between the plugin mechanisms.

At the most basic level, both of them work exactly the same way: When the FB/ALUI user goes to a page that displays the app, the platform/application makes a REST-style HTTP request to the (predefined) URL of the application, retrieves the content, runs it through some transformations, and displays it embedded in the FB/ALUI application. There are a few differences (e.g., ALUI has a greater selection of preferences pages, but FB has more end-user pages; ALUI has less restrictive HTML requirements, but FB provides more out-of-the-box transformer tags; ALUI sends more contextual data with the initial request, but FB makes it easier to get more on callbacks), but basically, they have the same architecture.

More than seven years ago, ALUI (then Plumtree) devised this remote portlet (then gadget)architecture for the portal to address a few problems we'd found with plug-in web app platforms, and now it looks like Facebook has decided on a similar solution. Why would both choose this?

Well, the more typical type of plug-in architecture is to run the plug-in/portlet/app directly inside of the platform. This is what other portals, Firefox plugins, Photoshop plugins all do. The problem with these is that because they run inside your application, they can cause your application to break. Most commonly it's just because of instability or poor performance, but potentially because of malice. You can deal with this in one of two ways:

• Test and certify and label apps, and inform people that they are taking risk if they install uncertified plugins
• Isolate the plugins so they can't damage the platform

Testing and certification doesn't really solve the problem, though. All it does is make the certifier liable in some way for bad plugins. It also discourages people from creating applications, and scares people away from using them, and requires you to set up some kind of certification bureaucracy.

Isolation it is then. In fact, the best isolation was not running the plugin at all, but making the plugin/portlet/app developer run it themselves. FB/ALUI would just take the final output from the plugin and include it. It happened that back in 2000, we thought that HTTP was a great mechanism for this, because it's no big deal for someone to run a web application on their own. (Back then, we didn't know it would end up being called REST, but there you go, we accidentally built a REST-style plugin architecture.)

This brought a lot of attendant benefits as well:

• Platform-independence: You can write your portlets/applications on any web platform, even without official support from the platform owner. For example, ALUI officially supports Java and .NET, and FB supports Java and PHP. But third parties have developed, for instance, Perl and ColdFusion libraries.
• Lots of portlets/applications: Because the isolation of plugins from the main platform is so effective, you don't need a lot of monitoring and control over people developing portlets/applications. So you can allow anybody to do it, which means lot of them get done. At the same time, if they do a bad job, they can't hurt the platform.
• Security: We ran on top of HTTP, so we could just use SSL for no additional effort
• Sessions: While in 2000, believe it or not, some people were still disputing whether cookies should be used to maintain sessions, we didn't worry about it. We just stuck with HTTP as she was implemented, and the solution came to us for free. (Mostly. In the early days we had questions about their scope.)
• Network flexibility: We didn't have to worry a lot about firewalls. And because HTTP was pretty good over long-distance networks, we rarely had to worry about network latency due to protocol chatter.

Anyway, we're continuing with a good idea. As I just barely alluded to in my last post, Ensemble continues to use a similar architecture for pulling in content to get mashed up.

Platforms and Portals and Ensembles

Marc Andreessen just posted (well, "just"--I'm going back through my RSS feeds) about platforms on the Internet. He identifies three kinds of Internet application platforms, which he calls "Level 1", "Level 2", and "Level 3". Now, they're numbered and he calls them "Levels", so you might think there's some kind of hierarchy, but he does take pains to say that no one is inherently better (or cooler or whatever) than another. Nevertheless, it's sort of implied that the order he uses (which could be expressed as "decreasing amount of infrastructure a developer needs to provide to program on the platform") is good, and seeing as his current company is building a "Level 3" platform, I can't blame him. If you don't like the implicit hierarchy, call them "Access", "Plugin", and "Runtime" platforms, respectively.

BEA doesn't directly sell an Internet platform (that's capital "I" Internet), but we do make things that our customers use to build Internet, internet, and intranet platforms. In this scheme, ALUI is very much a "Plugin" platform, like Facebook. In fact, ALUI has been one of these since at least the year 2000. Most of our customers have deployed ALUI as a "Plugin" platform internally, where various departments or divisions develop and manage their own content and applications that plug in to the enterprise portal and extend its functionality.

Now, as for "Access" platforms, ALUI has one, but for the most part it's used to enrich the "Plugin", i.e., the com.plumtree.remote.prc Remote Client "Access" API supplements the rest of the com.plumtree.remote "Plugin" APIs (such as the Portlet API, the Authentication Service API, or the Crawler API). It's like how the Facebook FQL "Level 1" API is used to enrich apps that run via the FBML "Level 2" platform. I suppose also that SOA methodology in general, and BEA's SOA tools are concerned primarily with delivering good "Access" APIs.

What about this "Runtime" or "Level 3" platform? Well, Salesforce.com has the most prominent one, and apparently Andreessen's company is building another, and he lists out a bunch of others that are getting there. But you know, BEA has Ensemble, which lets our customers deliver one of these types of platforms as well. If you deploy Ensemble to your developers, you'll have system that developers upload code into, that also runs and manages the code. It happens that our platform is for developers to build mashups, so it's actually a combination "Level 3" and "Level 2" platform--base content comes from plugins outside, but the "mashing" code runs inside Ensemble. And we still provide a "Level 1" API, and expect it to be used as well by the outside code.

(Now admittedly, if you deploy Ensemble, you have a load of other challenges that our software doesn't automatically do for you, like hardware and redundancy and backups--but we're a software company, and we deal with the software part of the platform.)

(BTW, here's a hint that Facebook is working on a "Level 3" or "Runtime" platform. The API docs for Authentication mention "Internal Facebook Apps (coming soon)" along with the regular "External Web Apps" and "External Desktop Apps".)

Exercise for the reader

Sorry about the thin posting, but I've just been on vacation, and then catching up from vacation, and just generally busy.

I took these photos at the Gartner portals conference in Las Vegas this week. Why am I posting pictures of IBM's exhibit booth here? Ah, that's for you to figure out by examining the photos. Your hint is schaddennfreude.

Okay, here it is again, just so you know it's not a fluke:

And here it is a third time. Wow, they sure printed those words big:

Further research required: An empirical challenge to my last post

I have no idea how these blogs fit in in any way with what I said the other day. Uh, maybe if most of what you do without a blog is manage a continuous relationship with your consumers, the blog isn't much different. Or maybe it is. You decide. (I found the link here, if you're wondering.)

Well, they don't have wikis or tag clouds yet.

Successful Blogging

So I said a lot, but maybe it's time for me to be specific. What does blogging actually do for your organization? I want to talk about blogging because I think it's technologically simple, but has the potential to be the most radical re-shaper of business. These Social Computing applications (that's what they're calling Web 2.0 applications now, and as aurally unpleasant as I find it, I'll roll with it nevertheless) could really transform your business, not just make it run incrementally more efficiently.

Okay, okay, I keep saying that it will. You keep saying, come on, it's just an outlet for self-indulgent amateur nobodies to spout off on things they know nothing about. You say, what's the point? What's the application that I make with a blog?

Well, let's see. We could use it to publish out announcements and press releases a lot more efficiently than before. We could use it to run a series of educational tutorials and seminars. We could even use it to put out entire publications. Sure, blogs make all those work, but really, they're more about RSS (syndication) than blogs. And really, that's an Org 1.0 way of looking at it. In the Org 2.0 world, the press releases or whatever aren't the application. The blog itself is the application.

Huh? Well, remember this: the point of Social Computing is to amplify how humans work together, and the blog is a way to amplify meeting a person, learning about a person, learning from a person, having a relationship with a person. You can use a blog as a slightly less bureaucratic announcement platform I guess, but that's boring. No, you want to use a blog to establish and develop relationships between people.

Here's an example.

Continue Reading...

Web 2.0 and Org 2.0

I said I'd talk about Web 2.0, but first a detour. (Of course.)

In a recent article, economist Tim Harford summarized some of the key ideas in a paper by Paul A. David, "The Dynamo and the Computer: An Historical Perspective on the Modern Productivity Paradox". While I'll further distill it, I really encourage you to read one or both of the pieces.

I want to start with the idea that new technologies, though they may have the potential to massively and positively transform institutions, and although it may even be envisaged how they may do so, nevertheless take far longer to do so than their proponents expect, even in the face of years of high levels of investment in the new technologies. Professor David's paper was written in 1990 as an investigation into the question of why, as the Nobel Laureate economist Robert Solow remarked, "we see computers everywhere but in the productivity statistics". (This problem, the "productivity paradox", persists today; in the last handful of years we have seen some increases in productivity that are thought by some to be attributable to technology investments of the preceding decades.)

David gives us a history of the electric dynamo and the electrification of factories (you really should read the paper; at least read section II) as an analogy to help us think about the possible future of computers. While he talks about some of the specific changes that electrification wrought on factories, he doesn't go into specifics of what computer technology may do and how it might do it. Well, he's an economist writing 17 years ago, and I'm a (hmm...let's go with...) technologist and visionary with a blog today, so I'll be your huckleberry.

Electrification did little to change factories and work practices for decades, and there were many reasons why. The one that that is most analogously relevant is this: The major benefits of electricity didn't come from replacing steam with electricity per se; they came because electricity allowed "unit drive", which could let you run a factory in a cheaper, more efficient, higher-quality way than was possible with "group drive", which was steam engines limited you to. Electricity allowed you to go from a steam Factory 1.0 to an electric Factory 2.0. (Two examples of the differences: 2.0 buildings could be smaller, lighter, single-story structures because shafts and belts could be replaced with wiring, which was both lighter and allowed more flexible placement of powered machinery; and 2.0 factories could be more efficient because machines could be laid out according to flow of materials and tasks, and furthermore easily rearranged according to process changes.)

But even though the first central electric plant in the country had been operating since 1881, it wasn't until the 1920s, four decades later, that factory productivity growth started to improve. What happened? Well, there were all these existing buildings and machines and training and instructions and processes and institutions and relationships and habits that were all based on the steam-powered Factory 1.0. Even when they electrified an old steam factory, and even when they built new electric factories, they would simply replace the steam engine with an electric dynamo, and would end up with an electric Factory 1.0, not a Factory 2.0.

We've done similar with computer technology. Yes, we have email, and email has substituted as a better and cheaper memo and mail and bulletin board and meeting and runner and pneumatic tube. And yes, we have web sites, and they are a lot better than bulletin boards and forms and catalogs and brochures and manuals and calling up a rep to ask a question. But they do the same thing. Yes, better and faster and cheaper and with much greater reach, but it's the same thing. Going further back, pre-networking, computer reporting and printing was more efficient than accountants with (paper) spreadsheets and ledgers and typing and stenography and typesetting, but it was again the same thing. (Now I'm not discounting that at some point, speed and efficiency are are quantitatively so much greater that that in itself induces qualitative change--email has arguably reached that point--but that's a emergent rather than a contemplated change.)

Continue Reading...

Web 2.0 defined by me

A few times, one of my friends or family has asked me, "So, what's Web 2.0?" because they were shopping for a new laptop and wanted to see if they should get one that had it. (Yeah, that's right. I'm so hip, I have friends who aren't in the technology business and have never heard of an iPhone.) So I tell them they don't need anything special on their PC to use Web 2.0, and that yes, even Macs can use Web 2.0 (Sure, if they use Firefox instead of Safari.) (And then sometimes they think maybe they need to call the cable company and upgrade to a Web 2.0 internet. I set them straight there too.)

But I do give them, I think, an accurate and useful explanation. Provided they ask me when I'm feeling generous, I do. Now, I do explain to them that it's taken on the status of a buzzword and so gets slung around and stuck onto almost anything, to the point where so many different things claim to be "Web 2.0", well it does seem like hype. But nevertheless, there are only a handful of core concepts and features that make something "web" into something "Web 2.0". (Hmm. I wonder how soon the day it loses its capitalization will come. Hey, there was a time when people capitalized "internet".)

I start with a little mythbusting. Sometimes people think Web 2.0 is about ajax, or at least a lot of script and DHTML, or a bunch of richer, fancier UI features. And then I dismiss it, and say that while a lot of Web 2.0 apps have certain types of UI, it's the least important Web 2.0 feature. I mean, sweet mother of God, look at MySpace. The HTML looks like it's from 1998. Facebook too, while not unattractive, is only very lightly javascripted. And del.icio.us, again, no fatty client UI going, just a little ajax. Blogs, again, not a lot of Flash or flash. And YouTube, yeah, uses Flash, but again, not really any ajax. Not that there aren't a lot of rich UIs for Web 2.0 apps out there. Just, it seems apparent that you can cover a lot of Web 2.0 ground without it.

I give a little more credence to the thought that Web 2.0 is more about mashing up, or at least about mashup-able APIs and interfaces. So, RSS and REST, and yes, using ajax to fetch XML is a bit more Web 2.0. I buy some of that, but to be honest, that's the part that makes me say "eh", or maybe "meh". I'm sure someone will pitch a fit at me over this, but I think that the "technology" is barely deserving of the name. I would almost call them "techniques", except that there are a few standards standards starting to actually come out. I think that REST and RSS and JSON and BLAH and YADDA are useful conventions, but I don't think they're particularly revolutionary. After all, all you're doing is getting data via an HTTP request. If anything, they are counter-revolutionary, against the bloat of SOAP, and bring web development back to "stuff you can read with a web browser and write with any web server". (SOAP was too enamored of moving data around with the HTTP protocol, and didn't care enough about the pieces at the ends of the data connection that made and that ate the data. I guess someone figured you could just write all that stuff.) (And btw, I would like to point out that the Plumtree/ALUI Remote Portlet protocol (called CSP) was from the beginning and still is to this day a REST-style protocol.) Now, I think that these technologies are important, and they enable Web 2.0 to really work, and I think they will become ever more important to future Web 2.0 applications. But I don't know if they embody the essence of Web 2.0.

Nope, my thought is that the essence of Web 2.0 is: Content for the Community created by the Users. Now, we can get a little loosey-goosey on the exact definition of "Content" and "Community" and "Users", but I think that's a good summation. (Man, I just couldn't come up with a word for "Users" that started with a hard "c". If someone helps me fine-tune that into something catchier, I'll give them a dollar. No, two dollars.)

I've resolved to make my posts a bit shorter, so it'll have to be for the future to discuss how this is useful for companies. Oh, just a taste then: It's your individual employees who know to do many things, particularly in knowledge-worker organizations. But what we often want is for the individuals to be able to share and leverage what they know, so that it becomes institutional rather than individual knowledge. Knowledge for the Organization created by the Employees (and other Business Partners).

PEP: Pages & Ensemble & Pathways

As you may have heard, our new Web 2.0 set of products have been released. Now, I'd been planning to blog about these things for a while, not least to let people know a bit better about what they are and what they can do for you. And then other people go out there and try to steal my thunder. Well. Seems to me I need to step up my game.

Basics first though. What are Pages and Ensemble and Pathways? (And BTW, they are collectively called PEP, "the Web 2.0 products", Social Computing tools, and formerly called RGB, for "Runner-Graffiti-Builder"). Now you can go to another BEA blog http://en.terpri.se. (Yes, I too think it's just clever enough to be embarrassing.) and get the long story. Here's the short one:

• Pages: Like a wiki, but instead of just text and HTML, you can also put tables, fields, blogs, and other objects that display real-time structured data from external sources on the page. Oh, plus drag-and-drop layout and editing. But it's got that wiki quickedit stuff, that real-time collaborative space with data, and not just text, plus wiki-style version tracking.
• Ensemble: Like the ALUI portal, but without the portal pages HTML, just the gateway. Or a super web app transformer proxy. It lets you take existing web apps and wrap security and login control around them. It lets you track and audit their usage. It lets you run pagelets with tags (like that are just web apps, like portlets) and mash them up with each other and with some Ensemble-provided Ajax pieces on the server. Imagine you can do some of the GreaseMonkey stuff to snips of HTML, but you do it on the server so there's no browser or plugin or client installs.
• Pathways: Uh, well, this is just searching. Oh, but wait, it's not! It's searching and taxonomizing and tagging and associating and finding related information and users. I guess that's it on the surface, but what's important about Pathways is that it isn't just a program that works on data. It's a program that works with people and what they do with data, and helps people uses that information to find more data and more people. Recursive, yes? You might be interested to know that Pathway's main algorithms are recursive pretty much like I described. (With a bit more rigor, sure.) And you know what else? Pathways I think is culturally the most Web 2.0 of the PEPs. That's going to get its own post some day.

Okay, so what makes these special to you? (Of course they seem special to BEA. Everyone thinks their own baby is the cutest baby in the world.) Well:

• Yeah, there is other blog software out there that's very good, like Movable Type and WordPress. And there's all kinds of Wiki stuff. And there are some beta hosted sites like Pipes and Popfly that lets regular non-hardcore-developers use drag and drop and hook up Web Services to make mashups. But as far as I know, there's nothing that combines them, and there's certainly nothing out there that lets end user make mashups with enterprise data on top of that. Plus, like the rest of the Web 2.0 stuff, it hooks into your enterprise security system, allowing you to assign rights and roles using corporate identities, groups, and roles.
• There's nothing like Ensemble out there. Well, except ALUI portal, and that's only like it in some ways.
• Sure, there are tons of search engines and categorization engines and appliances. This isn't just a search engine as I said. Yeah, yeah, in the end, you're using it to find "stuff". But it's qualitatively different because instead of just looking at documents and maybe an imposed taxonomy, it looks at how stuff is used and what each person is actually doing. And then, importantly, it makes people and relationships just as important as documents and data. And, well, if you're the type of organization that relies on human thought, that might work out for you.

btw, apropos of nothing, this is creepy and disturbing.

TCP, UDP, Unicast, Multicast. WTF? I thought this blog was supposed to be about portal.

This post gets pretty low-level, but I think this is important. As much as everything in our business relies on us abstracting away things at lower levels and specializing, abstraction only works up to a point. In some ideal, interfaces are clean and well-specified and focused; the application doesn't worry about how the OS works, and the OS doesn't worry about how the hardware works, and layer 4 of the OSI model doesn't worry about how layer 3 works. All you are supposed to do as a specialist at one level is assume the levels below you just work. But does that really work? If you're writing an application, it's better if you really know how the OS works and you do have to worry about how the database stores characters, and a good OS must worry about how the hardware works, and you're on crack if you think that TCP didn't worry about how IP was implemented.

So here, even though we're supposed to be about web applications and services that run up above OSI layer 7, we're going to go down n look at layer 4 (and even a bit lower)  and see what's happening. We'll discuss what the differences are between TCP and UDP, what multicast is and how and why is works and doesn't work. Trust me, it'll be useful, and it all leads to some info about how PTSpy works in G6 and some ideas on how you can use it better.

So, let's start with HTTP. That's the protocol that matters directly for most of what we do here. HTTP, along with most of the rest of our application networking (SQL*NET, ALI Search) run at network layer 7, the application layer, on top of TCP which operates at layer 4. So what's TCP?

TCP (Transmission Control Protocol), along with UDP is one of the main underlying network protocols of the internet. Both of those are built on top of yet another one, IP (Internet Protocol). IP works one layer below TCP and UDP (layer 3). So to understand TCP, let's look at IP. We'll then work our way back up and talk about what TCP does on top of that.

IP: Internet Protocol

IP provides the basic ability to route a packet of data from one place to another. It does this by providing each "place" or "device on the network" with a specified address (the IP address) and specifying how such devices should move packets around based on the address. Now, the difference in functionality between IP and the protocol layer below it (layer 2, the data link layer, typically represented by Ethernet or WiFi) is that at layer 2, devices always know exactly how to send to every device on the network. At layer 2, either you know exactly how to get data to your destination (btw, usually represented by a "MAC address") or it's not possible to get it there. (For example, Ethernet and WiFi simply broadcast the entire packet onto the whole network, and the destination is expected to be listening for its MAC address and pick up the packet. If it's not there or not listening, Ethernet can't get it there. [And btw, network "sniffers" works by taking advantage of this broadcasting.] PPP, used in modem dial-up connections, can send everything to one destination only: the machine you dialed in to.)

What IP does is provide a way to connect separate networks so that devices on one network can send data to devices on another network without knowing where they are or how to get there exactly. Hence the appellation inter-net, "between networks". It does this by specifying routing rules that define what a network device does with a data packet with any destination address. Basically the rules are: if the destination is local (i.e. you know where it is because it's on the same network) send it there; otherwise, send it to one of a list of routers that vary according to the address. A router follows the exact same rules, except that it's assumed that it belongs to two or more different networks at the same time; and therefore has more local destinations, and also usually has a longer list of other routers for unknown destinations.

Very specifically now, IP does no more than give you the ability to send a single packet of data to a destination specified by a single IP address, and that's it. It can get that packet to any place on any network (unlike the lower level protocol), but that's about it. Significantly:

• It does not provide notification of delivery or receipt or failure.
• It does not provide any notion of "port numbers" or other ways of segregating packets at the destination IP addresses.
• It does not provide two-way communications.
• It does not order or otherwise group multiple packets in any particular way.

The simplest analogy is to think of IP like the postal service. You drop a postcard into the mail with an address on it, and somehow it makes it to the address you wrote on it. If it makes it, or if it doesn't, you don't know. When it gets to the house, you don't know if the roommate reads it. And if you want a reply, your recipient can't just write on the same card and hand it back to the mailman; they'd have to write write out their own note, stamp and address it, and send it out again themselves.

TCP: Transmission Control Protocol

While IP does not provide any of those features, TCP does. If you looked at the list of the features that IP doesn't provide, and then you looked at the analogy of the postal service and said, "Huh? Of course you can have two-way communications. People used to write letters back and forth and have conversations all the time," or "Well, you could just request that either the recipient or the postal service send you back a return receipt," or "Come on, dummy, you can sequence your postcards by just writing numbers on them and telling the recipient to read them in order and let you know if any are missing," well, then, you're right. That's exactly what TCP does. It takes the basic IP (or postal service) and specifies how to use it and what to put in your message to get all those extra features.

So what TCP really winds up doing is implementing the concept of multiple reliable connections between IP devices. What does that mean? It means that you can send a sequence of messages (packets) about a specific conversation (port, or connection) and that the packets will be received in the same order they were sent, and that there won't be missing packets in between sent packets. (Also, there won't be duplicate packets.) It does this by attaching to every packet both a port number, to separate and identify multiple connections or conversations; and a sequence number, to let the recipient identify when data might have been lost in transmission. In addition, it specifies the recipient has to acknowledge every data byte that was received (not necessarily explicitly: it might simply say, "I've received everything up till byte 13456"; and it can also acknowledge selectively, "I've received everything between 845 and 13433") so that the sender can know if missing data needs to be resent. And finally, every connection is implicitly two-way, not just for acknowledgements, but also so that the recipient can just send data "back" rather than having to explicitly re-address (sort of the equivalent of sending a self-addressed reply envelope with every packet).

As you can see, if packets are getting lost or arrive out of order, there is actually quite a lot of work that TCP has to do. If we're continuing the postal service analogy, TCP is a bit like a personal assistant who collects your mail, groups it, sorts it in order, brings and reads it to you, takes your replies and organizes and sends them back. If the postal service is super-reliable, his job is easy, he's just a middleman passing paper around. If the postal service loses a lot of stuff, and/or you have a great volume of mail, he may have to do a lot of work, request that things be re-sent, and keep track of and store a lot of messages.

UDP: User Datagram Protocol

UDP on the other hand is a lot simpler. It does what IP does, but it adds the concept of a port, so that you can send messages to a specific recipient at one IP address. It doesn't have order, or connections, or two-way communication, or acknowledgements.

You might think that UDP is unreliable, because, you know, TCP is supposed to be the reliable one of the siblings. But in fact, over the same network segment, or over LANs with good quality gear and not excessive traffic, UDP is in practice very reliable. If there's no packet loss and packets arrive in order (which is almost always the case on a short LAN link), there's no need for any retransmissions of packets, so all the acknowledgements and waiting around of TCP is just a bunch of wasted overhead, creating latency. For applications that can tolerate packet losses (say, real-time audio and video), it's often a good choice even over not-so-good networks. It's also used for small messages and notifications. For example DHCP and DNS both use UDP. And notably, the Unix Network File System (NFS) used UDP over LANs. While you'd think a file system should have used a reliable TCP connection, the implementers of NFS felt they could get better performance using UDP and building their own mechanisms for ensuring reliability that were tuned to the application.

Btw, it's called "User Datagram Protocol" because it's was designed by operating system guys. "Datagram" is just another word for "packet". "User" in this case doesn't mean, like, you; it means a program on a computer that is not part of the operating system. The idea was that IP is low-level and it was used by people who write parts of the OS, while UDP provides mostly the same functionality (the datagram), but is intended for "user" (non-OS) programs.

Multicasting

I simplified the TCP/IP/UDP discussion up there a little when I implied that IP (and hence UDP and TCP) would route packets from one device on a network to another device. More accurately, IP routes packets from one IP address to another IP address. The trick with multicasting, or sending the same packets of data at the same time to many devices is that certain IP addresses can be designated as multicast address and assigned to many devices at the same time.

The first thing to know about IP multicasting is that there only exists UDP multicasting. There's no such thing as TCP multicast. Why not? See, the point of multicast is to be efficient with the network and send the same data packet to as many different, possibly unknown computers. But each TCP connection can potentially require retransmission of different lost packets, or different delays or ordered arrival and assembly of packets. So managing and sending all that for each possible device would be resource-intensive even if it were possible, which would defeat much of the point of using multicast. (And it's not possible anyway because multicast can't know where the outbound packets actually wind up.)

Oh, and via back-formation, regular non-multicast UDP (and TCP) messages are called unicast.

The next thing to know is that multicast will often not be sent over a router to another network. There are a few reasons why:

• A low TTL for most multicast packets: All IP packets have a time-to-live, or TTL. Unlike with DNS records, TTL on IP packets refers to the maximum number of network hops that a packet can make to get to its destination. Unicast packets typically are allowed to cross about 30 networks (i.e., get routed by, or "hop", 29 routers). Getting across the internet rarely takes more than 15 hops, so the limit is really just to kill packets if they get stuck in loops in badly configured networks. But most applications that send multicast set the TTL to a low number, often zero (where the message doesn't even leave the originating device),
• one (so it will go only to computers on the local net), or two (where it will cross a single router). It's quite rare to have an application that is intended to multicast to unknown machines over an entire campus network, much less the entire Internet.
• High TTL thresholds set on most routers: Many network routers, particularly WAN routers and internet gateway routers, have a higher TTL threshold set, such that they won't send out multicast packets whose TTL is below (say) 15. This is intended to prevent accidental leakage of multicast off a local network.
• Routers are simply not configured to route multicast at all, or only configured for specific addresses, or otherwise blocking multicast packets.

UDP multicast might seem a bit exotic, but it's actually more common that you think. It's not used for web video sites like YouTube, since those need to send a video at a time each user demands, rather than at the same time for all users, and it's not used for VoIP traffic. However, it is used for a lot for discovery and automatic confguration in apps like Skype, iTunes, and uPnP. It's also used in a few places in the ALUI portal.

Spy Logging

Since G6 was released, Spy (formerly PTSpy) and the Plumtree Logging Framework (AL-something Logging Toolkit or whatever) have worked using UDP. The application that has messages to log just sends it out via UDP. Now, recall that UDP doesn't worry about things like actual delivery and order and error correction and flow control. The message can be sent out as "fire-and-forget", or "drop it in the outbox and forget about it forever". There's no record-keeping or waiting, so it's very cheap and quick for the program to just put send the message.

In fact, it costs pretty much the same for a program to just send out the log message via UDP as it would for it to check to see if the message should be filtered. So, the G6 portal (and collab, and the api and automation servers, etc.) doesn't bother checking; it just sends out the UDP log message unconditionally.

In contrast, 5.x and earlier portal versions logged positively: every message had to written to a file (or the screen) before the program could proceed to the next operation. This impacted performance severely enough that you could only enable tracing when you were actively debugging, and even then had to try to limit what you logged. Throwing out the UDP packet causes so little delay that G6 traces a lot more, and it sends the trace messages the whole time the program is running.

Now, we also have a set of programs that read these UDP packets from the network. The most familiar one is Spy itself, which captures the packets, picks out the ones you told it you wanted to see, and displays only those. There is also the Logger Service, which works similarly, but reformats the messages and writes them to a text file.

You might not even actually know that the ALUI G6 Logging Framework is using UDP to send log messages, because normally the applications are set to "local logging only" which sets the packet TTL to zero, which doesn't let it leave the originating computer (but which can be picked up by a destination on the same computer). At the same time, Spy is usually installed along with the application. When you start Spying, it just picks up the packets from the local application, and it looks like the old 5.x PTSpy connecting locally to the locally-installed application. But in fact, the application and the viewer for the Spy messages are now very loosely coupled over the network, and run independently of each other.

While UDP does offer great performance advantages, so much so that we now can have much more detailed logging available when necessary, there is the possibility that some log messages can be lost, just because of the unreliable nature of UDP. This isn't really a disaster, but it could be annoying if you're debugging and you don't know it may be happening. It may happen when there is a lot of activity on the portal (and hence a lot of messages being sent), the network is very busy so packets are lost, or the capture program is running slowly. In practice, the most common case in which messages are lost is when the Spy program is running slowly, maybe because the machine it's on is busy, or because of some other resource issues. We've found in particular that the X-Windows Spy, when writing to the screen, can miss a lot of messages. I usually recommend using the command-line logger on Unix anyway.

One more thing to note is that Spy is actually sent out via UDP multicast, not unicast. This means, first of all, it's possible to have several programs picking up and reading and recording log messages from a single application. For instance, while you are looking interactively at Spy, the Logging Service can also be recording the same or a different subset of messages to a file.  Second, you can also monitor several applications remotely using a single (or multiple) programs on a remote machine. You simply need to make sure that the application "local logging only" option is set to false, and that your network and routers will route the packets from the ALUI applications to the logging listeners.  Then, just open up Spy, select View->Set Filters, then Edit->Add Message Sender. Spy will look for applications that are configured to log via UDP multicast over the network (and which can reach Spy) and allow you to choose to capture and record the messages. 

 Collab Clustering

The other place that we encounter UDP is when configuring Collaboration Server clustering. Collab clustering is used for cache invalidation among the Collab servers. When an object is updated on one Collab server, a message needs to be sent to all of them identifying the object. That's a small, simple message.

There are several configurations for setting up Collab to send the message. The two most common should be the "lan-cluster" configuration and the "lan-multicast-cluster" configuration. Both of them use UDP to send the messages. The unicast configuration requires you to list out every other server in the cluster in each server's config, but will work on any network where the servers are actually reachable to each other. The multicast setting requires less configuration and config maintenance, since you all you do is set the multicast listen address on each one to the same value, but you run the risk that the network may not support or route multicast among your servers. It's also hard to know if it's working without running network traces to look for the Collab cache invalidation messages, as for the most part Collab will run fine and no one single user is likely to notice inconsistencies in caches across different servers.

Well, I hope that was informative and interesting. And useful.

January 2008

Search this blog:

Archives

November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
April 2006
March 2006
February 2006

Categories

Recent Entries

The actual Oracle OpenWorld opening keynote

Looking at the OpenSocial API

OpenSocial. Yawn.