Compliance - Never Talk When You Can Nod

Sometimes it's not what you say, it's how you say it...

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Wed, 30 Apr 2008 16:13:55 GMT

When I have time to spare I love to spend time browsing other people’s Blogs; one of my usual browsing locations is the Big Men On Content Blog. Earlier this month they posted an interesting article which discusses a CMS Watch article about SharePoint vs. Exchange Public Folders. It wasn’t the actual content of Lee’s Blog entry that caught my eye so much as the way that he phrases his thoughts; he is about as subtle as I am, (for which he earns my admiration.) Here are two examples taken from his entry:

“...Sharepoint, when all the marketing and hoopla is stripped away isn’t ECM anyway. It’s really just Microsoft Portal with a tightly coupled workgroup functions - some of which deal with content...”
“[Microsoft] wrote their own portlet spec and called them webparts. (arguably the worse name in technology - might as well have called them naughty bits...”

Thanks for continuing to entertain Lee!

ECM delivered using SaaS

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Sun, 20 Apr 2008 16:01:07 GMT

As I mentioned previously, we had an internal writer's summit a few weeks ago. One of the recurring topics that came up related to how to deliver ECM-related functionality in the brave new world. I'm busy enough trying to keep up with SharePoint and Compliance to dive head first in to the wonderful world of SaaS but I wanted to share some of the specific items that I thought were interesting.

Normally I'd pretend that these ideas were my own but this might be a career limiting move given that the ideas were stolen from people like Howard Shao (founder of Documentum) and Mark Lewis (runs the entire Content Management and Archiving division at EMC)!

SaaS Should Be Delivered Like Electricity

This gem from Howard was memorable primarily because of the simile that he used. Currently, if you are using a SaaS application and you need more capacity, access to extra functionality, etc. you have to call your service provider and make a formal request and they provision those changes - this might typically take days to happen.

Howard's argument is that these services should be available to you completely on demand. The current state of affairs is synonymous with buying a new appliance bringing it home and then having to call PSE&G and asking them to provide another 65Watts of capacity for your house.

Later in the day he added a great addition to the simile. He mentioned that with a SaaS model you may need to have a local cache of content or even some functionality either to supplement performance or for access if the SaaS service is temporarily lost. He compared this as having a generator at your home so when either the grid goes down or when a brown-out hits you have access to extra power locally

Howard's observations are always astute and in this case I thought that this really brought home one of the challenges of abstracting the service from the customer - the customers become ignorant and ambivalent to the operational aspects of the system. Their expection is that the system will be available all of the time, will be as efficient and responsive as every other system in the organization, integrate with everything seamlessly...and they want it for free!

ECM's Unique Demands on a SaaS Implementation

Howard very succinctly summarized, (that's not a pleonasm BTW, you can un-succinctly summarize something), the challenges that ECM brings to a SaaS deployment model. I thought this was worth 'memorializing' so here we go:

ECM causes a substantially greater demand on SaaS storage requirements and on the capacity of the network. ECM payloads are typically multi-megabytes in size and users expect content to arrive within seconds.
Customers may insist that some types of content are stored on-premise. Customers may feel that very high value intellectual capital should not be stored off-premise. Personally, I think that this is unwise and I think that the concern will change over time - let's face it, if you have a decent SaaS provider then they should be able to protect your content better than you would ever be able to. That said, there are sometimes country-specific compliance requirements that may demand that some content is stored in a certain geographic location.
In order to provide adequate levels of security, compliance and performance, ECM systems will need to manage content that is on-premise and off-premise. This will have to occur in a way that is transparent to the end user.
The SaaS solutions will need to support tiered storage devices and repositories. Again, this needs to be done in a transparent way.

Hierarchical Storage Management meets SaaS

The last two items above both point to the fact that the SaaS systems will have to be able to abstract from the end user the actual location of the content. Typically we see this type of functionality implemented internally by customers - if you use a decent ECM system it will be able to move content between storage devices based on the value of content or how frequently that content is accessed. Users typically have no idea where the content is stored, they just access it. This concept of HSM is typically just used to move content between different types of local disk storage but imagine if the HSM system was running at your SaaS provider's off-premise location but one of the HSM target location was your local system.

If one of the HSM target locations was your local system then we could set up rules that moved specific content types to your local system if the content was being used very frequently, was highly dynamic or you had a regulation that determined that the content had to be stored in a specific geographic location.

Does TiVo Represent the Ideal ECM/SaaS Architecture?

Someone mentioned TiVo as an example of this type of concept. Many of the broadcast TV companies use ECM solutions to manipulate, manage and store programming; they run all of the systems and provide you with 100 channels of garbage 24/7. However, if you want to keep a copy of some of their programming locally you can manually capture something to your local cache in real time or you can even set rules to cache content to the TiVo box.

Conclusions

I don't really have any conclusions other than to say that SaaS-based ECM is going to happen whether we like it or not and thank goodness that companies like EMC have people a lot more intelligent than I working out how to pull it all together!!

Eight Reference Architecture Organizer

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Sun, 20 Apr 2008 15:47:31 GMT

Someone asked me if it was possible to have a single page with links to just the "Integrating SharePoint with Traditional ECM Systems - Eight reference architectures" articles. I think that they were politely saying that they had no interest in my other articles...so this is it, alternatively, you can click here to filter by the relevant category. I'll try to make it "stick" to the top of the Blog.

Reference Architecture 1: Keep Systems Separate, Restrict Usage.

In this architecture the publishing of content between SharePoint and the Enterprise Content Management system is performed manually by the end user. This is not an approach that I’d advocate but it is included for completeness and because it is a method frequently used today simply out of necessity.

Reference Architecture 2: Loosely Coupled Solution

This is probably the key architecture behind any real unification solution today. In this architecture, content is moved between systems depending on its value to the organization. For example, work in progress might live in SharePoint and formal records might live in your ECM system. The move between systems is performed by a process and can be invoked manually or triggered by a specific event.

Reference Architecture 3: Use SharePoint as a Portal Container

In this commonly seen architecture, SharePoint hosts Web Parts that point at your SharePoint repository or at your Enterprise Content Management system. For example, one part of the screen’s real estate is displaying content from SharePoint while another part is displaying your ECM system’s Inbox. This is stretching the definition of “unification” IMHO but for reasons that will become clear later I think that this approach is often essential to make up a complete solution.

Reference Architecture 4: Passive Unification in Web Part

In reference architecture 3 the unification occurs within the SharePoint portal container. In this architecture the unification occurs within a specific Web Part; a single Web Part displaying content from a variety of disparate systems. We will look at this unification model from a passive perspective – only allowing the user to view or export the content.

Reference Architecture 5: Active Unification in Web Part

Building on reference architecture 4, we will see what happens if you attempt to add more complex operations to the affray; supporting operations like checkout, managing versions, connecting objects to a workflow, etc. The addition of these more complex operations increases the complexity exponentially.

Reference Architecture 6: Passive Back-end Aggregation

The first five architectures are all focused on unification. This penultimate example creates a passive aggregation model where we use tools to create a virtual aggregation across the SharePoint and traditional ECM environments. We can then monitor the aggregated environment and take actions as required. For example, we might compare the access rights on a piece of content that lives in 10 different document libraries and report on access inconsistencies.

Reference Architecture 7: Active Back-end Aggregation

The Holy Grail or Pandora’s Box? Real aggregation of the SharePoint and traditional ECM silos allows us to perform many operations that would be very difficult or even impossible otherwise. Having all objects from all environments addressable within a single space and supporting operations on those objects allows us to really address some of the current limitations in a scalable & secure way – think “single instance storage” for example.

Reference Architecture 8: Synchronized, Intelligent, 2-way Shortcutting

In this model the ECM system stores the only copy of the actual payload of an object, (the document, spreadsheet, image, etc.). A shortcut/stub/proxy/pointer is created in SharePoint; this points to the object in the ECM system and mirrors all of the appropriate object metadata. Actions applied to the pointer object are redirected to the ECM copy when appropriate. Both the SharePoint object and the ECM object are protected by event triggers so that changes to the properties, location, etc. can be synchronized and when either is deleted the other will also be disposed of.

Note that this RefArch works for content that already exists in the ECM system, is being created by a 3rd party process or is "published" from SharePoint.

Unification at the Middleware Layer

You may have noticed that the integrations above all manifest themselves in the client or server layer however the majority of SharePoint’s strength lies in the “middleware” layer. I am not ready to discuss middleware-level integrations yet but I expect to be able to add some more meat in to that layer of the sandwich by the time the book is ready to go to press…you may have to part with some cash to learn those bits!

Reference Architecture 8: Synchronized, Intelligent, 2-way Shortcutting

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Sun, 20 Apr 2008 15:46:27 GMT

In this model the ECM system stores the only copy of the actual payload of an object, (the document, spreadsheet, image, etc.), however the object can be managed, consumed and deleted from within SharePoint.

A shortcut is created in SharePoint, the shortcut points to the object in the ECM system and mirrors all of the appropriate object metadata. Assume that you have a Word document called "widget_spec.doc" being stored in the ECM system. The document was created by me on 04/20/2008, I have full access to it and the rest of the world has read-only access. A shortcut would be created in SharePoint that has the appropriate access controls, the object name, title, author, creation date and also the location of the real file in the ECM system, (but not the actual file).

Any actions applied to the shortcut object are redirected to the ECM copy when appropriate. If you click on the shortcut and open the document that action is redirected to the ECM system which grabs the real document and streams it back to SharePoint. The user doesn't know, (or care),that the file came from the ECM system.

Both the SharePoint object and the ECM object are protected by event triggers so that changes to the properties, location, etc. can be synchronized and when either is deleted the other will also be disposed of. This is important because you need to allow the object to be managed from either within the ECM system or from within SharePoint without the shortcut and the original object getting out of sync.

Note that this RefArch works for content that already exists in the ECM system, is being created by a 3rd party process or is "published" from SharePoint. What does this mean? It means that you could move content from SharePoint in to the ECM system and leave a shortcut behind, (kinda like HSM for SharePoint isn't it), you could ingest content in to the ECM system from any source and in real time expose that content in to multiple SharePoint document libraries or you could take existing content in your ECM systems and make it available to one or more SharePoint sites. Attractive proposition isn't it? We call this architecture 'projection'.

I am not going to delve any deeper in to the technology behind this solution for similar reasons to reference architecture #6 but needless to say, this is a very interesting area for ECM vendors right now.

The end...

That's it for the reference architectures for now. Committing these to Blog has been a great therapeutic exercise...thanks for sticking with it!

Spontaneous Thought Memorialized

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Thu, 17 Apr 2008 21:40:33 GMT

We had a writer’s workshop a couple of weeks ago here at EMC. It was a great idea; we had a one day summit for internal and external authors, bloggers, wiki contributors, journalists and other general media contributors. We spent one evening and the whole next day not just discussing the intricacies, trials and tribulations of writing but also discussing some of the new concepts that are affecting our industry. It turned out to be a very interesting and informative day.

Barclay Blair, (the co-author of Information Nation), mentioned that blogs were dangerous because they represented “Spontaneous Thought Memorialized”. I looked at my Blogs and feel that mine are more like "a photograph of verbal diarrhea". This is probably why Barclay’s book is a best seller and mine…anyway.

I’ll be writing some more entries based on some of the sessions at the workshop over the next few weeks; look for articles on HSM across the cloud and maybe even some thoughts on Web 2.0.

Reference Architecture 7: Active Back-end Aggregation

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Wed, 09 Apr 2008 17:54:37 GMT

So unlike little old RefArch 6, this guy is the real deal. In this architecture we aggregate the actual content from a multitude of SharePoint sites. The content is transparently taken from SharePoint's control then stored and managed in a truly aggregated single location. In theory, there are a number of different ways of doing this:

You can replace the entire SQL Server layer with an ECM solution that emulates the entire SQL stack. This gives you control over the content, metadata, lists, calendar items, Blogs, etc... in fact everything that SharePoint creates.
You can journal content directly from SharePoint in a synchronized model, a slave copy of the object will remain in the site but the master copy lives in a centralized ECM system.
You can journal content directly from SQL Server leaving SharePoint unaware that this is happening.

Architecturally the net result is that all of your content from all of the key SharePoint sites ends up being stored in a single, centralized location. It is then managed as a single set of content, it can have hierarchical storage management applied depending on the value of the content, it can be managed with a single set of security & compliance policies, it can have duplicated content thrown away...if you want to see an example of this architecture in play then take a look at the more advanced email archiving solutions in the market.

So what's the best way to do this? So herein lies a bit of a Blogger's dilemma; rumor has it that quite a few people read my Blog - most importantly, you do and you are my favorite reader. I do actually have an answer to how this is best carried out but I'm sorry to say that I am not going to share it at this point. If it were entirely in my hands I'd consider it but the solution to this problem requires a fair amount of collaboration and technical assistance from Microsoft and from some of our key partners and I am not willing to divulge the details until we at EMC have made a bit more progress. Sorry.

Don't you just love it when this happens? You get to the end of a 2 month Blog saga and the penultimate page has been torn out. Hope you don't hate it too much because part 8 is going to have a similar ending. In my defense, the solution to this problem will blow your socks off and will be worth the wait.

If you are under an EMC NDA and are attending EMC World this year then maybe, just maybe we can chat about the details.

Reference Architecture 6: Passive Back-end Aggregation

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Sun, 06 Apr 2008 18:51:25 GMT

Let me preface this entry by saying that this reference architecture is similar to reference architecture #1 insomuch as it is not something that I recommend or endorse but it is something that I see in use fairly frequently today; in fact many of the solutions recommended by Microsoft fall in to this category. Best case, feel free to use this approach while you wait for more appropriate solutions to come along.

In the previous architectures we were trying to create a unified model so that an end user could simultaneously see content that resides in the SharePoint systems and also the traditional ECM systems. In this, (and the next), architecture we are going to directly deal with the SharePoint "silo sprawl" in the data center. I call this approach 'aggregation' not unification. The advantage of architectures 6 & 7 is that they are completely transparent to the end user - that's also their failing; these architectures are so transparent that you cannot use them to expose the advanced capabilities of the underlying ECM solutions to the end users. FYI - I am working on a cocktail of solutions that give you the best of both of these worlds but I'll probably keep that under wraps until I have it ironed out a lot more.

The general premise here is to leave each instance of SharePoint running independently as individual disparate systems but to create a virtual aggregation of the content and potentially the metadata behind the scenes. This approach does not actually change any part of SharePoint, the reference architecture's processes and technologies simply monitor the SharePoint implementations and create an aggregated view of the content stored across all libraries. This aggregated view can then be used to make security decisions, perform risk analysis, monitor file usage, etc.

For example, one could create a master index of all content in all SharePoint instances inside of the organization. This master index would contain the basic information about the contents of all of the document libraries along with the security access settings and a hash value of each of the pieces of content. You could then analyze this information and look for instances where a file’s security is not consistent. Let's consider a use case: assume that the company’s year-end financial results are stored in the finance department’s SharePoint document library with very restrictive security in the weeks before the official financial results are announced. If the passive unification process looked in the master repository index and noticed that this same file was also available on one of the finance department's collaborative SharePoint sites with no security then you might want to do something about that! Note that it would know that the two files were the same file even if the file name had been changed because the hash value would uniquely identify the file.

There are plenty of other use-cases for what to do with this data once you have access - monitoring data duplication, managing disposition, etc. but is this really an aggregated back-end? It isn't really because nothing is actually aggregated in reality but it can be useful because it creates a virtual unified view of the data and once again it is a very non-invasive, low impact approach to the solution.

One issue to consider with this approach is scalability. If you have a billion data objects in your SharePoint systems then your enterprise-wide content index would have a billion entries in it. The other consideration is how to keep this index synchronized as content is added, altered and especially removed from the systems. Those types of operations are very CPU-heavy.

Talk to the market leaders in database, search and indexing technology for advice on these issues. A billion objects synchronized across an enterprise is theoretically within the abilities of most enterprise database systems however I have talked to customers who generate over 1.5 billion records a day, (yes, that's the right number), these are transactional records currently ingested in to specialized storage but they could easily be stored as XML output files. Assume that they generate 1.5 billion records each working day they will end up with roughly 390 billion objects a year! Scalability becomes an issue very quickly with this kind of ingestion.

Again, I see this as being an architecture that is included for completeness and because it contains some concepts that are useful elsewhere, (think eDiscovery for example). Also, it can be "done on the cheap"; you can achieve passive virtual aggregation using the tools that you own today. A more useful reference architecture is to create some form of real unification at the back end...see the next reference architecture for more on that thrilling concept.

SharePoint For The Enterprise

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Tue, 01 Apr 2008 08:43:10 GMT

Craig Le Clair from Forrester Research posted an interesting article last week, (I was on vacation skiing so oddly enough I didn't read it until this week). In it, he suggests that the upper limit on a SQL 5 installation running on a 64 bit architecture is about 500GB - oddly enough 5 is the average number of drinks I had each night when I was on vacation last week, there were 64 runs on the ski slopes and I fell over exactly 500 times on vacation last week. I suspect that Microsoft will not agree with these numbers but in truth they are pretty realistic, I really did limit my drinking, go to a big resort and fall over a lot ;=)

Seriously though..I am sure that Microsoft will dispute the 500GB limit so I'd like to propose an informal pole, (not a ski pole which I used last week on vacation). Do you have more that 500GB running somewhere and if so - what is your architecture?

Did I mention that I was on vacation last week?

SharePoint vs. ECM; Same Battle as SQL Server vs Oracle?

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Mon, 17 Mar 2008 12:29:47 GMT

I had a really interesting comment posted on my "SharePoint 2008 Report" by Marko Sillanpaa. Marko is half of the team over at http://www.BigMenOnContent.com and has been around the ECM space for a long time. His question was interesting enough for me to promote the answer to a blog entry. Here's Marko's comment, (reproduced without his permission or any regard for whether he minds or not.)

Thanks Andy. This is a great conference summary for those of us who wanted to be a fly on the wall.

I’d like to ask a question about your first bullet. You said that SharePoint sees themselves as a platform. How should we look at Documentum, a solution or a platform? It’s easy to see how a solution and a platform would work together. But when I look at two solutions to solve the same problem or two platforms to build the same basic sort of solutions, I just don’t see the value.

It sort of like why have both SQL Server and Oracle in the same space. If I need to build a database application, I’d look to the one that does the most for me. I wouldn’t put some tables in Oracle and other in SQL Server. Nor would I necessarily start in say SQL Server and then move them over to Oracle, unless I was looking to warehouse the data. Maybe I answered my own question. Is Documentum a data warehouse for SharePoint data?

This is by far my most 'rambly' posting ever so I've highlighted the actual response in bold at the bottom. Ignore the rest, it is complete self-satisfying drivel.

Let's start with the most obvious question - is Marko calling me Andy because he knows that I hate that or was it just him being overly familiar? (FYI: I am not short of nicknames for Marko if we want to start playing the silly-name-game so bring it on.)

Secondly, my comment was actually that the attendees at the conference viewed SharePoint as a development platform. I know that Microsoft do but I was surprised that every single person with whom I spoke agreed 100%. I loved Marko's analogy of SharePoint and Documentum being like two database systems. He succinctly asked the question that I think a lot of people are pondering right now. Are SharePoint and Documentum just different flavors of the same pie?

I wonder what Marko expects me to say or what he thinks I might want to say if I wasn't under 24/7 Microsoft Taser watch. Would I say that we've been doing it longer, they don't know what they are doing, they are the new kids on the block, they'll be gone in 12 months, Microsoft is just an overgrown startup, SharePoint will never catch on, my ECM system could beat up your ECM system..? The answer is...all of the above!

Seriously though, it is a great question and no longer just some academic, positional or defensive debate. The truth is that the pundits who really understand what is happening in the market are past the "it is a head-to-head" competitive situation argument. The relationship between a traditional ECM solution and SharePoint is much more subtle yet fundamental than that.

Before I start my longwinded analysis I'd like to address a word that I hear banded around with regards to this relationship. "Coopetition" - this is used to describe the situation where two companies are simultaneously competing and cooperating. Firstly, I hate made-up words, (except voluntold - to describe when you are told to be a volunteer), and secondly that's not a true description of the relationship between Microsoft and the traditional ECM vendors. Let there be no doubt - Microsoft are competing head-to-head in some areas of the business but in my ever-so-humble opinion they are competing head-to-head in less areas than they are not. Grammatical elegance notwithstanding, what I mean is: the gaps between SharePoint's capabilities and a traditional ECM solution are larger than the areas of overlap.

Here are a couple of great questions - Why is there such a big gap? Are Microsoft incapable or incompetent? Let me answer the first question, (the guy with the Taser is staring at me right now so I might not address the second question at all.) It has taken Documentum 14 years to build the capabilities of their ECM system. If I was starting from scratch I'd realize that even the behemoth that is Microsoft could not do it all in a reasonable length of time. If I was Microsoft, owner of the desktop, master of the productive application & champion of development tool where would I focus? Hell, I'd make sure that my solution was not just unified or connected to the desktop - I'd make it a core, native part of that environment, I'd close that relationship tighter than a...insert your own euphemism here.

I'm an amateur quantum physicist, (really I am; I'm studying string theory and multidimensional universes right now), and I know that in a parallel universe somewhere this posting slipped back in time and influenced Microsoft to take this approach so kudos to me for this foresight.

If I seem to be rambling on interminably and you are getting bored then it is because I am high right now; 37,000 feet and 5 cups of coffee high to be exact. I'm on a 5 hour flight back from Seattle and if I am stuck in 14C bored you might was well share some of my pain.

So, SharePoint and traditional ECM solutions overlap. They both have a repository, they both provide library services, (view, check out/in, etc.), they both have clients, etc. I'd contend that SharePoint's focus is on integrations in to the Microsoft desktop at the client and integrations in to Microsoft environments at the back end. So where would a conventional ECM solution add value to a deployment of SharePoint?

What about a situation where the content is not being created by an individual who is working from within a Microsoft application? A scanned image, a document created by a CAD system, a piece of transactional content, structured data, report output, bulk ingested content, etc. In many ECM deployments the majority of the content in the system was NOT manually created. In order to manage the ingestion of this type of content you need a mature platform with the ability to handle the subtleties of these complex data types. For example, a single piece of content created by some common Apple applications actually consist of a group of actual binaries - the master file and a set of resource fork files, same goes for many CAD applications. The underlying ECM system needs to know how to maintain these relationships, handle versions and deliver the correct objects in to the right locations for consumption.
How about managing XML-based content. That's a nightmare area that if Microsoft have any sense they'll leave to those of us who thought it didn't look too difficult!
What about long-term archiving? SharePoint is probably not the ideal location for content to reside in for the next 25 years in a dormant state. For that matter what about short-term archiving - if the content is no longer active then why hold it in SharePoint?
Data aggregation. Do I need to say any more, 80% of my Blog entries discuss this.
Breadth of solutions. Microsoft tend not to develop vertical or point solutions; they develop applications, platforms and tools. OK so their partners do this en masse but the ECM vendors tend to do many more of them and in a more unified, proven way. Typically if you want to manage physical content, adhere to HIPPA, enforce retention, manage complex web sites and digital assets from within SharePoint you can either go to five Microsoft partners and get five stand-alone non-inter-operating solutions or go to a single ECM vendor get them all, fully unified with access from the native SharePoint environment.

There are many more examples but the bottom line is...if you view SharePoint as owning collaborative, in-progress, manually created Office-centric data then it looks like a pretty good fit. If you then add traditional ECM to manage heterogeneous, specialized, multi-channel, high-value, long-term, process intensive content then you truly will have the best of both worlds.

The guys from Microsoft might not like the following analogical response to Marko's original analogy but cut me some slack, can you think of a better one? Here's my response: I see the "Oracle vs. SQL Server" comparison as being more like "MS Access vs. Oracle". Microsoft Access is an amazing database, especially if you work in a Microsoft environment and are doing standalone, departmental applications; it does everything a database needs to do but within a limited scope. If you wanted to roll out a long-term, shared, enterprise-strength, secure, scalable solution you'd either go with Oracle/SQL Server or perhaps you might back-end Access with an centralized Oracle/SQL Server system that integrates in to your other enterprise solutions. Obviously, SharePoint is a much larger, more integrated and more extensible solution than Access but as an analogy I do see the comparison as being valid.

Here's how I see it from my ivory tower - SharePoint is a great access point in to the world of real enterprise content management. It created an entry point in to full blown ECM that 100,000,000 consumers can now access from their native working environment.

Let me preempt Marko's next question, isn't it likely that Microsoft will simply grow SharePoint in to a true enterprise content management system? Let's be clear, I am not party to Microsoft's internal long term world dominance strategy, (unless you assume that world dominance is indeed the strategy), but from how I see SharePoint being positioned in the market and how it plays in to Microsoft's core market I'd guess that they'll certainly mature in the enterprise areas but never really address it to the level that a true ECM system would today. Also bear in mind that any savvy ECM vendor is not going to sit still, they will continue to dominate in the areas that SharePoint is not prepared to address. My assumption is that SharePoint will focus more heavily on being pervasive in to Microsoft's core businesses and will turn in to a hybrid of a development platform, pseudo file system and an operating system overlay for corporate usage rather than a heterogeneous ECM system. Why do I think that? Because the former market is much bigger, less high maintenance and more aligned with Microsoft's portfolio.

Comments? Other than expounding the virtues of brevity or decaf coffee.

SharePoint-ECM Reference Architecture 5: Active Unification

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Fri, 14 Mar 2008 16:14:51 GMT

In the previous reference architecture we unified SharePoint and the ECM system at the Web Part layer and provided a limited subset of functionality, specifically passive operations. There's no doubt that this "passive connectivity" provides a huge amount of value, uses a nice, familiar paradigm and can constitute a complete solution in some cases. However, there are many times when you need to be able to act more fully upon the content in the disparate systems rather than just browsing and viewing the documents.

For illustrative purposes, let’s simplify the underlying architecture and then look at some typical problems with performing more active actions on content. Assume that behind our unifying Web Part we have two separate SharePoint sites. Our web part displays a unified view of all of the documents in the “Standard Operating Procedures” folder in one site and all of the documents in the “Corporate SOPs” folder in the second site.

Take a use-case where Standard Operating Procedures in the company need to be routed for approval before they can be published for general consumption. Approval of documents such as these is performed via a workflow process; let's look at this process from an end-user's perspective.

Active Unification Part Use Case

Matilda has two standard operating procedures that need to be updated. She finds the two documents in her unifying Web Part and checks each of them out of their respective SharePoint sites. She marks up the necessary changes in the documents, checks them both back in, re-selects them both and starts the “Route SOP for Approval” workflow.

So what's the problem?

This seems like a particularly simple and highly efficient scenario but it makes some gross assumptions. Assuming that one SOP lives in the first site and the other lives in the second site then Matilda was actually starting one workflow in one system and another workflow in the second system. What if one of the SharePoint sites did not have a workflow called “Route SOP for Approval” implemented? What if it existed but the name of the workflow was subtly different? What if it existed in both locations but Matilda had access to the workflow in one system but not in the other?

The problem is made worse if the underlying architecture consists of different types of systems. What if one SOP was in SharePoint and the other was in Documentum? Do you have single sign on implemented between these systems? Would Matilda even have an account in the Documentum system? You get the idea - even when both systems support the same operations the implementation and invocation of those systems will be significantly different.

In order to resolve this issue the unifying web part has three options:

Expose the lowest common denominator. Only expose operations that can be guaranteed to work across all systems that might contain content displayed in the Web Part; that might just be view properties, view content, check out/in and view version history/audit trail - those operations that I label as being passive. You are certainly going to have to suppress the more complex operations like virtual document management, XML manipulation, digital asset management, multi-channel publishing, retention application, etc.
Assume that the functionality exists in both systems. When I say assume I really mean ensure. Through a process of checks and balances you would have to ensure that the same configuration and functionality exists and is available in all systems in the federated environment. Having a centralized policy management system would be a huge help in this scenario.
Dynamically decide what capabilities to display in real time. As you select the document(s) and invoke the right-click menu the Web Part would have to determine the available functions in real time. For example, if you select two documents that happen to reside in the same repository, you can display any capabilities supported by that repository. If the documents reside in disparate repositories then you might expose the "start workflow" option for ll documents but then only display workflows that exist in all repositories and have exactly the same name.

Don't get me wrong, these options are not without merit, option #1 is actually a fine solution if the end user can do her job with the "dumbed down" functionality. If you run a tight IT operation then #2 might work but if someone points the unifying Web Part to an incorrectly configured repository then it could cause wholesale chaos. Option #3 on paper looks like the preferred solution but it takes a lot of work to build this functionality and a lot of cycles to perform this real-time look up, resolve and display, (aka it will be slow).

Conclusion

Under the right circumstance I believe that this architecture can provide a very attractive solution for well defined problem areas. If you have control of your repositories, a well run IT department and a fairly competent development team then you are probably a fictional organization but in a good position to use this approach.

Next week tune in for the first in the series of data aggregation as an approach...in the spirit of Harry Potter movies I am thinking about making reference architecture #8 in to two blog entries - not to make more money but to ensure artistic integrity - honest.

SharePoint 2008 Conference Report

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Wed, 12 Mar 2008 22:16:39 GMT

I spent last week in Seattle at the SharePoint 2008 Conference. As a opportunity for me to attend sessions it sucked but for a good reason; I spent almost the entire conference talking to customers, partners, analysts, competitors & the Microsoft SharePoint team - it was...what's the word? Illuminating is probably most appropriate followed closely by exhausting.

Let me pick some of my top observations - serious and humorous:

SharePoint: Solution or Platform? Before last week's conference I think that I viewed SharePoint as being a business solution or at least a broad set of capabilities. Admittedly, a solution that needed development work in order to solve a specific business requirement. Almost without exception the attendees at the conference viewed SharePoint as exclusively being a development platform. I am not saying that they don't roll out SharePoint almost out of the box to solve business problems but that's almost an aside. They see SharePoint as being a platform that gives the developer access to a plethora of integration points in to the user's working environment. Bear in mind that the audience was primarily IT not developers so this was not "I am a developer so everything is a platform" situation.

Business Opportunities Outside of Core SharePoint: Next year I am going to buy extra tickets for the conference. Would you believe that there were people lined up at the door who did not have tickets and were hoping somehow to get in on spec. I'm thinking next year they should have a parking lot for tailgating and people hawking tickets. Maybe they could have some up for auction and they could give the monies to charity. Seriously guys, you should do this...or maybe just get a bigger venue.

ECM or Data Center Management? I used a presentation at the conference that was based on my world renown and revered eight reference architectures; it was the first time that I had used this deck. I re-learned a lesson that I should have not had to re-learn - know your audience... Here's the thing, if you are a proponent of enterprise content management then my deck probably made sense but if you live exclusively in the world of SharePoint then you could not care less. That said, take 3 or 4 of the reference architectures and label them "Managing SharePoint Data Center Sprawl" and you'd have more attendees than that Gates chap had at the keynote! Better still, call it "Preventing the need to slow down your SharePoint deployments because off the liability and risk caused by silos in the data center" and you'd get Microsoft's blessing, (marketing might have something to say about the title though.)

SharePoint Might Just Catch On. I am pretty sensitive to things...ask anyone who works for me, Mr. Sensitive they call me. I'd have to say that this SharePoint nonsense might actually be successful. I heard a lot of people call the current 2007 version "raw" and I'd tend to agree but there are a lot of compelling things that SharePoint brings to the table. Some of them still do not make sense to me, (portfolio management as a core capability to be added for example), but perhaps that's just because of where my attention is focused right now.

How Orange is Too Orange? If you attended the conference this is for you, otherwise it will not make sense. I sent my deck to EMC marketing for approval before submitting it and their feed back was "It is a bit orange." I guess that's better that them trying to change the actual content. Rather than addressing this directly I added something orange to each slide; I struggled to find 32 orange things to plaster on to the slides but being a consummate overachiever I managed. I'd like to think that no one left my session with less vitamin C than when they entered!

When in the USA, Only Expect One Person in the Audience to Understand a Father Ted Joke: Whether SharePoint is really an enterprise content management would indeed be an ecumenical matter - Thanks for laughing Mick!

Who reads This Drivel? People from Microsoft do...would you believe that they not only still talk to me, they actually treated me really nicely - they even bought me dinner one night. You'd think they'd know better.

The EMC Microsoft Practice are Excellent: As you know, I tend to take a fairly agnostic approach to the fact that I work for EMC and try not to sound like an extension of the EMC marketing machine; I am of course always 100% loyal to anyone who is paying me. That said, I cannot tell you how many people commented on how great these guys are - it turns out that EMC has over 400 certified Microsoft Professionals in the practice. It is without a sliver of self interest that I expound the skills of these people just 1 week before I start the process of getting them to provide massive amounts of resources to my projects free of charge. J

The Hilton Seattle. By far the worst Hilton I have stayed in ever...IMHO. Avoid it.

I've registered for Tech Ed but will probably not get a chance to re-do the presentation at that event but I'm looking forward to yet another geek-fest!

SharePoint 2008 Conference Invite

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Thu, 28 Feb 2008 22:02:19 GMT

If you are lucky enough to have a ticket to the hottest show in town next week then I invite you to come along and hear me talk about the eight architectures, (it says seven in the abstract because I had to submit it before the eighth one arrived). I'm speaking at 10:30AM on Tuesday March 4 in Room 616-617.

If you want to chat about any of these topics, especially if you have feedback on the architectures then pop over to the EMC booth and they will know how to get hold of me.

I'll be asking lots of questions, validating my thoughts and also seeing what Microsoft and the partner community are up to. I cannot promise that I'll Blog during the conference but if I can find time I certainly will.

SharePoint-ECM Reference Architecture 4: Passive Unification in Web Part

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Wed, 27 Feb 2008 13:13:52 GMT

I represent the previous three architectures as being “unification” however technically they do not actually unify SharePoint with the ECM system. The first architecture simply recognizes the problem, the loosely coupled solution provides movement of content and the Web Parts specific solution just creates a simultaneous view of the two disparate systems.

This fourth architecture provides a level of real unification albeit only at the client level. There are a number of content types that might benefit from this client-level unification depending on the actual functionality needed. For example you might unify data between discussion threads, datasets, messages, inbox entries, etc. However, we will take the most frequently use-case which is the unification of documents between systems. (I'll Blog about those other structured and semi-structured data types later.)

The objective of this architecture is to create an environment where the end user is not aware of where an object actually resides. If the end user needs to see three documents in order for her to do her job then she should see all three documents in a single Web Part; we should abstract their actual physical location from her. If she does need to know where the data resides physically then that should simply be a value in a column – we certainly should not insist that she navigate three separate systems or even three web parts in order to get access to the data.

In this reference architecture, a single SharePoint Web Part is used that is able to federate queries out to many disparate systems. In laymen's terms, the Web Part grabs information about documents in 'x' different systems but shows the results in a single list.

If the end user simply needs to search lists of relevant documents and then view the content then this reference architecture is relatively simple. If the user needs to perform more complex operations then we start to see serious technology issues, (Note that Reference Architecture 5 will address that scenario.)

How does it work?

The SharePoint Web Part contains logic that “binds” it to predetermined data in the disparate systems, for example, it might be mapped to folders 1 & 2 in the ECM system, folder X in one SharePoint system and folder Y in another SharePoint system. When the user invokes the web part it queries each of the systems and returns the results in a single unified results set. Other uses for this architecture include displaying a search results set; in that case it would take a cross-system query, federate the search across systems and then consolidate the results.

Convenience Begets Problems

One point to note as we move through these architectures is that as we increase the functionality we also increase the technical challenges. This will become very apparent in the next architecture but just consider a single subtlety in this one relatively constrained case. Mapping user security...Consider this example; I am logged in to SharePoint, I invoke this Web Part which queries 4 other ECM systems and 2 other SharePoint Document Libraries. I don't mean to be existentialistic but who am I in these other systems? Do I need to have credentials on those other systems? Do they need to be mapped to my home SharePoint account? Can I rely on SSO for this? Who will maintain this? Where will the mapping between systems be held? Should I just act as sysadmin in the remote systems and throw security and data confidentially out the window?

Conclusion

If you can deal with the cross-system issues then this approach starts to show some promise. Users should not care where their data lives - they just want access to it and this approach offers that. Be aware, this approach addresses only the 'passive' use case. I don't mention check-out/in, lifecycle management, workflow, BI, etc... Add those to the equation and...well you'll see in the next reference architecture!

Disaster strikes the seven reference architecture model...

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Fri, 22 Feb 2008 12:56:53 GMT

Dateline Hopewell, NJ. Early today disaster struck the seven reference architecture model as an eighth contender emerged as a late entrant...dozens of Andrew Chapmans will be dispatched to try to change blogs, presentations and reports...

I had a great conversation today with Erin Samuels the EMC SharePoint Product Manager about a customer use case and realized that the proposed solution did not really fit in to the seven reference architectures. Even though it was a great idea, I was keen to bury it just to maintain the little credibility that my seven architectures have but she would not let me. Try as I might, I cannot squeeze this new reference architecture in to the existing seven. I was going to pretend that it was just a derivative of one of them but it is too important to not have standing alone.

The advantages of RefArchs 6 and 7 are that SharePoint is working with native SharePoint objects; this means that you do not need to suppress any of the standard SharePoint functionality. RefArchs 1-5 all potentially compromise the functionality supplied out of the box by SharePoint. What if you could have an architecture that had native objects in SharePoint and also native objects in the ECM system. Ladies and gentlemen I give you "Reference Architecture 8: Synchronized, Intelligent, 2-way Shortcutting..."

Reference Architecture 8: Synchronized, Intelligent, 2-way Shortcutting

Note that this RefArch works for content that already exists in the ECM system, is being created by a 3rd party process or is "published" from SharePoint.

EMC's Real Acquisition Strategy...

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Thu, 21 Feb 2008 20:39:52 GMT

I am not sure that this information is really supposed to be shared with the general public, I overheard it in the exec bathrooms in Hopkinton this week so that makes it pretty "public" domain to me. It turns out that EMC is involved in an evil plot to thwart the future of science. We are buying up all of the cool mathematical and physics-related equations in the world and will then hold them to ransom.

EMC - Obviously we already own Einstein's most famous equation.

Pi - Looks like we are buying the most amazing mathematical constant in the world.

RSA - RSA is actually the same of an algorithm for public-key cryptography.

Rumor has it that we will be buying Binomial Theorem Inc, Quadratic Equation Ltd and the letter 'x'.

Watch this space...

SharePoint-ECM Reference Architecture 3: Use SharePoint as a Portal Container

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Wed, 27 Feb 2008 08:56:20 GMT

In the previous two examples we focused on moving content from the SharePoint document library in to the traditional ECM’s repository behind the scenes. That “publish to ECM” paradigm may work for you but you must consider that once the content has moved in to your traditional ECM system’s repository you will lose sight of it from within SharePoint.

In this third reference architecture a set of SharePoint web parts are created; these connect directly to the traditional ECM system and allow you to view that content from within the SharePoint environment. In a typical implementation, some parts of the SharePoint portal will be displaying native SharePoint content and other parts displaying ECM content. This is a fairly common approach to SharePoint/ECM unification today.

A user using this architecture can see content in a variety of systems from within a single web screen. As you can imagine, this is a very attractive proposition and can be an excellent solution in some cases. This is the classic client-level unification model where all of the unification process is delivered at the SharePoint portal framework level.

Where this architecture fails is in the area interoperability between systems. Although you are able to view content from disparate systems from a single interface there is no actual unification of content or processes. For example, you cannot drag an object from the web part that points to the ECM system and drop it in to the SharePoint web part.

The ECM content that you view does not have to have been published from SharePoint, it could be existing content managed by a different application. Equally, you could view other features of the ECM system: workflow tasks, inbox items, discussion threads, etc.

If you combine this reference architecture with the “Loosely Coupled Solution” architecture you do get the beginnings of a usable solution. You can publish a piece of content from SharePoint in to the ECM system and to the end user is simply moves from one web part to another.

Conclusion

This solution can be an extremely effective, efficient and low risk approach and should be considered a practical solution. It is especially useful to expose ECM functionality and content to "occasional users" who typically live in SharePoint - people who rarely use the ECM systems. However, it really comes in to its own when it is coupled with some of the other architectures. After looking at all eight of these architectures I will show how they can be combined in to real life solutions.

Archiving vs. Backup

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Mon, 18 Feb 2008 12:09:39 GMT

We brandish around the term archive without much thought do what it means. I was reading some internal EMC literature the other day and saw a great slide that outlined the difference between an archive and a backup. It made me realize that sometimes people use these terms interchangeably when in actual fact very different. It was always my assertion that it was only the intent rather than the physical process that different between a backup and archive but the differences are more complex than that.

Rather than belaboring the points here is a table that outlines the key differences to train a backup and archive. I found it useful and interesting but then I am spending president's day writing Blog entries so maybe I just need to get out more often!

Backup	Archive
Secondary copy of information	Primary copy of information
Used for recovery operations	Available for information retrieval
Improves availability by enabling application to be restored to a specific point in time	Adds operational efficiencies by moving fixed/unstructured data out of the operational environment
Typically short-term (weeks or months)	Typically long-term (months, years, even decades)
Data overwritten on periodic basis (monthly)	Data retained for analysis or compliance
Not useful for compliance	Useful for compliance

SharePoint-ECM Reference Architecture 2: Loosely Coupled Solution.

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Sun, 24 Feb 2008 21:51:24 GMT

In the second of the eight architectures I take the general premise from the previous example and extend it. The content still lives natively in SharePoint while you are collaborating on it and is then "migrated" in to the traditional ECM system at a predetermined point in time.

As a matter of interest according to recent research, for most customers this movement of content from SharePoint to an ECM systems occurs because of one of the following reasons. The document:

Becomes a formal business record.
Is being collected as part of a legal discovery process.
Needs to have extra protection applied, the application of digital rights management for example.
Is being archived for operational reasons, e.g. the document library is no longer operational and can be archived in its entirety.
Should be made available for re-use and re-purposing; perhaps being included in a formal structured business process.
Will be published to web site or an external multi-channel device.

In this reference architecture the migration is done using an automated process which “publishes” the content from SharePoint in to the traditional ECM system. Fundamentally, the automated process is simply extracting the content from SharePoint, caching it locally and then uploading it in to your ECM system but the fact that the migration is controlled by a process not a person allows you to more easily perform some auxiliary tasks. For example, when the content is moved across from SharePoint it is possible to bundle up some of the object’s history, metadata and related content and to move that across too. For example,

The original location of the file in SharePoint.
The audit trail information about the object.
The security access settings.
Some of the metadata, (creation date, original author, version number, project name, signatory information, etc.)
All versions and/or renditions of the document.

Having this information derived from SharePoint allows you to do the following:

Identify that the content in the ECM system was originally imported from a specific SharePoint instance.
Identify that a piece of content in the SharePoint document library has been archived at some point and where it was archived to.
Retain the entire version or rendition history of an archived item.
Maintain the security permissions model between systems.

In fact, you can pretty much bring anything across from SharePoint to the ECM system so long as you have a way of extracting it from SharePoint and then storing it in the ECM system. You might even collate information from other sources during the archive process and include that in the archived content.

Unfortunately, the most common approaches to this architecture will typically implement a "once only" and also "a one-way street" model. Content is moved over to the ECM system and the publish process is then finished. The publishing is a one-off process triggered by a change in state of the document or an external event. In some scenarios this is not a problem, for example when publishing to a records/retention archive or for eDiscovery collection when the content published to the archive will not be changed. In order to avoid this limitation you would have to have agents running on both the SharePoint and ECM systems that would monitor changes to the source or target objects and would be able to reconcile changes to either the source document or the archived copies...not a trivial task.

Conclusion

Again, this very loosely coupled model leaves a lot to be desired – the good news is that you end up with content in the most appropriate place at the most appropriate time but it is typically implemented as a one-off process. Even if you have a post-migration synchronization process in place you are really just moving content from SharePoint in to the ECM system...there has to be a better solution...doesn't there? Actually, this architecture comes in to its own when you combine it with some of the architectures that follow. After I've described all seven architectures I will focus on how to start combining them to get real efficiencies...the suspense is killing you - admit it.

One other consideration that you might notice is that you are potentially storing each piece of content multiple times which is a disk space and compliance liability.

SharePoint Archiving Assumptions...

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Fri, 15 Feb 2008 08:41:24 GMT

One of the great things about blogging is that you have to think hard about what you are writing, literally tens of thousands of people might read your blog...granted I am lucky if half of my immediate family read this drivel but conceptually you get the point. Realizing that once you've written something you cannot retract it really does make you think twice about what you say.

Engaging my brain before I speak is not my natural state, just in case you wondered so this is quite a challenge for me.

Point in case - I'm sat on a flight back home typing the "SharePoint-ECM Reference Architecture 2: Loosely Coupled Solution" entry and I realize that I've forgotten to mention one key point in my SharePoint-related ramblings...I am focused on the archiving and de-duplication primarily of unstructured content...PDF files, Word documents, etc. I am neglecting the oh so important structured data types that SharePoint manages - discussion threads, calendar entries, etc. There is no doubt that these structured data types - especially the former example - are critical components of a compliant archiving solution.

Given this I'll do two things,

Firstly I'll be explicit when I am referring specifically to archiving, unification and aggregation of unstructured content vs. unstructured content
Secondly, I'll talk to the people from whom I steal all of my ideas and trick them in to telling me how to deal with archiving structured data. I'll then post some entries related specifically to that topic.

I'm pretty sure that the core compliance issues are fundamentally the same no matter the data type but as usual, the devil is in the details...

Document Collaboration - A New Approach?

andrew@nevertalkwhenyoucannod.com (Andrew Chapman) — Fri, 08 Feb 2008 12:41:38 GMT

I was reading one of Craig Randall's recent posts about creating new market opportunities and it got me thinking about how we collaborate around documents today and whether it really is optimal.

Back in the day when I was a UNIX programmer I worked with someone whose theory was that some developers were really good at writing the first 80% of the code and some were really good at the last 20%. We all took this to mean that he was just not a very good programmer...but maybe he was on to something.

Currently, when we collaborate on documents we tend to do the following:

Each person completely finishes a section - they write chapters 4-6 of the technical manual while someone else writes Chapters 1-3 and 7-10. The pieces are then combined in to the finished document.
You write a document and then route it for comments, incorporate the comments and then rinse and repeat until the document is complete.
One person or team writes the entire document and then it is routed off for approval, processing, multi-channel publishing, printing, etc.

The point is that we don't collaborate until we have finished something to the best of our ability. We tend not to send something half finished off for someone else to finish - or even finesse.

So, back to Carl's 80/20 model for developing...how might it apply to writing documents? I am writing a book; I spend a lot of time reading, researching, formulating ideas and then I write them down very quickly - I "brain dump" to paper all of the intellectual capital that spews from my brain. I then probably spend 5x the time building a logical framework around the thoughts, making them flow, cross referencing & validating them, etc. This is a huge task but not something an editor could do. However, if I could afford a team to assist me I could have a process like this:

I brain dump my ideas to the page.
A subject-matter-savvy person then reads, reviews and validates my thoughts. They then re-organize my writings to give them flow and logic.
I review that the concepts have not been changed
An editor then edits the document and formats it to suit.
I review and approve.
The publisher prints the books

Is this really that different to how we collaborate today? As an author it is because there are certain tasks that only I can do but there are other tasks that I could outsource.

Having finished this fairly uninteresting rant I realize that a ghost writer effectively does this today...maybe what I am looking for is a technology that supports that process. I have tried software that helps to gather research for book writing but they are a far cry from mature enough to support this type of process.

So, Craig...if I write the first 80% of a solution, will you finish it for me?