as forgetful, distracted humans we often find ourselves realizing we’ve lost something. but what about those things you loose but don’t realize you’ve lost? before we descend into an overly reflective view on that topic, let me drive at a more secular example: data loss.

today it was again reported that another company mislaid customer information and had paid a heavy price for it. the hope is that other companies learn from the steady stream of such mistakes and avoid similar penalties. but what puts this particular case into a whole new category is that the loss of data went unknown and undetected for a year. this is the hidden menace of data loss. it can be bad enough to mislay critical or sensitive customer data,  but it can actually be very hard to detect that you’ve done it. that time lag increases the embarrassment and also widens the window of opportunity for bad things to happen.

like the car keys, things rarely go wrong when the keys are in one place – hanging on the hook, or resting in the car’s ignition. it’s when those keys are moving around, especially being exchanged from one driver to the next, or (in my darling wife’s case) from one handbag to the next, that things go wrong. the same is true of data. while there is ample need to protect and defend data resting in a database or file system, the data loss examples i’ve seen that have caused the biggest problems usually occurred while the data was in transit, such as the routine transfer to a remote backup system, as was reported in today’s story.

managed file transfer is designed to address both sides of this issue – first the actual leakage of data, by leveraging secure channels for transfer such as SSL, and encryption/authentication at either end. second – the ability to track what has been moved, and where it’s gone to, making it easy to know when something has gone somewhere it shouldn’t.

we’re busy readying a future update of our managed file transfer solution and working with clients around the globe on helping them reduce the chance of becoming the next headline. helping to make these stories a thing of the past is the mission we’re driving. and for our clients it’s more than a matter of avoiding fines and headlines – it’s a “survival instinct”. because winning your customers’ trust is hard to do and as easy to loose as a bunch of keys.

my first job interview was for an IT company based near Basingstoke. as soon as the interview started i detected something wrong – a distinct lack of respect.  as my interviewer introduced himself and his company it became immediately clear he had little respect for his customers, even less respect for his female and junior employe es, and no respect at all for people of certain races. in addition he showed no respect for how other’s might view the nature of their business – which although entirely legal, raised for me serious ethical questions.

given i knew nothing of this in walking into the interview, but was able to sense it all in under 5 minutes shows just how ingrained and unashamed my interviewer’s lack of respect was. at an appropriate pause i drained my coffee cup, stood up, thanked him for the beverage, and told him that to continue this interview would be waste of my time and his. as i left the room, i vowed never to work for a company that failed to show respect, both for its employees collectively and individually and for its clients.

as an ibmer, i’ve seen many changes within the company over the last 10 years. i’ve seen restructures and policies adjust as the markets shift. i’ve even struggled to remember the names of some of the managers i’ve had for brief spells, but even after a decade of job changes and office moves, one thing has remained constant: ibm’s respect for my and my colleagues, and our respect for our clients.

so it’s great to see culture of respect we deem so precious rewarded. everyday i see that reward in the goodwill our clients extent to us, and in the critical aspects of their enterprises that they entrust to the software we develop.

last monday, i got another reminder of the value of respect, in the form of Barron’s list of the The World’s Most Respected Companies which can be accessed here and read about in Smart Money here. i’m proud to see ibm listed 4th, because i know how hard my colleagues and i work to earn our clients’ trust.

Barron’s list prompts the question: can you actually turn respect for your company into dollars for your enterprise? the answer, of course, is yes. but respect isn’t a corporate asset. it can’t be traded. it can’t be accelerated by aggressive programs. it starts with a belief that – whether it affects your bottom line or not – it’s the right and only way to conduct business. its requires everyone to recognize that long term relationships are built on trust and it needs to weave itself – not just into the external interactions with partners or suppliers – but into the internal interactions between your employees. there’s simply no place within the company for any who lack that respect.

was i rewarded for walking out of that interview? a fellow student, friend of mine, didn’t come away with the same impression as me. he interviewed right after and accepted the job. he phoned them just days before he was due to start, surprised that no-one had given him any details about his first day. he was told that the company had lost the major contract which we’d been told was “in the bag” and therefore had no job to give him(!) clearly, this lack of respect extended to new hires too. while he did find a great job eventually, it wasn’t easy.

my reward? my next interview was at ibm. i could sense the difference immediately, and now, 10 years later, i espouse the value of respect to anyone who’ll listen, here from my desk in hursley.

as you can see, the user interface just rocks. frankly though, isn’t it about time the way we interface with IT moved on? the two-dimensional pointing device we call a mouse is around 30 years old already. the keyboard with its QWERTY layout dates back to the 19^th century. and more recently motion-sensing controllers like the Wii Remote have revolutionized the way people play games (and the people who want to play them!)

when i watch the vision of the future of computing portrayed in Minority Report though, something isn’t quite right. something about it doesn’t seem so cool and advanced and futuristic. infact something seems rather silly, old-fashioned and lame. it’s not the cool UI that’s bugging me. if you check out the youtube vid i referred to, you’ll see it 44 seconds in. it’s the bit where a character removes a transparent card shaped like a SIM-card from one machine and inserts it into another.

presumably these cards represent some futuristic form of data storage akin to a USB memory key. see-through memory keys? cool! two futuristic computers that can’t automatically exchange data files without requiring a human to move the data between them? not cool! is this the way we’ll be sharing data between IT systems in 2054? i sincerely hope not.

the computing interface in this movie is seen as revolutionary and many pioneering innovators are desperately trying to make it a reality (Oblong is one of those). but what about the future of integration between those IT systems? should we aspire to a future where we need to manually eject and re-insert transparent discs between computers then? no. clearly not. this primitive form of integration has been called “swivel chair” integration, refering to the way that humans have to swivel back and forth between machines, rekeying data, and has been universally derided as error-prone and archaic since at least the late 1980s.

so why on earth should “swivel chair” integration appear in a vision of the future over 40 years from now? perhaps there’s a good reason. so much manual, swivel chair style integration is still going on today decades after better solutions had emerged and even amongst the most technologically advanced and progressive organizations. it’s not the minority. today archaic, unreliable forms of data movement between IT systems makes up the Majority Report.

the proliferation of USB memory keys is just one example. how often have you had to resort to this method, even though it can be inconvenient and quite impossible to secure and audit? how easily could those transparent memory cards from Minority Report go missing or get duplicated even though these contain very sensitive information about past (and future) criminals? like Cruise’s character, John Anderton, we now live in the world of “pre-crime”. auditors that want to be convinced that our organizations won’t break compliance in the future, in addition to checking we haven’t in the past. how we move data between IT systems is a crucial part of that analysis.

if Minority Report showed us the future of the computer user interface that we now aspire to, what is the future of integration, of file transfer between systems? well, i’ve seen it, and it’s not an illusion created by Hollywood’s green screens. it’s real and it’s emerged from only slightly less glamorous origins – IBM’s Hursley Labs in Winchester, UK.

the future is managed file transfer. and we’re helping companies around the world realize their aspirations right now. meanwhile it’s time for me to slip on some gloves, connect a bank of flat screens up to my Thinkpad and pretend i’m fighting pre-crime from my desk in hursley.

i quickly began to realise how much more effective these sessions could be – both for me and for my clients – when i gave the powerpoint a rest, and stopped doing all the talking, started asking questions instead and got my ears really working. to start with, the folks that organize these sessions looked at me oddly, as though i’d forgotten to prepare a slick pitch whose success was gauged by how few questions it evoked at the end. but i’ve found you can’t start presenting what you have effectively, until you know who your client is, where they are now, and where they want to go. over the past few years, i think we’ve all come to appreciate that two-way discussion beats one-way instruction.

today, i had a new experience in terms of presenting to clients. and it’s helped me do something very important when explaining the value of messaging and connectivity software – presenting via a translator. i came away convinced that anyone in product management, marketing and sales should seek opportunities to present to people who do not speak their own language. it’s all too easy to rely on 45-50 slides to convey the value, benefits and features of your offerings. why? important decisions in life can be made without so many words. and most communication is non-verbal anyway. with a bit of effort, and with a bit of distance between you and the medium of powerpoint, you can create “big animal pictures” that convey more thoughts than 20 densely bullet-pointed lists. Case in point here, is that most interest and questions in this case were generated by my “big animal pictures” about the new high availability features of WebSphere MQ V7.0.1 and another on the role messaging can play in enabling a Smarter Planet.

but most importantly, presenting with translation forces you to say less and to say it slower. rambling is out. asides are out. to be effective you have to condense your entire value proposition into two or three sentances. cutting out of the fluff like this is refreshing. and not just for the clients who don’t speak your language. presenting in this more thoughtful, concise manner means your message is no longer implied. it’s not in the slide somewhere. your message has to be lucid, direct and explicit.

if you present at all, i seriously recommend setting yourself the challenge of finding an opportunity to present through a translator to a team who don’t speak your language at all. you’ll learn a lot from just preparing with the translator anyway about how to simplify your messages and get to the point. (don’t present without  preparing with your translator!) i promise you’ll learn a lot about streamlining, condensing, communicating your messages. and if you come away from the experience as i did, you’ll be thinking seriously about honing the presentation you give to your English-speaking clients too. meanwhile i look forward to my next opportunity to present via a translator again, as i start trimming my presentations down to the raw, core message – here at my desk in hursley.

but i’d like to argue for another root cause. a different ultimate reason to the road chaos. it might seem too obvious to be profound but here goes: the rush hour. the simple fact that in that short window of time, everyone had simultaneously brought their cars onto the road to fight to get to work. stand back for a moment and think about it. we synchronize our daily commutes for maximum disruption. we batch up all the day’s traffic to try and execute it all in the same small slice of time. is there a solution? my usual habit is to begin my working day early, checking email, planning the calendar, determining whether i’ll be more effective getting things done at my desk in hursley or at my home office. as a result, when i do commute into the office my day- and journey – starts a little later than most. this does mean i park further away from my desk than i otherwise might, but on the other hand my commute is stressless and traffic-free and above all efficient. in short, i’ve learned how to un-batch myself from the rest of the traffic and improve my efficiency.

can IT learn a lesson here? does it needlessly batch data processing, unnecessarily synchronising network traffic and processing? yes. i see it all the time. in every size of company, every industry, every geo, every stage of IT sophistication. batch transfers. overnight updates. regular, daily tasks. all coordinated simultaneously so that one little hiccup causes gridlocked chaos. one large financial company operates this way even today, batching up its over night updates to transfer from each region to head-office. it has a window of, say, 6 hours to transport all these batch updates so that come the morning all the head-office systems are synchronized and up-to-date. the batch takes 4. a hiccup (on the network for example) that can’t be detected and resolved less than 2 hrs into the batch transfer is not going to finish in time and will cause major headaches when the next day begins. [incidentally, the process for detecting a problem in this case involves an application whose algorithm for sensing a fault is dubious and inconclusive and that is intended to alert the night security personnel(!) hardly a bullet-proof solution].

but the main issue i want to focus on here isn’t the reliability – it’s the scheduling. businesses need to get away from the batch mindset. it’s synchronising things for maximum fragility and minimum efficiency. “overnight” doesn’t exist in most industries anymore. doing things in little batches – micro batches if you will – rather than in one big job, must be the way to go. spread the tasks across the day and night. stream updates continuously. turn the tidal wave into a swell. the flood into the flow. the tsumani into the constant trickle. take that big batch job – and in the words of MC Hammer – “break it down”.

infact, even this blog entry is another example of micro-batches yielding greater efficiencies. if i’d sat down to write this in one go, i would have had to carve out a lump of interruption free time. but i’ve discovered that never happens. instead i write my entries in micro-batches. a few lines at a time. so this entry really made no major hole in my week, just snatches and snippets here and there. just a thought, if you’re struggling to find enough time to craft entries, just as i did when i began to blog from my desk in hursley.

but back to plumbing for a second. i always think of WebSphere MQ, our family of messaging transports, as plumbing. a conduit for business data, just as plumbing routes and delivers a constant flow of valuable resources on tap. for us, minor leaks have never been acceptable. business data today is just too valuable to loose. whether it’s a customer’s order which if lost will turn them to your competitors, or it’s an interaction with a partner which will eat your profits if mishandled, or sensitive personal information that will embarrass the company and hit your stock or even land you in the stocks. obviously a key leak can come from data theft, and we provide security mechanisms in the pipe (in transit) like Secure Sockets Layer, and also for where the pipes connect to systems (at the application layer) like our Extended Security Edition.  recent surveys suggest that these malicious thefts of information are far less frequent than alarmist reports suggest, and that actually most data losses are accidental and as such are probably not detected quickly if at all. this is the way of loosing business data that my gas supply issues most remind me of. data corruption. it can be harder to detect situations where the information wasn’t hacked or stolen, but simply got mangled en route so that it’s unusable or simply disappears. either way your data stinks, like the whiff of methane, or dissappears altogether without a trace like a gas. little factoid for you: domestic gas is naturally odorless. that familiar whiff of gas comes from smells added artificially by gas suppliers, based on chemicals like Methanethiol. given the volumes of gas consumed by homes across the world, could it be claimed that the most popular perfume fragrance in the world is the aroma of rotten cabbages?

so, back to connectivity software… can we learn lessons from the gas industry? can we add something that makes it clear when information is leaking? perhaps checksums performed before and after sending information fits the analogy. if the checksums don’t match, then the data smells funny as is discarded or re-transmitted. certainly for this reason products like WebSphere MQ File Transfer Edition provide checksums to help ensure file data wasn’t corrupted. meanwhile, we focus on making the transport layer for connectivity as robust as possible to prevent leaks in the first place using techniques like queuing and two-phase commit to make mis-transfers invisible, automatically resend information and make sure it’s always written to persistent storage in a way that can be recovered if the systems fail.

however, the critical part of the gas industry solution isn’t the smell in the supply, it’s the nose on the consumer. so how do we provide IT professionals with “noses” for their connectivity? it’s got to start with governance and visibility. governance over what happens and visibility into what’s happening, and specifically what data went where and when, and – because IT can change more frequently than copper or lead pipework – what the connectivity looked like when it happened.

this is the motivation behind the audit trail built into WebSphere MQ File Transfer Edition and behind applying governance and visibility to file infrastructure. (i”ll blog about what we’re doing here in detail soon).

so, can you detect a whiff of data leaks in your IT? if you can’t, perhaps you’ve accomplished something truly incredible in your IT. but in that case, it’s more likely your sense of smell needs checking. my advice? get a bigger nose. apply governance to your file transfers, your connectivity, your SOA.

meanwhile, it’s time for me to give my plumber a call from my desk in hursley.

actually today (friday) i’m taking a day off to rejuvenate and i will be giving the computer a rest too. so i’ve taken advantage of this clever feature in blogger where you can schedule a new post for a future date. so if the above link to the announcement hasn’t quite gone live please try again later.

anyhoo – we’ve cram-packed the V7.0.2 update with lots of good stuff!

for example, we’ve introduced a new type of agent (called a Bridging Agent, although i confess i was also keen on Special Agent) that can “talk” protocols other than WebSphere MQ. in other words can send files over alternative transports to MQ. transports we’re supporting in this update are FTP and S-FTP. why on earth would anyone want to move files over anything but MQ? well – the objective here is to enable co-existance with existing environments such as home-grown solutions based on those protocols as well as support situations where the sender or receiver simply can’t run MQ. because the protocol is handled by a Bridging Agent it’s fully integrated into the graphical, command line and XML scripting interfaces. most importantly the Bridging Agents feeds all FTP and SFTP status into the single, centralized audit trail making it easy to track all the file movements regardless of transport. although FTP and SFTP aren’t as robust as MQ, we provide check-point re-start to improve the reliability across FTP & SFTP too.

if you read Gartner’s comments in their latest Magic Quadrant for Managed File Transfer (G00170848, 18 Sept 2009) you’ll have noticed that Frank makes mention that IBM needs to expand its product beyond the MQ protocol alone and alludes to IBM having something up its sleeve in this regard. well, this was it. FTP and SFTP support via these new Bridging Agents.

the protocol support provided by the Bridging Agents are client-side only and are designed to complement the extensive Server-side protocol support provided by IBM B2B gateway offerings like WebSphere Partner Gateway and DataPower XB60 B2B Appliance, in addition to partner onboarding and management.

speaking of IBM DataPower Appliances, in V7.0.2 we’re also supplying documented and tested configurations for integrating WebSphere MQ File Transfer Edition with DataPower Appliances. This enables file transfers to be sent across WebSphere MQ File Transfer Edition networks and on to trading partners via the DataPower XB60 B2B gateway across a range of protocols like AS2. this great article on developerWorks from my colleagues in Italy describes how to take advantage of DataPower as a file gateway in the DMZ and link it to WebSphere MQ File Transfer Edition to handle the internal traffic.

V7.0.2 also bolsters the security support with finer grain access control to agent resources at the user and group levels. this extends the “sandboxing” feature we have to limit agent’s access to the local file directories. we’ve also found the time to extend the platform coverage to include iSeries, HP-UX 11.11 on PA-RISC and more. also, to help our clients move to production faster we’re providing a scriptable interface for creating the file transfer artefacts and enable the configurations to be stored. this can help automate the transition from pilot to production.

we’re excited about this V7.0.2 update and i’m keen to know what you think. please let me know. and i’ll bring you more news, as ever, from my desk in hursley.

peering down from the clouds, the familiar world takes on a whole new shape. tying this back to my day job, makes me realise how much our clients would benefit from a bird’s eye view of their IT systems so that they can step back and take stock of what’s going on. in our busy lives and jobs that might seem an expensive step to take, but it may help put IT priorities into perspective. and that could be priceless. in particular i want to take a look at how we can do this for WebSphere MQ and our other SOA Connectivity products. to see how we can provide IT Architects with an abstracted high-level view of their connectivity solutions, see how everything’s interrelated, where the cross-dependencies are between applications, services and middleware artefacts like queues, channels, message flows etc. if you’re one of our clients and you’d be interested in exploring this with us, please drop me a line. together we can help get some more perspective on connectivity. and while it’s breathtaking to be up at four and a half thousand feet, it’s nice to be back on terra firma, safe and sound, at my desk in hursley.