Confessions of a Former Integration Hand Coder
I recently read Ventana Research’s white paper on Issues with Hand-Coding Data Integration — Avoiding the Perils of Custom Code and it got me thinking. As a solutions engineer, I spend a good part of my day showing prospects and customers the benefits of using a commercial integration platform like SnapLogic. But I must confess that as recently as two years ago while working at an enterprise software company I myself decided to hand code an integration.
It sounded so simple at first. Just two endpoints. The Director of HR requested a simple one-way integration that would load hired candidates from our cloud-based recruiting system in our Human Capital Management (HCM) system. Sure, there were some data gaps in end point business processes, but nothing too crazy. For example, one system used the label “Candidate” and the other “Employee.” Both systems had well developed APIs and the documentation was solid.
I should have known better. As with most integration projects, it got complicated fast. For example, how do you get positions into the recruiting system in the first place? Ok, now it’s a two way integration and the number of data mappings just tripled. Other questions began popping up around on-going administration. From which end point does the integration administrator manage the schedule of data flows? Does he view these flows in recruiting or the HCM? Or maybe there should be separate views in each?
As we went through the recruiting to HCM integration, additional integration opportunities presented themselves to the Director of HR. “Jim, that’s great, but wouldn’t it be cool to also pull in budgeting and market survey data into the requisition?” By the time I left that particular software company, that simple one way integration between two endpoints had grown to encompass multiple systems and business processes and there was no end in sight.
I learned some good lessons on that project. Fact is, despite an integration project being presented as a one-off solution, that’s rarely the case. Business users want (and deserve) rich data input, rich data monitoring tools, and a centralized user interface. New end-points get added. And enhancement requests can’t take advantage of the original coding. Peer software developers aren’t familiar with project-specific coding languages and libraries. All of these considerations fall outside of the original specs and cause huge delays.
I’m fully reformed now. The smart alternative to hand-coding is a commercial data integration platform that utilizes a hub-and-spoke model like SnapLogic. Instead of connecting each individual data source to all others directly, each source connects to the central hub. For example, in a hub-and-spoke model (as shown in the graphic above), changing one system of the five would require only one change instead of four separate sets of changes. This model requires fewer connections, enabling organizations to add, remove or change data sources with much less effort. SnapLogic also provides a standard methodology for data integration processes so it’s easier to maintain these connections even if it’s done by those who weren’t involved in creating the original connection. Finally, SnapLogic provides a slick, browser-based drag and drop UI that enables collaboration between IT and business, as well as an iPad Dashboard to monitor the health of your entire integration ecosystem.
How about you? What’s been you’re experience with hand-coding integrations vs. commercial tools?