Multishoring’s 10 Step System Integration Process – Why integration methodology matters!

The system integration process steps you follow determine the quality of the integration project deliverable. If you don’t actively consider and evaluate the need for all the potential project elements, you are likely to fail or deliver a less than optimal integration solution. Our experience helps you to avoid common pitfalls and learn from good- and best- practices we have observed in the field.

Defining the process of integration helps to mitigate risk and assure quality

Not every integration project requires all of the activities described below. However, every systems integration plan or scenario should be assessed to determine which of the following integration tasks and activities are required to deliver an integration solution of the desired quality within the operational constraints under which the integration is being developed.

Knowing what you have deliberately left out of the scope of the software integration process is essential if you are to fully understand the likely project risks as well as the operational residual risk profile of the delivered integration solution.

Integration process step 1 – Requirements capture and analysis

Knowing what’s needed is the first step towards delivering it. It’s common for people to underestimate the scale of the task they face and underspecify their requirements. This invariably causes problems down the line and causes them to incur additional costs and pushes timelines out.

• Use case and integration scenario definition

Identifying the data flows, process dependencies and the touchpoints between the various systems involved is critical. Even the most complex integration scenarios are composed of relatively straight forward building blocks.

• Target systems assessment and interface discovery

Understanding what is viable and the various options available for all of the integration points is essential. Having the full picture allows you to weigh up the pros and cons of each potential approach in order to determine the optimal solution.

• Data mapping exercise and trust hierarchy definition

Modelling the composite data structures and determining which data source populates which entities helps define which systems may be relied upon to give the best possible single version of the “truth”.

• Design envelope definition

Declaring what is expected and desirable helps ensure that the delivered integration performs as required in all situations irrespective of what the world throws at it.

• Performance parameters

The integration should perform in a manner so as not to impact the user experience or prevent the system from delivering business value.

• Anticipated volumetric throughput

Understanding the likely transactional flow and how that flow fluctuates over time enables you to scope the integration properly and avoid volume related performance degradations.

• Security and traceability requirements

Depending on the criticality and sensitivity of the data being shared and manipulated via the integration it will be necessary to implement appropriate controls to assure it’s secure and all interactions with it are traceable.

• Pre-existing integration infrastructure discovery

Leveraging the investments of the past to drive the improvements of the future makes sense. There’s no point reinventing the wheel if there is a perfectly serviceable ecosystem of integration tooling available.

Integration process step 2 – Strategy definition

Defining the desired end state, or at least an interim target state, helps to focus resources and activity. Considering what is and what could be, enables you to make what will be more certain and achievable.

• Current integration ecosystem review

Most organizations have lots of tools. Too many usually. Many are underutilized. Some are misused due to a lack of understanding as to what they can do. Some are not used as much, or as well, as they could be.

• Skills and capabilities assessment

Assessing the current capabilities and capacity of the team as well as their proficiency with the tooling / platforms available to them helps ensure your people are utilized to best effect. Skills gaps are identified, and appropriate training plans developed to ensure your people can support the integrations you need.

• Current contractual commitments and previous investment review

Understanding the terms and conditions associated with the legacy tooling landscape is important. Just because you have access to a platform, it doesn’t mean that using it will be cost effective. Evaluating the current licensing models and evaluating the projected costs helps keep budgets under control.

• Workshops and policy definition

Thrashing out the various options and debating the relative merits of different courses of action helps to drive better decisions. Policies need to be accepted by all parties involved if they are to be adopted and implemented fully.

Integration process step 3 – Evaluation of integration options

There are always multiple ways to approach any given integration requirement. Selecting a model or method that works is relatively simple. Selecting the optimal approach that delivers what the business needs within the defined constraints at the lowest cost and risk is what really makes the difference.

• Reference architecture reviews and comparison

We have years of experience developing and deploying integrations that satisfy our clients’ business, technical and operational requirements. By evaluating the various models and reference architectures that are viable, we help clients to select the integration approach that not only works but is the best fit for their constraints.

• Relative merit identification and scoring

Using a variety of objective criteria such as risk, operational overhead, supportability, middleware costs etc. we help to ensure that all of the potential integration options are evaluated dispassionately to eliminate familiarity bias and make sure the most appropriate solution is selected.

• Cost, risk and level of effort estimations

Using our extensive experience, we prepare detailed estimates of the level of effort and associated costings for all of the integration approaches being considered. This allows the business to have foresight into the full lifecycle investment needed and avoids budget overspend.

• Business case development and presentation

We help technologists explain the rationale behind the recommended approach in a way that can be consumed by the business. By translating the technical characteristics of the proposed solution into business focused traits such as risk mitigation and future-proofing we help everyone to understand the consequences of the decisions they make.

Integration process step 4 – Open issue identification and resolution

Throughout any development project there are always issues that are identified that need to be addressed. We don’t shy away from challenges, nor do we bury our heads in the sand and pretend they’re not real. We diligently work through open issues to get to a conclusion that is satisfactory to all stakeholders.

• Development of technical risk register

The first step to minimizing and mitigating risk is to understand what risks are present and what controls are necessary to prevent them from causing problems during the project. Maintaining a risk register allows everyone to see what the current issues are and stops nasty surprises.

• Definition of potential controls / mitigating steps

Risk is inevitable. Having that risk manifest into a project impacting issue or incident isn’t. The deployment of controls and countermeasures helps to make it less likely that a risk will cause problems. Sometimes the risk is controlled to an acceptable level and sometimes it is eliminated altogether.

• Test case development / Proof of Concept definition

The only way to be certain that issues are resolved is to test the fixes, workarounds or contingencies developed to deal with them. Too many providers simply cross their fingers and hope that the problem doesn’t come to light. Hope is not a strategy. We make sure that issues are dealt with and do not impact project delivery.

• PoC delivery and results evaluation

Sometimes it is necessary to prove a proposed course of action before committing to it for the final deployed solution. In such cases we will provide full transparency into the testing we are performing so that you can be confident in the outputs you receive.

Integration process step 5 – Scope of works definition

A clear, concise and comprehensive description of the activities to be performed and the associated deliverables instils confidence and helps you to explain to your colleagues why working with a specialist integration provider is the best option.

• High level project plan development / validation

Breaking complex problems into a seemingly simple series of tasks is the mark of any professional. We make the impossible appear “easy” by nature of the fact that we’ve been there and done it many times before.

• Formalize level of effort estimates and costing

Experience and the pragmatism that comes with it is what helps us to accurately estimate the resources needed to deliver your integration. We pride ourselves on our ability to plan and deliver to the plan, meaning that you don’t have to go back to the business for additional funding mid-project.

• Select contract mechanism in conjunction with you

Time and materials? Fixed fee with milestones? Retainer model with a consumption based uplift? How we contract for the project depends on how our clients are able to finance the development and their willingness and ability to contribute to, participate in, the development and deployment process. Sometimes we augment their internal team, sometimes we take the lead and manage the integration end-to-end.

• Determine contingency and develop risk mitigation plans

We must always expect the unexpected. Preparing for contingencies and potential issues make them less scary and reduces the impact on project schedules and budgets.

• Define tasks, deliverables and associated outputs

We specialize in “right-sourcing”. We develop the software integration plan and assign the right resources to perform the right tasks irrespective of where they are located. Getting it done right and getting it done right the first time is what drives us.

• Success criteria definition

Every stage of the project has clearly defined outputs and objectives that build towards the end goal. By hitting our milestones and delivering on the interim checkpoints we ensure the project comes in on time and to specification.

Integration process step 6 – Technical setup

Developer productivity is dependent upon the tools they use and the environments they work within. We ensure that our people are productive from day one by carefully managing and monitoring our technical environments.

• Sizing and pre-requisite definitions

We work with our client’s operations team to ensure the planned production environments are capable of supporting the projected loads at the required performance level.

• Access rights provisioning

Security, traceability and, ongoing maintainability require access management to be considered from the start. We ensure the access model used within development is as robust as that required in production to give you the confidence and peace of mind that your data is being handled responsibly.

• Middleware or integration platform deployment

Whether it be a net new installation or the reconfiguration of an existing platform, we ensure the tooling is functioning correctly and that it is capable of delivering the end-to-end transactional flow performance you need.

• Adapter deployment and baseline configuration

Where appropriate we will install and configure any productized adapters associated with the integration ready for them to be used for the in scope integration scenarios.

Integration process step 7 – Data assessment

Data is at the very heart of every integration. It can make or break the project and consequently it needs to be considered very carefully. We analyse your data landscape and identify issues that could affect the effectiveness of the delivered integration.

• Data stream analysis

Using techniques such as sampling and synthesis we determine the make-up of the data sources that are part of the integration to ensure that potential problems are identified early in the process and can be addressed.

• Volume, variance, completeness and quality

Data issues come in many forms. Sometimes the data is of dubious quality. Sometimes it is incomplete. Sometimes it contains errors that will cause downstream processes to fail. In order to deliver a robust integration that can handle such exceptions elegantly it is necessary to really understand the datasets and their limitations.

• Data enhancement / augmentation requirements

Supplementing the existing data with additional content from external sources helps to enrich your solutions and gives you more options when it comes to analytics and automation.

• Trusted source hierarchy definition

Not all data sources agree. It is common for there to be discrepancies between datasets. It is therefore essential to evaluate the relative trustworthiness of individual data sources, and indeed the individual data points or records stored within them, if a single unified version of the “truth” is to be developed.

• Data reconciliation process and rule definition

Sometimes you’ll want to use data from one source and sometimes it will be more appropriate to use an alternative source. Defining these rules and embedding the logic within the integration ensures you get the very best data set possible from your disparate data sources.

• Test data requirements definition

Using production data within test environments is inherently dangerous. Techniques such as data obfuscation or the use of synthetic data generators that mirror real-world data characteristics mitigate the risks and provide you with the test data you need to be able to be confident that the deployed solution is capable of handling operational loads without incident.

Integration process step 8 – Integration development

This is where the rubber meets the road. We build the integration through the configuration of platforms, adapters and middleware or messaging buses as well as via the development of code to access APIs and SOA interfaces.

• Test data creation

Without sufficient test data that accurately represents rea-world scenarios it is impossible to deploy any integration with confidence that it will perform as required.

• Configuration and coding

We use accelerators, pre-configured adapters and proven integration stubs to minimize the risk and reduce the time take to deliver a functional integration that truly works. Our team have built and deployed thousands of integrations and are used to working to tight timelines with mission critical datasets to deliver enterprise grade integrations.

• Testing

Testing provides you with the confidence that not only is the integration working but more importantly that it’s working as desired. Getting data from point A to point B is relatively simple. Doing that at scale, within the required performance parameters and, being able to handle data of variable quality is what makes the real difference.

• User Acceptance Testing (UAT)

We support your team to test the integration to ensure it is functioning as expected and required. It should be noted that it is usually impractical for UAT to pick up many common integration failures that may be due to resource contention, bottlenecks and excessive exception rates.

• Stress testing / Soak testing / Load testing

For organizations that are looking for peace of mind we are happy to support, or conduct, load testing to ensure that the integration can handle operational loads and projected transactional spikes. We will often overload integrations beyond the design envelope to ensure they do not fail and to gain an understanding of how performance is likely to degrade in burst scenarios.

• Penetration testing

Our security specialists will attempt to hijack transactional messages, intercepting them and attempting to inject malicious code into their payloads to ensure the integration does not increase your threat surface area or introduce a vulnerability into the end-to-end solution.

Integration process step 9 – Post deployment preparations

You wouldn’t undergo major surgery without planning to spend time in the recovery room and convalescing. Your integration needs similar levels of care if it is to bed in and become an established part of the IT ecosystem.

• Monitoring regime definition

Knowing that the integration is running is nice. Knowing how well it is running is much more useful. By tracking key characteristics and behaviors you can have confidence that all is performing as it should. Where thresholds are breached or trend analysis suggests they will be, our team will proactively intervene to investigate and remediate issues before they impact your systems.

• Technical handover

We ensure that your team know what they need to know to operate the integration successfully without having to refer to us. We also ensure they know when to call us in to prevent them turning a minor drama into a full blown crisis.

• Documentation provision

Depending on your requirements we will provide documentation regarding the integration design, data architecture, adapter configurations as well as the day-to-day operational tasks that should be completed to ensure the delivered solution performs as required.

Integration process step 10 – Ongoing maintenance & support

The integration is complete and it’s working. But that’s not where the story ends. Successful integrations needed to be maintained as the systems they connect invariably change over time and the business’s requirements flex to accommodate changing market conditions.

• Monitoring data analysis

By analyzing the outputs of the monitoring tools focused on the integration we are able to ascertain if something has changed that is likely to jeopardize the stability or performance of the end-to-end solution.

• Exception level tracking

Keeping an eye on the rate at which exceptions are raised helps to identify when the underlying assumptions that the integration was based upon are no longer valid. Everything changes over time. Source and target systems change. The way in which people use those systems change. Tracking exception levels helps to proactively alert you to when the integration needs to be reconfigured or realigned in response to changes upstream or downstream.

• Systems update reviews and impact assessments

We will work with you to proactively review system and configuration level changes to source and target systems in order to determine the likely impact on the delivered integration. Where it is likely that planned changes will have a detrimental impact on the integration’s stability or performance we will work with you to implement contingency actions to mitigate those effects until such time that the integration can be reconfigured.

• Ongoing mapping and configuration updates

Data models change. The reconciliation rules associated with accessing multiple source systems fluctuate over time as data quality varies. Fields are repurposed or retired. All of these things may require the mappings used within the integration to be revisited and reconfigured.

• Integration risk assessment and control effectiveness review

We will assess the potential for issues and how well the deployed control measures are working in order to determine the level of risk associated with the integration.

Multishoring’s 10 system integration process steps deliver the results you need

Integration is what we do. Our team understands how to mitigate the risks, avoid common pitfalls and, ensure your project is successful. Whether we’re in a turn-around situation helping to get an off-the-rails project back on track, or working on a net new integration, we focus on the things that truly matter and make it work within the constraints you define. We leverage best practices for system integration approaches where we deem them sufficient. Where we need to go beyond current best practice, we do so.

Depending on your appetite for risk, available budget, or timeline, we will work with you to define the optimal schedule of works to include the integration steps you need to feel confident in the delivered solution and its ability to run as designed issue free for as long as you need it. We don’t over engineer projects to artificially inflate the scope of works and we don’t skimp on activities that will materially impact the integrity of the delivered integration.

Integrations may be complex, but they do not need to be complicated. We use our experience and expertise to take challenging integration scenarios and make them happen. We do what is needed to make your integrations stable, secure and scalable so that you can focus on other things.

Share article on: