Migrations, integrations and your organization - making sense of the data
As consultants, sometimes we encounter scenarios where, due to M&A activity, client growth, competitive factors, discovery or by other means, an organization might wish to migrate or integrate data from other business applications or disparate systems. In this blog post, I'd like to define the difference between these two terms and highlight some powerful options for your organization's needs. Up to bat first: What is the difference between a migration and an integration?
MIGRATIONS
A data migration describes the process of transferring data between different file formats, storage types, or computer systems. Oftentimes, data migration scenarios present themselves during system implementations. Traditionally, this was an incredibly tedious task - it required numerous human resources dedicated to the project. Today however, data migrations are usually performed with much more ease, less resources and more programmatically through a well automated solution.
Database migrations are common when a business might need to move from one database vendor to another, say Oracle to Microsoft. In such cases, the migration often requires an actual transformation of the data since the underlying data format may change significantly. This in turn can have consequences on the applications layer. Fortunately, modern apps are written so that your information may be noncommittal to their respective database tech. In other words, going from MySQL to SQL Server is significantly less daunting than it has been in the past. The modern and increasingly more open nature of these technologies means that with the right strategy and ample testing, you can ensure that the desired functional and technical performance is desired without much headache.
Application migrations involve moving from one application to another. Oftentimes, this can be cross-functional: Different vendors or different systems. For example, moving from a new ERP or CRM platform will almost certainly require significant transformation as these data models are often application or vendor specific. Furthermore, these systems are highly customizable and often feature their own specific and unique metadata. Generally, an application programming interface, or API is accessible to protect the applications data model and integrity. Often times, an organization has rights to this API but it is subject to the software warranty of the actual application. This API is a conduit into the application and the way in which other tools can talk to and interact with the application. This is an important concept and one I will discuss later in the blog.
In my next blog, I will discuss data integration in more detail and explain why this is a critical discussion to have with your organization in 2015.
Database migrations are common when a business might need to move from one database vendor to another, say Oracle to Microsoft. In such cases, the migration often requires an actual transformation of the data since the underlying data format may change significantly. This in turn can have consequences on the applications layer. Fortunately, modern apps are written so that your information may be noncommittal to their respective database tech. In other words, going from MySQL to SQL Server is significantly less daunting than it has been in the past. The modern and increasingly more open nature of these technologies means that with the right strategy and ample testing, you can ensure that the desired functional and technical performance is desired without much headache.
Application migrations involve moving from one application to another. Oftentimes, this can be cross-functional: Different vendors or different systems. For example, moving from a new ERP or CRM platform will almost certainly require significant transformation as these data models are often application or vendor specific. Furthermore, these systems are highly customizable and often feature their own specific and unique metadata. Generally, an application programming interface, or API is accessible to protect the applications data model and integrity. Often times, an organization has rights to this API but it is subject to the software warranty of the actual application. This API is a conduit into the application and the way in which other tools can talk to and interact with the application. This is an important concept and one I will discuss later in the blog.
While every data migration is different, there are certain commonalities. Typically, data on the old system is mapped to the new system, providing a design for the data extraction, transformation, and loading (ETL) process. Therefore, part of design is relating the old to the new: the data formats, the structures and the requirements. Oftentimes this "relating" is often lost, but the T in ETL is incredibly important. Programmatic data migration can often aid highly with this. However, very minimally, data will be read from the old system and loaded, or written to the new system.
One determination to make during the initial planning phase of a project is the actual data quality itself. Often times, migrations can be a great opportunity for automated and manual data cleaning because this helps to improve the overall data quality. For example, if you are taking a list of contacts and migrating them into an ERP system, it pays to have these addresses validated for accuracy, cleaned and cleansed of typos and duplicates. Typically, systems offer this functionality out of the box with duplicate detection and the like. As data becomes more complex than the aforementioned scenario, automated/manual data cleaning will be an effective strategy to eliminate headache in the back end of the project. It will also enable you to eliminate redundant or obsolete information and match the requirements of the new system more efficiently. This will always positively affect the project and can ease validation and testing, ease of use, complexity and overall decrease the scope of your requirements.
Automated and manual data cleaning is commonly performed in migration to improve data quality, eliminate redundant or obsolete information, and match the requirements of the new system.
Data migration phases (design, extraction, cleansing, load, verification) for applications of moderate to high complexity are commonly repeated several times before the new system is deployed.
Some other more technical considerations include providing an input file spec for interrogating the dataset to be transferred and ensure that it meets your predefined criteria of the target environment.
Your organization may also opt to have on-the-fly data validation occurring at the point of loading which can report on errors and progress. Something to seriously consider however, is the "scope" of your data elements. If they are highly integrated and complex and the relationships full of dependencies essential to system functionality, then you may have to do more digging to get better insights into your migration.
In either case, your organization should subject your data load to stringent data verification and testing. It is so important to determine if your data was accurately translated. This means it is complete and supports both your technical and functional processes inside the new system. A strong strategy during this process is to do a parallel run of both systems to identify potential disparities or even worse, data loss. In my experience, I have also seen organizations keep one copy of say an application license for years after the migration for referential integrity. Mitigating erroneous data loss should always be a strong, strong consideration.
To provide a quick recap: Within a migration, you will define a source and a target. The data will flow from the source to the target, and during this process the data will be extracted and loaded, often enduring a transformation process often and in between. The data may also flow in multiple directions, be defined through multi-step processes and may be taking place in parallel with other migrations. A migration may take hours, days, months or years depending on the scope. The results? Data will be moved or copied from one environment to another, and removed or decommissioned in the source. It is best to conceptualize a migration as a project with a start date and an end, or "go-live," date.
INTEGRATIONS
While migrations are a temporary part of the IT organization's efforts, data integrations by definition are a more permanent part of IT solution architecture. A data integration defines and describes the way data flows and is combined. The end result is often presented at the functional level - say an application or data warehouse - and a unified view of the underlying data. As such, it is a process and not a project activity: A key distinction between an integration and a migration. Managing this on going process is often done with standard ETL technologies which supply operational data, say telecom or oil/gas meter data from different geographical regions to data warehouses. This process becomes significant in a variety of situations, industries, and sectors. Indeed, data integration has become a buzz word as the globalization and pace of technological innovation accelerates and creates complex data sets and code halos. Undoubtedly, the volume and need to share existing data (Ever heard of Big Data?) is experiencing unprecedented growth.
At this point, you might be asking: How this is applicable to my business? For this question, I present a few customer stories:
One determination to make during the initial planning phase of a project is the actual data quality itself. Often times, migrations can be a great opportunity for automated and manual data cleaning because this helps to improve the overall data quality. For example, if you are taking a list of contacts and migrating them into an ERP system, it pays to have these addresses validated for accuracy, cleaned and cleansed of typos and duplicates. Typically, systems offer this functionality out of the box with duplicate detection and the like. As data becomes more complex than the aforementioned scenario, automated/manual data cleaning will be an effective strategy to eliminate headache in the back end of the project. It will also enable you to eliminate redundant or obsolete information and match the requirements of the new system more efficiently. This will always positively affect the project and can ease validation and testing, ease of use, complexity and overall decrease the scope of your requirements.
Automated and manual data cleaning is commonly performed in migration to improve data quality, eliminate redundant or obsolete information, and match the requirements of the new system.
Data migration phases (design, extraction, cleansing, load, verification) for applications of moderate to high complexity are commonly repeated several times before the new system is deployed.
Some other more technical considerations include providing an input file spec for interrogating the dataset to be transferred and ensure that it meets your predefined criteria of the target environment.
Your organization may also opt to have on-the-fly data validation occurring at the point of loading which can report on errors and progress. Something to seriously consider however, is the "scope" of your data elements. If they are highly integrated and complex and the relationships full of dependencies essential to system functionality, then you may have to do more digging to get better insights into your migration.
In either case, your organization should subject your data load to stringent data verification and testing. It is so important to determine if your data was accurately translated. This means it is complete and supports both your technical and functional processes inside the new system. A strong strategy during this process is to do a parallel run of both systems to identify potential disparities or even worse, data loss. In my experience, I have also seen organizations keep one copy of say an application license for years after the migration for referential integrity. Mitigating erroneous data loss should always be a strong, strong consideration.
"Chuck's Consultant Hat"
After discussing database and application migrations, it is only fitting that I now touch on Business Process Migration. We can talk business applications all day but these are simply enablers for businesses and their respective processes and procedures. These business processes operate through a partnership of HUMAN and APPLICATION functions and actions. When these change, yes, you might migrate data from one database or application to another to reflect organizational activity. And, yes, you should definitely reflect the information about customers, products and operations. However, the drivers of mergers and acquisitions, business optimization and reorganization should not overlook the people, teams and human resources in place. Your people are your biggest asset, and migrations can deeply affect the operational and day to day activities of end-users. You should take great care to perform adequate briefing, training, user-acceptance testing and change management and bundle your strategy hand in hand with your technical, operational and competitive focus. A highly adaptive approach, strong synchronization and clear visibility into all steps of the migration for stakeholders should be a key requirement of your migration.
To provide a quick recap: Within a migration, you will define a source and a target. The data will flow from the source to the target, and during this process the data will be extracted and loaded, often enduring a transformation process often and in between. The data may also flow in multiple directions, be defined through multi-step processes and may be taking place in parallel with other migrations. A migration may take hours, days, months or years depending on the scope. The results? Data will be moved or copied from one environment to another, and removed or decommissioned in the source. It is best to conceptualize a migration as a project with a start date and an end, or "go-live," date.
INTEGRATIONS
While migrations are a temporary part of the IT organization's efforts, data integrations by definition are a more permanent part of IT solution architecture. A data integration defines and describes the way data flows and is combined. The end result is often presented at the functional level - say an application or data warehouse - and a unified view of the underlying data. As such, it is a process and not a project activity: A key distinction between an integration and a migration. Managing this on going process is often done with standard ETL technologies which supply operational data, say telecom or oil/gas meter data from different geographical regions to data warehouses. This process becomes significant in a variety of situations, industries, and sectors. Indeed, data integration has become a buzz word as the globalization and pace of technological innovation accelerates and creates complex data sets and code halos. Undoubtedly, the volume and need to share existing data (Ever heard of Big Data?) is experiencing unprecedented growth.
At this point, you might be asking: How this is applicable to my business? For this question, I present a few customer stories:
- Perhaps your organization's enterprise systems are not tuned to how your company is set up. You need to achieve compliance across your org and find a system that will help you deal with the challenges of cross-company accounting and operations.
You might wish to:
- Streamline financial systems and simply build insightful and meaningful reports into the business's different operations
- Accurately track project costs, profitability and your marketing/sales force efforts.
- Empower customers to self schedule appointments and get knowledge base articles and see what knowledge is most useful to them
- Provide your whole organization a better window into your customer's journey, from nurture to return or recommendation
Indeed, a more general and applicable scenario to any company is an integration between a CRM and ERP system. The sales, marketing and customer services business units can have their information integrated between the accounting, finance and planning organizations within your company. This can be very powerful and can enable a business to achieve a 360 degree view of your customers, from lead to cash generation, and automate the processes so that the two systems seamlessly hand-off and share information.
It is important to note that a CRM often handles the lead>opportunity>invoicing processes of the customer interaction. Therefore, a typical integration point in this scenario is the opportunity to order to invoice process. In this case, the ERP takes that information and handles the accounting side of the customer interaction - the order, the payment terms, invoicing, billing and delivery. Building out this solution is not a new concept: in the 21st century, there are solutions out there that do this out of the box, such as NetSuite. However, while NetSuite does offer a fully packaged solution, it is only available as a cloud-based solution, it doesn't play well with particular industries such as healthcare, and it may not be feasible nor make sense for you to uproot what works today. What if you are comfortable with your current ERP and CRM systems? Or, what if you'd like to keep your E-Business suite and confidential data on premise? This is a typical scenario where building a data integration might be your best bet.
In my next blog, I will discuss data integration in more detail and explain why this is a critical discussion to have with your organization in 2015.
Comments
Post a Comment