IT Consulting

End 2 End Data Platform Design & Architecture

We understand the different data platform types (data warehouse, data lake, ods, data mart, sql vs nosql data stores ...) characteristics and their relations. We know that to build up these platforms you need superb data engineering skills, a team with a broad range of skills (..., data modelling, operations,....data integration, ETL, jobs orchestration, automation, security, governance, ...) and an organisation that is supportive in achieving common goals. We can help you to architect your system, design its components as well as work on your teamwork best practices. In other words, we can support you with architecture review board meetings or the data engineering community.

Data Ops

It is not easy to design, build and run effectively a modern data platform. By definition, it is a heterogeneous platform with lots of components that need to work together. DataOps is a new approach that helps to solve this problem. 

Let’s stop and think about it. How do you 

• automate data pipelines, deliver on - demand development environments with huge data sets to test against (e.g. pseudonymized from production) ?

• "promote" job schedules and orchestration to a new environment?

DataOps is rooted in many principles of Agile, DevOps and Lean Manufacturing. Its goal is to help you better manage (automate!) your data, processes and teams. We can help you to enable DataOps at your organisation.

Data Services & Software Selection 

One of the fundamental problems with current technology is that there is an abundance of tools and services that can help you to achieve the same goal. The benefits of different approaches are not always clear, but also which of them are the most important ones. If you decide to use a Graph database that will cost you $$ per hour. However, this choice does not guarantee you that it will be effective without any additional costs. We will help you to select the right services that fit most into your environment.

Data Integration

Probably, you ask yourself…

• Should I build Spark pipelines or invest in an ETL Tool?

• Should I leverage dockerized environments (ECS / EKS / Batch) or invest in AWS Lambda?

• How will I integrate with external and partner solutions? 

​

The choice you make here will shape your target architecture. We will help you to make the right decision.

Performance Optimization & Cost Efficiency

We understand that cloud environments potentially mean better performance and lower costs.If your spark jobs run within minutes instead of hours, your data pipelines work in delta mode instead of full mode or if your SQLs run against partitioned data and not against whole tables, it all means that your software will cost you less to run and the number of milliseconds and seconds to run will be lower.