I lead the Data Engineering Practice within Scott Logic. I have a strong interest and expertise in low latency Front Office trading systems, software managing very large networks and the technologies involved in processing large volumes of data.
Artificial Intelligence
AI tools are being trialled in many areas including software development, with huge potential to transform ways of working. What might the impact of AI be on software architecture? In this post, I make some predictions and argue that software architecture may become more data-driven, with prototype architectures and architectural diagrams being generated from the actual code itself.
Data Engineering
The UK Government has an ambitious data strategy that aims to encourage and facilitate data sharing between departments and businesses. Elements of the strategy appear relatively straightforward, but how will the government fully realise the potential, and align citizens with this bold new approach?
Data Engineering
Organisations across the globe have been on a journey to find the optimal approach for managing and leveraging analytics data. In this post, I’ll set out each of the key milestones on the journey, to arrive at the latest milestone – the Data Mesh paradigm – and ask whether it is really a thing.
Tech
·
Video
Data Mesh: What's different about it, what it means to DWP, and how you might approach it yourselves
Data Mesh is an emerging architectural paradigm for big data that addresses the challenges inherent in the data lake and data warehouse approaches. In this talk, I define what it is, and Charlie Boundy, DWP Deputy Director and Head of Data Science, brings it to life through DWP's technology and data vision.
Data Engineering
What started as one faulty gas reading in the summer of 2017, ended up as a series of wasted calls where my bill kept getting higher and higher until it reached £11k. How could this have been handled faster and left me without considering moving energy provider.
Data Engineering
Distributed stream processing engines have been on the rise in the last few years, first Hadoop became popular as a batch processing engine, then focus shifted towards stream processing engines. Stream processing engines can make the job of processing data that comes in via a stream easier than ever before.
Data Engineering
Distributed log technologies have matured in the last few years. In this article, I review the attributes of distributed log technologies, compare four of the most popular and suggest how to choose the one that's right for you.
Resources
Data lakes? Most Financial Service organisations either have one or are considering one. This white paper explains key considerations and a warning to help guide their creation or optimisation to fulfil its requirement
Data Engineering
Yesterday the Financial Times boldly declared that BP saved $7bn since 2014 by investing in Big Data technologies. I spent a couple of hours researching Big Data technologies associated with BP members of staff to try and build up a picture of exactly which technologies they are using.
Data Engineering
Apache Spark has quickly become the largest open source project in big data, but why has it suddenly got so much momentum behind it?
Data Engineering
Welcome to my blog, Andrew's thoughts on Big Data. This page gives a little background on myself and the blog.