Observability will put you far ahead of your competition when it comes to large-scale IoT deployments. But should you build your own solution, or is it better to buy something off the shelf? Memfault knows observability like no-one else, so we asked its CEO what he thought about the “Build vs. Buy vs. Blend” conundrum.
In ipXchange’s second instalment in our interview and tutorial series with Memfault, we chat to François Baldassari, Memfault’s Founder & CEO! Obviously, we’ll be talking about observability in the IoT – don’t worry, that will be defined in a sec, or you can watch last week’s introductory demo of Memfault with Gillian – but the key focus today will be “Build vs. Buy vs. Blend”. Memfault has even prepared a neat handbook on the subject, which you can download by following this link:
But what do we mean by “Build vs. Buy vs. Blend”?
Well, engineers have a choice whether they want to develop a solution from scratch, buy something off the shelf, or combine this approach with some open-source or proprietary resources in combination with their own development. As François reveals, the question of “Build vs. Buy vs. Blend” is almost a trick question…
What is observability in IoT?
To begin, François explains that observability is a term that comes from the software world. Essentially, it is the idea that, by collecting telemetry data, engineers can see problems in software and resolve them without having to go on-site. When applied to the IoT and connected devices, observability enables designers to see any issues with connectivity, battery life, general performance, or other system errors without the end user having to self-report. This is all thanks to telemetry data being sent to a remote observability platform.
By remotely finding and fixing issues before an end user complains or gets in contact, you can improve the user experience. They also won’t have to send the device back for testing, which saves time, fuel, and money for all parties involved.
Memfault is a cloud-based platform which allows engineers to see these many insights for an entire fleet of devices in real time. All that is required is that each connected device has the Memfault SDK installed as part of its firmware build. So long as it can get that connection to the cloud, even via a peripheral device – such as a mobile phone in the case of a smartwatch, for example – the device can then report its performance to the Memfault platform. An engineer can then observe any issues, debug the device, and send an OTA firmware update to solve the issue. Memfault produces observability platforms for MCU-, Linux-, and Android-based systems.
Defining build, buy, and blend
Whenever a product is built, it is rarely from scratch. An embedded designer won’t build their own microcontrollers or smelt the metal for the wires and PCB traces. The key to efficient development is choosing when you need to build something yourself and when you can save time and cost by buying something off the shelf. François clarifies that using open-source frameworks also counts within the ‘buy’ category despite not involving a purchase.
For the most part, designers use a combination of build and buy. Blend means taking third-party solutions and modifying them to fit your needs. François uses the example of taking a free Real-Time Operating System (RTOS) and adapting it for specific application requirements. As you can see, ‘blend’ is a combination of ‘build’ and ‘buy’, and it allows you to get the best of both worlds:
Buying saves cost and development time, and building is where you can add your own innovation and differentiation to the market. By blending the two, you save yourself from reinventing what people have already spent years developing. This means that you can focus on innovation!
Does anyone build from scratch anymore?
Yes, but this is the most costly option. When you build something from scratch, you gain no benefit from a lot of money being spent by third parties in creating things that we take for granted every day. François gives the example of an e-mail client, which might take two years to build. The alternative is to just use a service like Gmail, and you’ll be ready to go within a day.
Of course, there are advantages to building a complete solution yourself: You can make it exactly to your specification, and you will own the IP. For something like an e-mail client, you might also own the servers, so you will have full control of the infrastructure that your product is connected to. There is also great satisfaction in building your own solution.
That said, buying the foundations of a design enables you to build on the shoulders of giants. You also benefit from the wisdom and mistakes of the past and can bypass your own ignorance. By buying or blending, your products will get to market faster, with far lower development costs.
Why buy 100% of a solution when you could build the 10% you need?
Hardware engineers love to build things themselves. That said, companies often drastically underestimate the cost of developing a solution. When it looks like you aim to use only 10% of the features offered by an off-the-shelf solution like Memfault, it turns out that this 10% is only the tip of the iceberg.
Behind that 10% of the features lies so much more than 10% of the development time and cost. François estimates that simply testing a system and making it market ready is 30% of this development. This is all time and cost that could have been spent developing the product functionality that will differentiate you from your competitors. Observability is not one of these differentiators as your customer will not notice its there in the perfect scenario.
Simply put, features do not scale the same as development time and cost. Building 10% of the features may in fact be 50% of the total development investment. Don’t be fooled into seeing only the tip of the iceberg. Blending your solution will cost far less than developing from scratch, so lowering cost should never be the motivator for developing in house.
For more insights on “Build vs. Buy vs. Blend”, with a load of stats to back up these claims, download Memfault’s very own IoT handbook here:
If Memfault can develop an observability platform, why can’t you?
Eamon then plays Devil’s advocate and asks François why someone else shouldn’t build their own observability platform when Memfault makes one all by itself. This is another trick question!
François is happy to admit that Memfault practises what it preaches. The Memfault platform is filled with tonnes of open-source code and is built on existing infrastructure from third parties. This even includes other observability IP, because why reinvent the wheel when the infrastructure to get telemetry data to and from the cloud already exists?
By using a blended development approach, Memfault has been able to focus on building a best-in-class observability solution for connected devices without having to worry about the foundations. François estimates that around 10% of the complete platform was built in house, and this is where Memfault’s innovation and differentiation comes from.
The opportunities for generative AI in software development
In keeping with current trends in the industry, Eamon asks François whether he thinks generative AI will have an effect on the ratio of build vs. buy during blended development. This was asked as we at ipXchange have seen many engineers using generative AI to write code. François sees this as a push-pull type of situation:
On one hand, it will likely result in much more software being bought rather than developed in house. This is because of the huge infrastructure costs of the Large Language Models (LLMs) used to generate content such as code, and the cost and complexity of training these models to produce market-ready content or easy-to-digest insights.
Building market-ready LLMs is out of reach for 99% of companies, so they will need to buy a solution off the shelf if they want to integrate generative AI into a product.
On the other hand, generative AI is slowly improving productivity for software developers. This is not to say that AI will completely write a solution for you, but it can certainly assist in the task and suggest smaller pieces of efficient code during the process.
AI for automated observability
Eamon also asks François whether he sees generative AI being useful for automated insights or debugging within an observability platform. François thinks about this a lot. He sees the value of using AI to analyse observability data and write code to resolve issues, but at this stage, off-the-shelf models do not know enough about observability or firmware development to be of much use.
That said, anomaly detection is somewhere where AI could be useful, but it requires that data be relatively periodic. For IoT monitoring devices where there is a normal state of operation, such as predictive maintenance, AI-based anomaly detection might be viable. For consumer devices like sports trackers, however, this might not be possible as the behaviour is at the whim of the user.
How do you differentiate yourself if everyone ends up using Memfault?
Memfault’s observability platform is still a relatively new technology, so at this stage, engaging with Memfault will make you an early adopter of best-in-class observability. This already puts you ahead of the competition by increasing customer satisfaction – you’re able to fix things remotely before they even realise something is wrong!
Of course, by not developing an observability platform yourself, it leaves you to focus on differentiating your products in terms of functionality. By partnering with Memfault, you can make sure that functionality is always working at its best.
You can even roll out new functions to a portion of your devices and see how people react. When it comes to product launches, it also gives you reassurance that when your devices are finally in the real world, they will behave as they did during testing.
François highlights that if a bug only happens once every 10,000 hours, you might never see it reproduced in the lab. But it will become a big problem once the product is launched at scale.
Without naming names, François retells the story of a company that had such a problem. Within a few days, Memfault was able to integrate its SDK into the product firmware, via an update. This was then used to fix the problem across 100,000 devices. It is this level of product observability that puts you so far ahead of the market.
What role will observability play for the future of connected devices?
Funnily enough, François is confident that observability in the IoT is the future. It is already a best practice in the software world, and that has shown proven trends that are around ten years ahead of the hardware domain.
François loves the idea of feature-focussed experimentation with partial rollout to 5% of users for market evaluation. This is very common in social media platforms, and ipXchange has certainly seen it happen on LinkedIn.
François is also excited about the role of observability in securing the IoT. It will enable companies to catch rogue or compromised devices in the act and prevent further damage to product reputation, safety, or privacy.
Eamon and François then laugh at the joke that the ‘S’ in IoT stands for ‘Security’. This is certainly a topic that ipXchange would love to discuss in the future, and thankfully, regulators are starting to crack down on this oversight.
How to get started with Memfault
If you’d like to learn more about Memfault and device observability, ipXchange has lots more content in the pipeline. As mentioned at the start of this article, Eamon previously talked to Gillian for a practical demonstration of Memfault’s platform. We highly recommend checking out that discussion, and you can also play with the platform yourself in Memfault’s Sandbox demo environment.
If you enjoyed this discussion on “Build vs. Buy vs. Blend”, don’t forget to check out this handbook that Memfault has produced on the topic:
We hope you enjoyed this discussion with François, and next time, there will be a Raspberry Pi involved in another practical demo. Look forward to it!
Keep designing!