Should companies go full blowing big data/data science platform right away?
In my opinion, you should first look at the different stages you are in. Are you in the Proof-of-Concept phase, where you are just working with offline data, where you are proving your concepts? Or are you in the MVP phase or in the creation of an MVP, where you are bringing in the first users, the first customers?
The possibility of later scaling as one of the most important aspects
What is very important: You always have to make sure that you can scale afterwards. Why?
If you have to make a complete technology switch - for example from Kinesis to something like Kafka - because the tool you first used may not have as many configuration options, it can get annoying because that stuff doesn't work together with that good anymore.
To answer the question at the beginning: No, I wouldn't go for a big data solution. Because that doesn't make sense.
If you're worried about the money, I would start on a cloud platform. I would start with AWS and see how that works. Because normally the tools are very scalable. Later on, if you're not satisfied with the tools of AWS, you can take the challenge if you want to use something more general and have a lot of work.
It's always a trade-off and you can't say that there is a right or wrong.
Previous calculation
One more tip on what you should keep in mind regarding microservices with these tools: Always make calculations and calculate the pricing. Because normally the pricing gets very very expensive. Or the whole thing becomes very very expensive if you scale it. If you are behind the MVP and if you go to mass in the validation phase. If you expand it to a few countries, for example. That can get very very expensive very quickly - so always remember that!
What do you think about it? Let me know in the comments :-)
>> created by Mira Roth
My free 100+ pages Data Engineering Cookbook
Follow us on LinkedIn
Check out my YouTube
Check out my full video on YouTube!