Elliot Gunn

@elliotgunn.com

1471 Followers 254 Following

technical marketing @ Kestra / elliotgunn.com / Canada

If you’ve written in the past for Towards Data Science, you may also have questions about your content appearing on their new site. We’re actively working on understanding what’s happened, and if you have concerns, please email us and we’ll help educate you about your options. In the meantime, we’re getting this publication launched to serve the large number of writers who have data science stories to share, now, and want a home for them. We’ve partnered with Paolo Perrone, an editor from the Medium community, to get started (look for an introduction post soon), but we want to work with you in building out this new community-driven publication. Let us know if you want to be involved in supporting the amazing community of data scientists that has developed over the years on Medium by replying to this email and we’ll be in touch.
Your guide to legendary data migrations Success stories, lessons learned, and tools to turn your migration into a milestone achievement.
And that provides a good pair of rules of thumb for these kinds of statistics. Any time someone cites a very impressive quantity like this, it's a good idea to ask questions like: what's the nearest relevant comparison? What else is true if that is true (for example, if health insurance really makes $1.4tr in profit, you can Google and find that health insurers tend to report single-digit profit margins, which implies that their revenue is about equal to total GDP). You can also ask how plausible it sounds to multiply the number by ten, or divide it by ten, or if that doesn't actually seem to change anything at all.
What it does mean, though, is that getting your developer tool to market today requires very different tactics and philosophies; and that the old modicum of “if you build it, they will try it” (or something like that) is no longer true. Good product and engineering, to some degree, are table stakes. Everything else – specifically go-to-market – has never been more important. Founders who can put their creative energy towards sales and marketing, instead of treating them like a necessary evil, are going to have the best chance of standing out from the pack.
Is the data even the same? Your team pulls an all-nighter one quiet Sunday evening to actually run your migration script, and everything miraculously moves from Oracle to Snowflake without any errors and only a few minutes of downtime. But you’re not done yet. You get a message from the machine learning team the next day: “Hey, our models are producing unexpected results. Is the data in Snowflake actually right?” Data migration experts grapple with a puzzle similar to the Ship of Theseus paradox: How do you know your data remains exactly the same amidst all the transformations? Trying to answer this question is challenging, and many companies have emerged to address this, and other problems, with solutions.
Don’t overlook other users of the database. Our analytics pipeline was getting data directly from Postgres, and moving it over to use MySQL ended up being time-consuming. This was the final blocker to decommissioning the old database. Using the Yelp-standard stack improved performance. This isn’t due to MySQL being inherently more performant, but by using the same stuff the company uses, we benefit from many people’s efforts monitoring and optimizing our database performance.