Show notes
Chad talks to Benjie De Groot, co-founder, and CEO of Shipyard. Shipyard manages, creates, builds, and deploys ephemeral environments. Benjie talks about how Shipyard became a funded company, discovering who their ideal customers are, and building out the core team so Shipyard can accelerate and figure out their next steps in how to bring it to the masses. Ephemeral Environments Shipyard Follow Benjie on LinkedIn. Follow Shipyard on Twitter, LinkedIn, or GitHub. Follow thoughtbot on Twitter, or LinkedIn. Become a Sponsor of Giant Robots! Transcript: CHAD: This is the Giant Robots Smashing Into Other Giant Robots Podcast, where we explore the design, development, and business of great products. I'm your host, Chad Pytel, and with me today is Benjie De Groot, co-founder, and CEO of Shipyard. Benjie, thanks for joining me. BENJIE: Thanks for having me. CHAD: Why don't we start by if you don't mind sharing a little bit about what Shipyard is and does? BENJIE: Sure. At the core of what Shipyard is working on is ephemeral environments; not everybody knows what that means. That is changing a bit. But essentially, what we're focused on is on every pull request or commit for a feature; Shipyard manages, and creates, and builds, and deploys ephemeral environments. So that's a disposable one-off on-demand environment that any stakeholder in your internal company can use. And we focus on the tooling around that, on build pipeline, and then security around that. And then all kinds of other cool features that are necessary that pop up. CHAD: Cool. So as a developer, I'm familiar with the concept of developing locally, putting up my pull requests. And also, we deploy a lot of stuff to Heroku. So I'm familiar with some of the infrastructure that Heroku might give. How did you arrive at saying like, this is a thing that I want to work on and believe should exist? BENJIE: That's a great question. I actually am also a developer; that’s my background. And throughout the course of my career, I've always been on the technical side of the company. And what that's translated to, because of passion, to be honest, is always taking on a DevOps type role, so throughout the course of my career, a lot of responsibility. I mean, I started off writing Bash scripts, went to Puppet, did Chef for a while, did Ansible. Somehow I went back to Bash scripts for a lot of this stuff. Then this company called DoCloud popped up, which obviously became Docker, and I kind of got obsessed with that. And then I had a bunch of friends at Google, and they were telling me about this creepy thing called Borg, and that became Kubernetes. And so, my career has kind of happened throughout that entire process. And throughout, DevOps has kind of been my passion. Along with my co-founder, Peter, I was a high-priced Kubernetes consultant in the New York ecosystem just a few years ago. And a lot of companies were trying to make the transition to Kubernetes. And Peter and myself came in and helped people that were struggling to find DevOps resources. And what that always kind of looked like was there was some bespoke version of a deployment system that was perfect for the person that wrote it. But obviously, it wasn't good enough for me and Peter, for Peter and myself. CHAD: [laughs] BENJIE: And so we would rewrite it, and it would be great. But then, eventually, we'd move on, and someone else would rewrite it. And there were a few instances where we ended up going back to companies and just reimplementing what we had already done. And throughout that process of being this consultant, we kept running into this ephemeral environment thing and building the same tooling over and over and over again. So Peter and I, on a weekend, kind of got, "Oh, let's make a tool for ourselves." So we did that. And we made this exoskeleton to help our consulting business. And as things progressed, we kept just adding features, and it was really fun, and it was great. And then some of our customers or clients saw that. And they were like, "Hey, can we click that button?" And we were like, "I guess." And so slowly, it turned into a product that was very duct tape-y and glued together, but it worked great. And to be frank, I had been through the VC process on the technical side in the past and didn't want to go through that again, the hamster wheel of need to raise more and more money and so very, very averse. And was very set on a really nice lifestyle consulting business, and hell was going to have to freeze over for us to take any VC dollars. And then I don't know if you heard, but in March of 2020, hell froze over, and [laughter] there was a little pandemic. And at the same time, we got some pre-emptive term sheets, yadda, yadda, yadda. Next thing you know, we're a funded company building out a really cool product. So that's the origin story of where Shipyard came from. CHAD: Really cool. I definitely want to come back to what building the product for you has been like, and the funding, and where you go from here. But let's come back to the product itself. As a developer, my normal workflow is I'm working locally. I'm able to run the application that I'm working on locally here on my computer. I put up a pull request on GitHub. I ask my team to review it. Once it gets reviewed from a code perspective or a design perspective and gets a thumbs up, I merge it back into the main branch. And I deploy it to a staging server, at which point I would ask my stakeholder, my client, whatever, "Hey, this thing you're expecting it's on the staging server for you to check out." And everyone else on the team is doing the same thing. So where does Shipyard come in, and why is it better than that? BENJIE: So where Shipyard comes in, it's after the local development but before you get to staging or really before you get to production because, in practicality, a lot of people turn Shipyard into their staging servers. But what happens is through webhooks, we hook into your GitHub. And we see that there's a new commit that comes in. And we automatically build and deploy a fully ephemeral environment for that feature. And what that gets you is a few things. One paradigm that we're seeing a lot of is when you make that PR, a lot of end-to-end test suites are being run automatically using Shipyard ephemeral environments. And what that gives you is, in some instances, before you even have a code review, you're passing the suite of tests. And what that gives you is you save a lot of time. If there's just a dumb migration error or some typo or something like that, you're not wasting human capital or human energy on those environments. And the other instance there that gets really interesting is by bringing up these environments earlier on, product stakeholders and QA stakeholders can do their jobs earlier on in the process. And so you can avoid a lot of merge conflicts. So like, you merge something, and maybe there's an edge case that you hadn't tested for, and the code review didn't pick up. Well, all of a sudden, staging is broken. And some other team member that's using the same process you were now they're blocked. Or the client can't see that environment, and there's some other type of problem. But really, we didn't invent this paradigm. This is what FAANG does. There's a reason why I can't remember the last time that Gmail itself a button broke, or there was bad CSS, or bad HTML, same thing with Facebook, same thing with Netflix. Obviously, we all know about –- CHAD: There's the obvious DNS outages. BENJIE: [laughs] Right. I was going to say we all know about AWS, especially in December of 2021. That was a tough month. But yeah, from a UI/UX and controllable release perspective, this greatly increases your internal stakeholders’ ability to get their hands on features earlier, find problems, and then get those back to developers. And the other thing, and maybe this is a question for you. But have you ever been in a situation where you built something, and it doesn't actually get reviewed for a few weeks? And then there is a bug, and you have to go back, and context switch off of what you're working on and go back and put a whole other mental model in place to go back and remember why did I use a switch statement here? That's a bad example but something to that effect. CHAD: [laughs] Yeah. Well, I really try to avoid that scenario by having tight feedback loops, but sometimes it's unavoidable. It might be you finish something right before a holiday or going away or something like that; that can happen. So it's happened to me before, yeah. BENJIE: Right. And how do you get your product people or your UAT teams...when do they get to touch the feature that you're working on? CHAD: It's usually not until after a code review when it's been merged into main and deployed to staging. BENJIE: So that's kind of how we make that feedback loop tighter. And what we've seen in practice actually is a lot faster, more reliable releases. And there's a significant increase in the cadence of releases that can happen and a higher quality of those releases. CHAD: You mentioned that some customers end up even getting rid of staging. And so that's really exciting and interesting to me. When they do that, what does the overall picture look like? Is the code merge manual? Or do you have customers that are doing continuous deployment off of a thumbs up from the person reviewing it in the ephemeral environment and getting that automatically merged, and then maybe canary deploy or something to production? BENJIE: Yeah, that's a great question. The thing to keep in mind here is that the majority of our customers are larger, and they have bigger teams because obviously, this is a collaboration platform ultimately. And so there's more value for the more complex teams and more stakeholders. So we don't have anybody at this moment that I know of; there could be, doing LGTM is good enough. So there's always a manual component. But what it looks like from a staging perspective is that your main branch is actually ostensibly your staging environment, and so all the ephemeral environments are sort of dev environments that are shareable. And then when you merge because a code review passes, and QA checks, and UAT, then it gets automatically built into the main branch and the main environment. And then some people do QA. They'll final pass a QA or a final end-to-end test there. And then there's also a manual promotion to production as well. That's the typical pattern we've seen. CHAD: Cool. One of the things that when I've used...sometimes a problem even with staging. But when I've used or been on projects with some ephemeral environments, getting good data in those environments can sometimes be a challenge. Is that something that Shipyard helps with? Or what's your recommended approach to that problem? BENJIE: So that was one of the biggest problems we had early on. We put a lot of work into that. We apply the same git branch model to data. So the way that we do that is basically if you...oh, by the way, I forgot to mention something. We use Docker Compose as our application definition. So we extrapolate from Docker Compose and transpile into best practice Kubernetes YAMLs. So there is a little bit of inferring and magic we do in certain places. And one of the places we do that is if you have a named volume...sorry, am I getting too technical, or is this --? CHAD: Not for me. And in fact, I have follow-up questions about [laughs] why you have that approach of converting. BENJIE: We will dive into that in a second. And I have a whole bunch of redhead friends that make fun of me about Compose all the time, but I stick to my guns on that one. But I'm happy to talk about that. At high level, if you indicate to Shipyard this is a persistent volume that we want to make sure that child environments get, then we will do an instant snapshot. And we will actually provide that to the generated child ephemeral environment. And ostensibly, what that does is it allows you to test data migrations as well on these ephemeral environments. Now, to go back to your initial question, we encourage...and we're working on some partnerships actually with some interesting companies. But we encourage people to specifically have their main data set on main be ostensibly a copy of whatever the good data set is. But obviously, you're responsible for pulling out your own PII and all the confidential stuff there. But the key thing here is you're maintaining one environment with the right data on it. And then all of the subsequent generated ephemeral environments inherit that and can then change that. CHAD: Yeah, that's cool. That solves a real pain point that I've had in the past when trying to work this way. BENJIE: One company that I think is really interesting around this space is Tonic.ai. And we're actually working on some stuff with them, I think. But we share an investor, so that's why I know them, for disclaimer purposes. But they're great. And they have some really cool tooling around mapping your database to PII and automatic detection of certain types of information that you don't want pushed into your staging servers and to your developers' hands. So that's one to check out, too, if you're looking for data help. CHAD: Cool. So do you want to get back to this Docker question? Why that approach of converting the Docker Compose into YAML for Kubernetes? BENJIE: So this is quite a controversial topic. CHAD: [laughs] BENJIE: But I will tell you where it came from. Hearkening back to our origin story, what we saw was we saw a pattern of a lot of companies going a little bit too all-in in Kubernetes; let’s just put it that way, where every single one of the developers is running minikube or even K3s or K3d or whatever. And all of a sudden, the DevOps people and the SRE people in the organization are spending most of their time supporting developers in local development environments. So early on in that consulting game, we realized we don't want to do that. So if you want to work with us, we think you can use Docker Compose for most things. Now, that's obviously not always the case. There are some companies and applications that have hundreds of microservices. So obviously, Docker Compose is not a very realistic fit for those people. But the majority of people can pretty much encapsulate their application in Docker Compose. So that's one thing. The other thing is I mentioned to you that I'm a DevOps engineer for years. I'm sick of new YAML formats or specifications. So I have a saying, "Not another YAML, I say nay." My co-founder, Peter, hates when I say that, but whatever, I like it. CHAD: [laughs] BENJIE: So that's another piece of this. And then the biggest thing here is that we look at Docker Compose as rabbit ears on a television set. So you know, like a 98-year-old grandmother can somehow stand on one foot and hold the antenna the right way, and it's static. The picture is perfect, and they can watch...I don't know why I'm saying Jay Leno. I don't think it's on the air anymore. CHAD: [laughs] BENJIE: Sticking with the grandma reference, humans are really good at figuring out stuff like that. [laughs] And that's kind of what Docker compose is. It's kind of like if you can make it work locally, Shipyard is going to take care of the rest and clean up a bunch of stuff for you. So that's how we look at it. Admittedly, we do have some Helm stuff we're working…
Full show notes at the publisher 
Our TOPPODCAST Picks 
Stay Connected
