Show notes
Chris is weathering through a slight lull, a holding period, where his team waits for new features to become available with some of the platforms they integrate with, and as they think out new facets of the platform they're building. Steph has been thinking recently about working in isolation. It's a topic that Joël Quenneville pointed out to her and mentioned. Can engineers work in isolation and be successful? Become a Sponsor of The Bike Shed! Transcript: CHRIS: Always be singing. STEPH: I can't remember if I've shared the story with you. But I had a beautiful little human moment with someone at airport security. Because when I travel with my mic, I always get stopped because there's the middle long, thin piece that looks like what you would screw on to a gun like for a silencer. And so [laughs] as he was going through, the person was looking at it, and then he called over a buddy. And then they called over another buddy, and there's like three TSA agents all looking at the X-ray screen. And finally, they're like, "Yeah, we need to flag it." So they moved it over. And then he was digging through, and he pulled out the big metal piece. And I said, "It's for a microphone." And he's like, "Okay," and he kept looking, and then he finally found the microphone. And he lit up because I guess he wasn't really sure to believe me at first when I said it. But he lit up, and he was like, "Karaoke?" [laughs] I was like, "No, it's for podcasting." CHRIS: But not 100% no because we do sing plenty on this show, so... STEPH: I think that's what made me think of it. It was your singing. [laughs] CHRIS: Yep. My wonderful, wonderful singing. STEPH: Hello and welcome to another episode of The Bike Shed, a weekly Podcast from your friends at thoughtbot about developing great software. I'm Steph Viccari. CHRIS: And I'm Chris Toomey. STEPH: And together, we're here to share a bit of what we've learned along the way. So, hey, Chris? What's new in your world? CHRIS: What's new in my world? We are in sort of a...what's the word? There's a bit of a lull right now, not like a big lull, but we had a bunch of clear work that came into the team, did a bunch of iterations, some testing, built some new features, et cetera. And now there's a small holding period basically where we wait for some new features to become available with some of the platforms that we integrate with and also as we think out some new facets of the platform that we're building. So we've got this little bit of time here where we're not necessarily building out as many new novel features. But instead, as a dev team, we're taking this moment to be like, oh, cool, let's tie down. I want to make a sailing analogy here, but I don't know sailing. It's like tie down the somethings and batten the hatches, maybe. That sounds like a thing. [chuckles] But so we have a couple of projects going right now. We want to really accept the truth and lean into Sidekiq. So right now, we have a mix of ActiveJob and Sidekiq jobs. And they're confusing, and et cetera, et cetera. So we want to kind of lean into that, upgrade dependencies, that sort of thing. We are, again, doing a little bit of work on the observability foundation of our system. so how do we know what's going on at runtime? Also, working on just some core features and functionality. We have done a little bit of an exploration into the event processing stuff, some of that that I've been talking about. It's actually been very interesting. So we're working with Customer.io as a platform, which is omnichannel communication behavior-based messaging sort of thing. So when a user does X, send them an email and then wait three days. And if they haven't responded, then do this other thing. And I think I've said this in previous episodes; I'm so wildly impressed with that platform. They have done such a good job. And I know that good software doesn't happen in a vacuum. In fact, if we're being honest, a lot of the software out there is not very good. And not only do they do a good job, but it's across...there's a ton of functionality in Customer.io. And it's interesting because we're finding ourselves leaning into it even more because it is such a solid platform and because it connects into our event system. Like, it's a segment destination, so all of our analytics events get piped into Customer.io, and then we can action on any of them. And the actions can be quite complicated. And this is where we're getting into good idea, terrible idea space. And to be clear, this is still just an exploration. But we basically wanted a way to do more. There are a bunch of different actions that you can take so, like send an email, send an SMS, or there are a couple of other slightly fancier ones. You can trigger an event within the Customer.io system. You can actually do an arbitrary HTTP POST, PUT, PATCH, whatever, any web requests you want to make. So if you want to integrate with essentially anything else out there, you can do that. You can send some structured data over the wire. And so we've now been like, okay, what if, and stay with me here, what if we use our analytic system and we send events whenever a user does something, and then that event eventually trickles down to Customer.io? Within that, we allow ourselves to respond to that event by emitting a different event within the system, within Customer.io. And then, via the webhook functionality, we fire that back to the Rails application. And then there we can do whatever we want. And in a way, that sounds absurd because we're starting from our app, and then we're sending some events down, processing them in certain ways, sending it back to the app, and then maybe doing something. In particular, one of the things we want to do is richly formatted Slack alerts. And Customer.io has a Slack alert functionality, but they can't have any of the fancy stuff. They can't link to our customer in the admin dashboard. So we found that that functionality is particularly useful for our admin team. And so we're like, ah, this feels weird. But if we were to do this loop out and back, then ideally, we get the power of Customer.io for non-technical users or non-engineering team users to configure workflows and to say, "When a user does this, I actually want to alert the admin team via Slack." And we want it to be rich and have buttons that you can click and all that kind of stuff. And although the thing that I just described seems complicated, is a word that I'll use for it, confusing at times, it isn't...like, I don't want to do all of that in the app. I don't want the app to have to think about how do I wait three days? We technically can do that with Sidekiq, but it gets us in trouble and whatnot, whereas Customer.io that's a core concept for them. And so, again, very much exploration. This will probably be a future good idea, terrible idea segment. But that's been an interesting one to explore. STEPH: You have quite a talent for you preface something as a bad idea, and you do a very good job of making it sound reasonable and good. [laughs] So it's interesting to be on that side of like, good idea, bad idea. It's like, I'm looking for the bad. And I have questions, but overall, [chuckles] you do a very good job of being very thoughtful and walking through why it makes sense or what are the benefits of it. So you answered some of my questions around why still send it to Customer.io versus just having it all in-house. So the fact that the admin team has access to it makes a lot of sense. I want to clarify one point. So when you send it to Customer.io, Customer.io then needs to send a message back to your application. And then that's when you customize the Slack message. Do you need Customer.io to send that message, or could you just fire off an event to Customer.io to say, "Hey, capture this, but don't do anything with this. And then we're going to send the Slack message because we want to customize it."? CHRIS: I think the key is that we want to leverage the fact that Customer.io is the platform that our operations team really is now becoming comfortable with and using for this behavioral automation workflow type logic. So that idea of when this event, you know, when this triggering event happens, if this condition is true, then respond in this way. And so because Customer.io is the platform that A, is quite good at that and B, is where our admin team is now thinking about doing that, one thing that we might do let's say a user completes some action within the application. So they fill out a form to submit their interest in some new platform feature. Initially, what we might want to do there is alert ourselves to say, "Hey, this happened. Take some action." And then eventually, we may want to instead switch that over and send an email to the customer with the next steps that they need to do. And the ability to gradually transition across that spectrum is really interesting to me, and again, Customer.io being the platform, sort of the hub for how we respond to these events. At the same time, I know that this feels like a generic message processing system that might be a Kafka queue somewhere else. And so I've got that in the back of my head of like, is this weird? I think it's a little weird. But it also, thus far as we're exploring it, is very approachable for the admin team, very familiar for them, and reasonably powerful. And also, there's a drag and drop editor for the events and the payloads. And it knows for this event, here's the stuff that's available to you. And so the ability for our admin team to interact with that interface is really great. And we don't have to build it. We don't have to think about it. But I will say I've worked at so many different companies that have their ad hoc system that makes it easy to do generic X, Y, and Z. And it's bad, and it falls down. And it's impossible to know when anything happens. And so, I've got a lot of concerns in the back of my head, which I will want to at least think through and understand the trade-offs that we're making if we pursue this path, but it is very interesting to me. So right now, a lot of this logic does live in the app. But it means that it requires a code change for anything that we want to do like this. We want to have a Slack alert whenever X happens. Now, the developers are in the loop for all of that. And really, it's the operations team that owns the decisioning on that. And so if they can also self-serve and instrument the action, the alert, the follow-up, the whatever it is, if we can give them those primitives in a platform that they already understand, that sounds nice. I'm intrigued, is what I'll say. So anyway, while we're in this lull period, we are trying out some fun stuff like that and exploring those sorts of things. STEPH: I like that perspective that you're putting on it, or at least the one that's standing out to me is the concept of ownership is like who gets to own these actions. But then beyond that, that's the part where I feel a little squirmy is, so we are using this third-party tool because it makes life easier. But then, at what point when we start building software around this third-party tool to then customize it back on our own side. Then if someone is in Customer.io, so if an admin user is in there and then they trigger an event, is there going to be confusion as to what's going to happen? And can they retry an event? Because I'm realizing my initial suggestion where it was like, hey, notify Customer.io that this is there but then also manage sending the Slack message that would prevent them from being able to have that retry capability. And that may be very much worth preserving. So then it's understood that hey, if you want to manage this, we are giving you full access to manage this work. We may customize it, but this is still the interface in which you go through to have three tries or to manage that workflow or these actions that get sent to users. CHRIS: Yeah. I think you've perfectly highlighted the why this might not be a great idea or at least the concerns to explore before adopting this more thoroughly. And even just the idea of adopting it more thoroughly, like, how tied into the system are we? How business-critical does this new external piece of software become? Because I've seen that to be really problematic where there are organizations that I've worked with that are like, "Oh God, we would love to move off of system X. But unfortunately, it's basically the one thing holding this business up." And I'm like, yeah, I get that. And that happens. So yeah, being really intentional with that. And that's why we're very much in an exploration place. But we have a bunch of stuff that we've done that required engineering work. And we're now seeing like, actually, could we map this into this other tool? And can we build the set of primitives in that space that now this team can own that whole experience? And then critically, can they debug it? Will we know when something goes wrong, et cetera? Those are always parts. At this point, I don't think I can just imagine a happy path. And I hope this isn't true for the rest of my life. But the work as a software developer, especially after having done a couple of rounds of it and as a consultant, I just imagine failure modes. It's all I do. I'll be like, okay, we just need to wire X up to Z, and then we need to fire off a request. And then, once we get the message back, then we can process them. I'm like, right. You just described 13 things that can go wrong. Now let's imagine each of the different failure states because that's all I'm going to do. Who cares about the happy path? Those are easy. Those write themselves. It's all of the failure modes that I need to think about. And someday, when I retire, and I go to a log cabin in the woods, and I don't talk to people for a while, maybe I'll go back to a place of only happy paths. But that is not my truth right now. STEPH: I can't tell you how many people in my personal life I have annoyed so much [laughs] because all I see are failure modes. And one, that's a delightful t-shirt. [laughs] I'd love to have that. And then yeah, I feel you because there are so many times where someone is...like, I'm with someone who's like a big idea person. And so they're just launching into what-ifs, and we did this. And I can't help it, and I have learned to help it. But it has been a struggle with some strong feedback from family and friends to reel it in. Because then I will start to think through okay, well, what's the details? And I have some questions. What happens when this happens? And yeah, all I see are failure modes. [laughs] It is very true for me too, and not always...not so great. So I, too, shall get a log cabin one day and try to forget all of that. CHRIS: I will say I painted that as a particularly glib version of myself. But some of what I'm doing right now, particularly joining an early-stage startup and taking the role of CTO, was very much to try and intentionally resist that. Because right now, I have to be really careful with how much of the potential edge cases and whatnot. I'm considering exactly how robust of a platform are we building? Very is the answer. But what about extremely? Because extremely is an option but extremely…
Full show notes at the publisher 
Our TOPPODCAST Picks 
Stay Connected
