Arrested DevOps

Latest Episodes:

devopsdays Toronto 2017 with Roderick Randolph, Arthur Maltson, Aaron Aldrich, and Amy Mansell Jun 13, 2017

Show notes

Bridget records live at devopsdays Toronto with local organizer Amy Mansell, who is on a podcast for the first time, and three speakers. Aaron Aldrich, a DevOps consultant and community builder at Cage Data in Connecticut, spoke on managing fires and is also organizing devopsdays Hartford, so Aaron is "secretly here for reconnaissance and stealing ideas from other organizers." Arthur Maltson and Roderick Randolph, both at Capital One, co-gave a talk on deep work and structuring a DevOps team. Roderick leads the DevOps practice for Capital One Canada, in what Roderick calls a software studio in North York. The recording ended early, because the batteries in the Tascam recorder ran out about ten minutes before the end.

Organizing a First devopsdays

Bridget notes the overlap between ops people and conference organizers, and asks Amy how the organizing team recruited a newcomer. Amy met Steve Pereira, one of the organizers, at a DevOps Toronto meetup about a year earlier, and asked a million questions about the role before saying yes. The appeal was a grassroots event in many cities, and Amy does community building in the day job. Planning one is "a full-time job in and of itself." Amy's pro tip is to talk a lot on Slack, and Amy admits the team may be annoyed by the number of direct messages.

Steve was too sick to attend, which became a test of the team. Amy says the organizers kept information in the group email and group chat, even for things needed from one person, and held regular meetings and kept shared files. When Steve said Steve couldn't make it, "there was almost just like a playlist that you could go through and check off." Amy adds that the alumni organizers were welcoming, which made the onboarding feel comfortable.

Foreground, Background, and Deep Work

Bridget asks Roderick and Arthur how they balance interrupt-driven work against heads-down work. Roderick says to be unafraid to say you need time to focus, pointing to research that "every time you're interrupted, it takes 25 minutes to get back to what you were trying to do earlier." The model from the talk splits a team in two: a foreground group handles the firefighting while a background group works on preventing fires.

Bridget wonders what happens when everyone who understands Project X is in the background. Arthur says that if only one person can answer a question, that's the problem to begin with, and the team is still working on spreading the knowledge. Arthur hopes to reach a "pair programming or pair opsing model." Bridget pushes back that deep contemplation and conversation are hard to combine, noting that Pivotal is a big fan of pairing. Arthur agrees that foreground pairs can learn from each other, then separate when someone needs contemplation, though the team isn't there yet.

On onboarding, Roderick says the team timeboxed the model for a month, and it worked well enough to keep iterating, and that being senior doesn't remove the chance to learn from other groups, which is one reason Capital One sponsors the event. Arthur credits Piyush Chugh, a former colleague now at Capital One Canada, with creating the foreground and background idea, and says Arthur took the credit at first.

Incident Commanders and Backups

Bridget asks Aaron what happens if the incident commander is a single point of failure. Aaron says the odds are low while an incident is active, but the best practice is to have someone who can step in, and adds that "best practices is a great word we all like to throw around and then pretend they're not real when we actually put them in place." Aaron says onboarding people to the incident process is about leadership setting the tone: being deliberate, and not letting someone work past capacity when they say they're fine, instead saying "I really need you to go home and get rest." The talk came from an incident that went well, and Aaron wanted to capture why, since the company culture side of incident response seemed more important than individual tooling.

Bridget says ops culture spends a lot of time on what broke and less on what went well, and asks Amy how events handle that. Amy says that on the day of an event, feedback is nearly instantaneous: are people smiling, talking, on time, did everyone eat lunch on time?

Keeping Questions in Public

Arthur says the challenge is redirecting conversations that would happen in private chat into a group chat where others can see them, since another engineer who has seen an answer before can give it when the one expert isn't there. Aaron says the team Slack has shifted from private to public messages, in part by reinforcing "default to the public channel," and by copying and pasting private messages to the public channel. Bridget adds that safety matters: if asking a question gets people laughed at, they'll ask in private. Arthur says senior people can use that privilege to show that even someone senior doesn't know everything.

Bridget chats with devopsdays Toronto local organizer Amy Mansell & speakers Roderick Randolph, Arthur Maltson, and Aaron Aldrich.

devopsdays Toronto 2017

Roderick & Arthur

Deep Work: A New Working Model For Ops Teams In A DevOps Environment

Aaron

Managing Fires: The Role Of Leadership In Crisis

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Velocity San Jose - discount code "ADO2017" gives 20% off for Gold, Silver, and Bronze passes.

Open CFPs

lots of DevOpsDays

When the Levee Breaks with Jeff Smith and Mark Imbriaco Jun 04, 2017

Show notes

Bridget and Matty record at GOTO Chicago about communicating in the middle of an incident, with Jeff Smith, a production operations manager at Centro who previously helped build out the SRE team at Grubhub, and Mark Imbriaco, who has spent about 20 years in ops at GitHub, Heroku and DigitalOcean and joined Pivotal "as of yesterday." Jeff gave a talk in Bridget's DevOps track that walked through a live postmortem of a real incident at one of Jeff's unnamed employers, in a complex microservices architecture. Bridget says Mark, a brand new coworker, got dragged to Chicago because of the postmortems Mark has written for places everyone has used. The cold open is Jeff's line, "Don't worry, it's fine. You're fired."

Context Is Not State

Jeff's big lesson was about context. Alerting on everything can be information overload, so you have to frame the information to tell a story, because you don't want to assemble breadcrumbs mid-outage. The other trap is silencing all the annoying alerts at once, since the landscape of the incident can change, and "you end up fighting the wrong fire." Bridget notes that Bryan Cantrill's keynote made a similar point about alerts that depend on the very systems that are down. Jeff's example is a billing error that got an Azure account shut down, where the only alert was that the website couldn't be reached.

Mark says the context problem gets serious in long outages. The Heroku outage ran 67 hours for the last piece to come back, and most companies never think about what happens past 12 hours, when everybody goes into superhero mode. At Heroku they staffed shifts so that two people always worked the incident, offset by 50%, so that with an eight-hour shift, someone new arrived four hours in and could build up context while the previous person was still there. Jeff says a single person carrying an incident start to finish is probably storing all the context, which is dangerous for the organization and the individual.

Bridget asks why an oral handoff matters when documentation exists, and Mark separates "context versus state." State is what's up or down on the monitoring page, but the story that led there comes only from the person who lived it, who was too busy to write anything but notes, "marks on trees as they went through the forest solving the problem." Jeff adds that notes record facts, while context is also the theory being chased, and a test script left running overnight means nothing to whoever reads it. Mark adds "you don't write down the negative results," so the next person may walk the same trail.

Managing Up During an Incident

Matty asks about individual contributors on a bridge with someone seven layers up asking what's going on. Jeff describes Grubhub's common bridge with business and technical people, where business people need to message customers about orders and keep asking how long the fix will take, and "you have to give a number." Matty's answer is the Scotty principle, and Jeff's number is 10, with no units.

Jeff's fix is to separate the troubleshooter from the communicator. A troubleshooter gives periodic updates to one person, who becomes the parrot of that message, instead of repeating it each time someone joins the call. The other tool is a running Google Doc, which Jeff thinks was borrowed from the Google SRE handbook. People thought it was crazy at first, then the chat room link to the doc answered the status questions. Mark says the same idea shows up as an incident command role at Heroku and GitHub: the commander captures context and remembers that Jeff is looking at the thing. And "plot twist, that person shouldn't be the one who's communicating externally to your customers either," because choosing how to word a message about bringing a database back up, without saying "recovering" and making people think of backups, takes its own effort. Jeff adds that writing a 15-minute update takes 10 minutes, and that "jumping in to help isn't always helpful" if you haven't checked in with the incident commander, since you could undercut a debugging theory.

Matty asks what a sysadmin can do in an organization that has no incident commander, and suggests offering a struggling peer to act as a firewall for communication, then taking the result to management. Bridget suggests a shared doc built from the chat logs, which Bridget says aren't publishable because of the rabbit holes, and building it during the incident might prompt someone to ask whether they missed something with the database. Jeff points out that the first person to grab an incident is already the communicator, scribe and incident commander, so offering to help means asking "what do you need?"

Mark's advice for the person being harassed for status is to say "I don't know. I'll tell you something new in 20 minutes," and to give that update even if nothing has changed. Mark compares it to calling the cable company, where the cadence matters more than the answer. Matty calls acknowledgment huge for reassurance, and Mark adds "Not knowing is worse than getting bad news."

When an Outage Becomes a Disaster

Jeff asks how you mark the point where you concede you have a disaster rather than an outage, since a 20-minute update promises progress and a disaster declaration lets customers stop checking back every 15 minutes. Mark says "I don't know, but I know it when I see it," and describes stretching the cadence: if nothing new is expected in 20 minutes, say you'll update in an hour, or two, and promise to tell them sooner if you can.

Mark describes the Heroku case. In 2011 EBS went dark in US East, with a cascading control plane failure across multiple availability zones, and for the first part you couldn't launch apps or restart idled dynos, which took about eight hours to resolve. The long tail was the 200,000 Postgres databases, where some EBS volumes took up to 67 hours to come back or for AWS to give up. Jeff says that with consumer apps you can time a visitor's lifecycle and declare a disaster once the pizza order isn't coming. Bridget says sometimes you know immediately, and describes the 3:00 AM call from an on-call developer saying the HBase cluster was gone and Amazon said it was terminated. Bridget remembers knowing right away that the startup might be done, and it took days to resolve with broken backups, though the company did not go out of business and was later acquired. Jeff's takeaway is "the cloud is just someone else's computer," which doesn't absolve you of recovering from an Amazon failure. Bridget points listeners to the Who Owns Your Availability episode and tells them that if they haven't checked their backups, "you have Schrödinger's backups."

A Plan Before You Need One

Mark says that joining a company, one of the first things to look at is how it communicates during problems, because it's all about managing expectations. The plan covers frequency of updates, tone for the audience, like Heroku's technical readers versus Basecamp's small businesses, and canned responses. It should be prescriptive enough that nobody makes value judgments mid-incident, usually less than a page, and "you should be completely transparent," since misleading people and later finding you undersold the problem is the worst outcome.

Jeff says the way to test an incident process is to simulate incidents. Jeff has been thinking about a talk on pulling the environment's incident creation into the system, like Chaos Monkey for incident management. They ran Failure Fridays and War Room Wednesdays, and Jeff's example prompt is "I just socked Cassandra in the face." Jeff adds that it's best if someone outside the exercise says what's dying, and that with a War Room Wednesday coming you can pre-generate your templates. Bridget warns listeners to clear it with people before taking production down for fun. Matty says to start on paper: "Make your database cluster a sticky note on the board" and pull it off, as good a test as breaking the database. Jeff describes a tool called NetImpair that adds jitter, injects latency or cuts off a node's communications, which simulates failure from that node's perspective. Matty tells a story of turning off the wrong server at Allstate years earlier, with no way to decode the naming scheme, and waiting in the cafeteria to see if anyone came running.

Please Don't Be Me

Bridget says the reaction inside a company can't be "am I fired?" Jeff describes the week's AWS cost-savings cleanup, in which everyone agreed to delete 67 terabytes of logs on EBS volumes and shut down instances, which broke things. The engineer who ran it was paranoid about being walked out, and Jeff says you can't assume people know they're in the clear, so you have to reinforce it, before repeating the cold-open joke, "Don't worry, it's fine, you're fired." Matty adds that it has to actually be true first.

Matty says even when blamelessness is true, nobody believes it until they break something and don't get fired. Matty also says punishing mistakes doesn't reduce them: "It makes them become subject matter experts in hiding mistakes." Matty recalls John Cowie of Etsy on an earlier show asking "how amazing is it when the only thing that happens when you make a mistake is you learn something new?" and cites Etsy's three-armed sweater. Mark says you'll always have the first reaction of "oh my God, what did I just do," but the healthier one is "Please let that be me," because then you know the problem. Jeff says senior people can give air cover by publicly owning mistakes, and Mark notes that a GitHub co-founder wrote a public post about deleting the production database, thinking it was a test database.

Bridget says that even at a place that understood all this, the morning after the HBase loss came with the sinking feeling of being probably fired, and Mark says the healthy fear is whether you let customers down, not whether you will lose your job. Matty adds that someone who doesn't care about mistakes isn't someone to run the systems.

Jeff says a related burden is being on the hook for not knowing something about a system with 5 million lines of code, and Matty's example is the CTO who asks why you weren't monitoring for that. Jeff says "You're always fighting yesterday's war." Mark points to research by Richard Cook and David Woods: "The way that complex systems fail is fundamentally unknowable," so "Give yourself a pass to be human." Bridget says Tim Gross once sent an email quoting Etsy's principles of blamelessness after an outage when Tim was a director of operations at Drama Fever.

Dialects and Postmortems

Jeff describes a people-heavy practice: one scribe passes updates to key contacts across the organization, who translate them into their audience's dialect, so developers get a developer version and business gets a business version, and marketing gets what it will publish.

Asked for a final word, Mark gives a formula for public postmortems: apologize and mean it, demonstrate a thorough understanding of what happened, and describe what you're going to do, saying "we think it'll reduce the likelihood of this kind of thing happening again" instead of promising it never will. Mark says that can rebuild confidence to a higher level than before the outage. Jeff says to bring the people you communicate with into the process and ask what they needed that they didn't get, and that there are three axes for action items: reduce severity, reduce likelihood, and get better at detection.

Bridget and Matt chat with Jeff Smith (Centro) and Mark Imbriaco (Pivotal).

Jeff

Mark

Disasters!

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Velocity San Jose - discount code "ADO2017" gives 20% off for Gold, Silver, and Bronze passes.

Open CFPs

lots of DevOpsDays

Let’s do the devops again with Nicole Forsgren & Tim Gross May 19, 2017

Show notes

Bridget and Matty record at GOTO Chicago with Tim Gross, a product engineer at Joyent, and Nicole Forsgren, CEO and Chief Scientist at DORA. Tim kicked off the DevOps track the day before with a talk on software-defined culture, and Nicole gave a talk on how metrics provide signposts and goalposts on a journey to awesome. Matty, who missed both talks, plays the listener, and notes that recording six episodes in a day leaves about three months of content. The cold open is Nicole's LISA story: "Who do you think pays your bills?"

Measuring the Squishy Stuff

Matty says it's easy to think you can't put science behind culture: "You can't put a Nagios monitor on a human." Nicole says people often respond by proposing to pull measures about people out of HR systems, which won't capture what teams mean by culture, which is high trust, information flow and collaboration across silos. The alternative is a proxy, and turnover is Nicole's example of a weak one, since someone may leave for a better culture, a worse one, a partner's job, or $1 million. Chat logs work only if Slack is the only way people talk, and Nicole's example of what no system will catch is a coworker dropping off a Diet Coke at a desk.

Nicole's answer is psychometric methods, meaning survey questions, done in a research-based way. The Westrum model is the example, named for the researcher Ron Westrum, whose work shows that in high-risk, high-performance teams, a culture that values information flow, high trust, risk sharing and boundary spanning predicts performance. Nicole says the Westrum typology was rewritten, with Nicole's help, into survey questions, six in the short version and seven in the extended one, and that they are open sourced for teams to use every quarter. In the DORA findings over the previous four years it was one of the highest predictors of delivering software with both speed and stability, and it also predicted profitability, productivity and market share. The 2017 State of DevOps Report was due June 15th.

Small Teams, Moving Needles

Tim has worked mostly in small and mid-sized organizations and worries that a survey would have no meaningful sample size. Nicole says "you can do it with small teams. It still works." Matty, who did this with Nicole at Chef and with customers, says to scope down to the size of a feature team, and adds that Matty doesn't care what number a team lands on, since there's no magic score. What matters is that the parts important to you move, which means asking regularly, not once for a pass or fail.

Four Principles of Software-Defined Culture

Bridget asks Tim to walk through the four areas from the talk, and Tim says the four are reliability, operability, observability and responsibility, and Tim takes them out of order. Tim jokes that they're a map, not an array, and Matty adds that it will be unsorted every time.

Reliability

Tim says unreliable software has knock-on effects on the organization. People up all night because of bad on-call burn out and fight with each other, and chasing the shiny normalizes risky decision-making. Nicole says teams need to understand that taking risks is a safe bet and that risks will be shared. Matty cites Charity Majors, who would say that if you haven't broken production, you're not trying.

Nicole says the S3 outage was a favorite case, because the published postmortem described a fat-finger incident and "nowhere does it say human error," which speaks well of both the culture and the systems. Bridget takes from that that building for reliability implicitly means not blaming people when reliability falls short. Matty agrees that "you can't work around human error," since people are going to make mistakes and that doesn't scale. Tim adds that many system measurements are themselves proxies for culture. Uptime, for example, is often a proxy for what your people are doing, and Bridget says it may just reflect whether you patch. Nicole says "anything that's a metric becomes a proxy," representing something in someone else's head, and Nicole recalls that as a hardware performance engineer, response time was "my jam".

Operability

Tim says operability is about delivering quickly and about keeping an application's behavior understandable and self-contained with the team that owns it. Tim objects to the trend of pushing intelligence out of the application into a third party or a platform, because it creates a cultural imperative that it's fine not to understand these things, and it widens the gap between the platform team and the development team. Matty says black boxes let people treat a problem as someone else's, and supply an excuse: if I don't understand how that works, how could I have done it better? Bridget sums it up as "microservices are a game of point the finger and plausible deniability," and Matty says "All of IT is a game of point the finger." Nicole adds "Wait, you mean containers won't fix my culture? What?"

Nicole says metrics shape culture and can help teams communicate across boundaries, but they turn problematic when a team throws a container or an app over the wall with a metric that only makes sense to that team. Bridget asks what makes or breaks operability, and Tim says the measurements have to mean something to the consumers of the system, not serve as cover. A vague service uptime number isn't what a consuming team needs. They need to know whether they're being throttled, or whether clients should refresh service discovery. Nicole says to tie metrics to a line-of-business goal, and Bridget cites James Turnbull's argument in The Art of Monitoring that you are not the consumer of your metrics, which Bridget says ops-focused people forget when they build dashboards around what wakes them up.

Outcomes, Not NGINX

Matty says that outside the echo chamber the simple point still needs making: "Outcomes are like the only thing that matters," specifically the business outcome, and for a nonprofit, Tim adds, that is a mission. Matty recalls a sysadmin freaking out that SQL Server was using all the memory on the server, when that is what the memory is for. Matty also says to write a Chef test for whether a web server does its job and not whether it installed NGINX, and Bridget says the test should look at ports 80 and 443.

Matty retells a story Sasha Bates told on the Ship Show about working for a large retailer right before Christmas, when a product team wanted to push a release and Sasha objected on stability grounds. The boss's answer was "your job is not to keep the website up. Your job is to deliver the features that the company needs." Bridget's reply: "I think the company might need a feature of being up."

Nicole tells of chairing the LISA conference in 2014, when Courtney Kistler, who had been leading the Nordstrom transformation, stood in for a closing keynote speaker who had a medical emergency. The ballroom of old-school sysadmins was restless about a talk on business transformation, so Nicole and Tom Limoncelli told them to listen, asking "Who the F do you think you're keeping email servers up for?" After about ten minutes of business translation, Nicole says, the room was into it, and "Courtney won them over hard."

Responsibility and People

Tim says the fourth principle is about externalities. When talking to people about containers, Tim says they don't care about containers: they have a mission, and people whose lives should be fulfilling. Tim adds that people are not just there to fulfill the mission, and on the CEO's duty to shareholders Tim says "fiduciary duty, which is bullshit, by the way. That's not actually a law."

Nicole supplies data: over four years, the DORA research found that employees of high-performing teams are 2.2 times more likely to recommend their organization as a great place to work, and research from Harvard found that employees who recommend their workplace predict higher revenue growth. So "even if you want to be a selfish asshole," making the workplace better increases hiring, retention and revenue. Bridget asks the room who is hiring, and every hand is up. Nicole says hiring costs more than retaining, and "Don't be a jerk." Tim says Tim's gut says the same thing, and it's good to see data.

Observability and Debuggability

Tim says the observability section moves from traditional monitoring toward tools for exploring systems iteratively and collaboratively, not a lone sysadmin watching a dashboard. The stronger point was debuggability, which Bryan Cantrill discussed in the keynote. When a stack has a black box, whether the operating system, a platform or a web server's event loop, people end up saying "and then magic happened." That leaves software less reliable and is dissatisfying for technical people, Tim says, because "This is all software. It's not magic."

Nicole adds that it's not enough to have data: you have to act on it, and the highest paid person in the organization "sucks at this. Use your data." For teams without instrumentation, Nicole says to start by asking people. Can you roll anything out without asking other teams? Are you testing, have you shifted left on security? Bridget adds that if you can't deploy a microservice without deploying three others, you may have built a distributed monolith. Matty says to do it iteratively: one question is better than none, and it's information you didn't have yesterday.

Nicole says some things only surveys can give you. Systems can tell you what's in version control but not what isn't: "Only your people can tell you what is not in version control," and what is bypassing your systems. Bridget asks Tim about the automation Tim built to capture the state of an AWS account before a migration, and Tim says you can't capture everything, so you start from the top level. That meant documenting the network first, "because nothing else runs without the network."

Nicole cautions that instrumenting the easy thing for a quick win creates a pile of metrics, and once something is measured people start paying attention to it. Matty adds that it creates a culture that cares about CPU. Tim asks whether asking people questions has the same effect, and Nicole says it does, because collecting any metric sends the signal that it matters. The problem comes when it turns into a demand to answer 10 on a 1 to 10 scale or else. Matty says the survey for the car Matty had just bought worked that way, and did the same for a Microsoft TAM years earlier, where a scale of 1 to 10 was really pass or fail.

Closing Advice

Nicole says to start measuring, do it honestly, and keep measuring periodically: "Even a bad baseline is super powerful." Tim agrees and adds that you will chase what you measure, so measure the right things. Nicole is glad they agreed, and Tim says there was no screaming on this stage at all.

Bridget and Matt chat with Nicole Forsgren (DORA) and Tim Gross (Joyent).

Nicole

Tim

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Velocity San Jose - discount code "ADO2017" gives 20% off for Gold, Silver, and Bronze passes.

Open CFPs

lots of DevOpsDays

Old Geeks Yell at Cloud with Andrew Clay Shafer & Bryan Cantrill May 08, 2017

Show notes

Bridget and Matty sit down live in Chicago with Andrew Clay Shafer and Bryan Cantrill, right after Bryan's keynote, for a panel Bridget pitches as two people with a lot of perspective on where the industry has been. It goes where the title suggests. The next room over asks them to yell less, and Bryan's reply is "It's in the title, it says yell!" Along the way, "Genghis Khan" becomes a code name for Jeff Bezos and "Serpentor" one for Larry Ellison, which Matty asks everyone to use in all future subtweeting.

The Nineties Sucked, and Then Everything Closed Up

Bridget asks how we got into the pickle we're in. Bryan and Andrew dispute the pickle, but Bryan is clear that "the '90s really sucked": a very proprietary, closed era in which people thought systems were done. Andrew points to the dark ages of the relational database and the Java middleware stack that "totally paused everything for a decade." Bryan adds that Java was not open source, and that Windows was deeply proprietary, "then there's asshole proprietary." Bridget sees real willingness to open source coming out of Microsoft now. Andrew says Microsoft has been forced to, since it lost its monopoly.

A New Proprietary Era

Bridget says the proprietary era is over, and both guests disagree. Andrew says that once you're in the cloud, it doesn't matter how much open source built the stuff at Google or Amazon, because you can't change that code. Bryan says "We are in a new proprietary era," and that it rhymes with the '90s, when everything was going to Microsoft and doing anything else was stupid. Now everything goes to AWS. Andrew says "Nothing's more proprietary than Lambda."

Bryan argues that the great myth Amazon created is that cloud is a terrible business nobody should be in, when cloud computing has very good margins and Amazon's overall margin is very low. In Bryan's telling, AWS is underwriting a war on big box retail, and a lingerie retailer that was a Joyent customer answered Amazon's $9 bra with "no, forget it". Andrew says Amazon runs analytics on what sellers and partners do in its marketplaces and turns around with its own products on both the retail and cloud sides. Andrew calls Jeff Bezos "the Genghis Khan of the internet." Andrew remembers Rackspace saying it was playing a different game than Amazon, after throwing a Hail Mary with OpenStack, and Bryan says "Genghis Khan does not follow you on Twitter. He does not care."

The Serpentor detour begins with Bryan raising Larry Ellison's philanthropy and a longevity institute, and arrives at cord blood. It ends with Bridget remarking that vampires and zombies came up earlier in the day than expected.

Lambda, State, and the Borg Paragraph

Bridget asks what the two see among enterprise customers, given all the hype that Lambda functions will save us. Andrew says nothing ever goes away, and describes sedimentary layers of mainframes and Java middleware. Some places are like opening a time capsule, where "you can kind of tell what year they stopped learning." Bryan agrees that mainframes still run but are not a growth area, since nobody holds a conference of several hundred people on z/OS.

Andrew says people over-rotate on running a function. What Lambda represents is a fabric of event sources, and S3 is one, so "you can't build useful things with functions, stateless functions, until you have these things." Bryan says Lambda is "a needle exchange for AWS services," and Bridget adds DynamoDB as the other example of services built to keep you in. Bryan, from "stateless land," is asked whether there's state in the world, and Andrew says there is, along with all the hard problems. Bridget says state is all the customer data and money people do business because of.

Looking ahead, Andrew says new applications should be much more aware of their own state. Andrew recommends the Borg paper, which has pages on schedulers and then a single paragraph that Andrew considers more impactful than the best scheduling algorithm: every application running in Borg has an HTTP endpoint that broadcasts metrics about its health. Andrew says that if you did just that in your applications, "you'd get 85%, 90% of the benefit of the way Google runs their applications." Bryan calls Prometheus the Google idea taken to the open source world. Andrew adds that frameworks should make doing the right thing the easy thing for the app developer, and that people are already struggling to monitor Lambda infrastructure.

Bryan wonders how much Lambda will be used in anger rather than for prototyping, noting that the people who love utility billing are utilities, since you can't predict costs in a utility model. Bridget describes a Minneapolis meetup talk from SPS Commerce engineers, who used Lambda a year earlier and then built their own Lambda-alike in-house because Lambda billing didn't fit. Andrew says that when people cite cost savings from Lambda, it's because they were running mostly idle compute instances.

Open Source Doesn't Die

Bryan says "the open source business model is the second worst business model on the planet," and the worst is being a proprietary infrastructure software company. The Docker CEO had just been replaced that morning, and Bryan recalls the painful period at Joyent when parts of the stack weren't open source. On OpenStack, Bryan says the problem it was solving was a middle management problem in soon-to-be-dead infrastructure companies, and Andrew disagrees that this was the initial goal. Andrew says it became a weird political marketing exercise with little engineering in the core, and that it pulled attention from other promising projects. Andrew's history begins with Eucalyptus, which Andrew says had the birthright to be the open source cloud and mismanaged its community, so that "OpenStack would never have existed if Eucalyptus didn't mismanage its community in the beginning." Bryan says both made the same mistake by trying to be an open source AWS. Andrew's view is that Amazon's advantage wasn't necessarily software but the socio-technical system that had run a massive distributed system for a decade, and that many of the organizations trying to build clouds couldn't manage a multi-node Rails app.

Bryan argues open source outlasts any company: "because open source software can't die." The system Bryan works on descends from Unix, with parts 40 and 50 years old. Postgres was dead on the operating table for a long time, Bryan says, and was revived when things changed. Bryan insists that Illumos has been independent of its Solaris roots for seven years, and that Solaris "is just a very brief proprietary era in a much longer system." Andrew says the Linux and Solaris kernels differ in the quality of engineering, especially around observability. Bryan likes small communities that share values, and says large ones are a mixed blessing. Node.js is the example: Joyent was the company behind it, and Bryan says Joyent and the V8 team shared values around observability, debuggability and rigor that the broader community did not. The community wanted promises, which Bryan calls a bad model "in the operability of software years down the line." Bridget sums it up: day one is very short and day two is forever.

Doing It Right Now

Bryan says doing things properly initially makes you faster than the limit, and that executive leadership has to understand it. Bridget pushes back that doing it right the first time is seductive and impossible, since nobody has perfect future knowledge. Andrew says "there's nothing more expensive than building the wrong thing," and that sometimes testing a hypothesis with a hack that lives forever is cheaper. Bryan says the complaint isn't about prototypes but about the corners you know you're cutting, like being in someone's code that has never been executed and thinking it would have taken an extra 20 minutes. Andrew says every developer on every keyboard is choosing "between doing things right and doing things right now," every second of every day. Bryan passes on advice from a senior engineer that every line of code is a business decision, so an organization has to value not cutting the corner and "be a craftsperson." Andrew adds that values are about who gets rewarded, and that on a resume with stints of 18 months, 18 months, 18 months, each move often came with a big raise. Matty says the same pressure shows up as analysis paralysis among Matty's customers.

Integrity

The reward point sets Bryan off. Bryan says the leadership principles of Amazon and similar organizations leave out integrity, noting that Amazon has 14 leadership principles and integrity is not among them. Bryan contrasts that with corporate values of a generation ago, when integrity topped the list, and says "we have stopped aspiring." Bryan admires a final email to Sun employees saying that in 30 years there had been no need to hide the newspaper from the children, and sets that against Uber's behavior. Andrew asks whether Uber has been rewarded or punished, and Bryan answers that Uber's valuation is pretend. Andrew pushes back that many developers have mortgages and families and are managed like factory workers from the Industrial Revolution. Bryan says they are not, that "our children don't die," and that anyone using their brain in a cube farm is among the haves. Andrew answers that the choices people make to get through the day under imposed management structures aren't something Andrew will begrudge them.

Closing Advice

Bridget asks for a closing statement in 60 seconds. Andrew says "This might not be the podcast you wanted, but it's the podcast you needed," and that integrity starts with individuals: to build structures that have it, you have to think more globally about politics and the implications of decisions, and elevate yourself inside your organization or by creating new ones. Bryan says we live in a great time, making "castles from thought," and "We are literate in a society that is broadly not literate," so it's incumbent on us to increase literacy. Bryan adds "Anyone who bets against humanity is simply ignorant of history," and tells listeners to find their own motivation and take the luxury of being true to themselves. Andrew's last word is that the panel might not have yelled at the cloud, but "we definitely yelled."

Bridget and Matt chat with Andrew Clay Shafer (Pivotal) and Bryan Cantrill (Joyent).

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Velocity San Jose - discount code "ADO2017" gives 20% off for Gold, Silver, and Bronze passes.

Open CFPs

lots of DevOpsDays

Enterprises with Bryan Liles Apr 17, 2017

Show notes

Bridget and Matty talk about DevOps in a large enterprise with Bryan Liles, a director of engineering at Capital One who started as a sysadmin over 20 years ago and has done security, networking and development. Bryan says up front that this isn't Capital One's position, only what Bryan thinks while working there. Bridget met Bryan when Bryan was at DigitalOcean, and Bryan says the appeal wasn't the hot startup but getting back to the basics where Bryan started, at an ISP and a hosting provider, before wanting to do something bigger than Bryan could do alone, which a bank offers. Bridget also apologizes in the intro for audio problems in the recording. The cold open is Bryan's line that "When you have more than 2 lawyers, everything becomes harder."

Start With Why

Matty says that a couple of years ago the statement was that DevOps won't work at big corp, and now the question is how to do it at big corp, which is a fundamental shift. Bryan says how is easy, since you can read a book, and what people miss when they look at Google or Dropbox SREs is why: why they needed that solution and what issues you have. Without understanding why, you never know what success looks like. Even at Capital One, Bryan says, the DevOps teams bring in CI/CD, which is a tiny smattering of what DevOps is.

Matty describes a CIO buying the big-picture message and never getting it to the boots on the ground, who conclude a DevOps initiative means installing Puppet. Matty recalls Jez Humble saying Jez will have a job for life because nobody gets this, and says leaders are now asking people to come to a town hall to explain the principles, not the product. Bryan says to talk about CI, CD, logging, monitoring and declarative infrastructure as principles first, then bring in tools, and Matty asks how you pick a CI tool if you don't know why you're doing CI. Bryan says a big company will just tell you it uses Jenkins, with 20 or 30 people on it, and the conversation loses whether the tool is helping ship code or just being used. Bryan says developers are often given a problem with constraints and told someone else handles ops.

Strategy, Practitioners, and Constraints

Bryan says at a larger org leaders such as directors, VPs and CXOs should talk strategy and where they need to be, and let practitioners figure out how to get there, with a feedback loop. Because tools like Terraform and GitHub Enterprise cost a lot, higher-level people get involved, and should realize they are there for financial support and direction. Matty adds that outcomes are all that matter, and Bryan adds that they must be repeatable, since if you can do it on Tuesday you'd better be able to on Wednesday.

Bryan's view is that "Constraints are a great thing." People who say their startup's problems would go away with more money are naive, since projects with lots of money aren't guaranteed to succeed, and Google succeeded partly because it was trying to do it with fewer people. At Capital One, moving into AWS, they hit account limits, like how many VPCs can be peered, how many images you can copy across regions at a time, and how many security groups you can have. Bryan's example of copying an AMI between accounts, when you have hundreds of accounts and can copy five at a time, means deploying images alone can take more hours than a week has, which forces creative solutions that wouldn't have appeared with unlimited freedom.

Cloud Custodian and Why Enterprises Matter

Bryan says open source is hard at a bank, with lots of lawyers and regulation, but they created Cloud Custodian, a governance tool they released to the community. It enforces 100% encryption at rest: Bryan booted an instance with something unencrypted and got an email 30 seconds later, and again after trying a second time. It also lets you limit things like which instance types can be booted. Matty says that feedback loop through open source and vendors matters, since compliance ideas on a whiteboard differ from a customer's real compliance issues, and someone will be the first at a given industry's problem for Pivotal or Chef.

Bryan says enterprises are also where the money is for vendors, since companies like HashiCorp can't live on VC money forever and have to work with bigger companies. Matty had written off enterprises as where innovation goes to die, and being on the inside as a vendor changed that: it's harder, but the payoffs are better. Bryan says "we're a big, slow-moving ship, but guess what? When we turn around and we actually point in a direction, you get all the force of that ship." The problem is people: ten people have about 100 conversations, and 10,000 people have far more, and Matty says that means 10,000 conversations where good ideas come from. Bryan's favorite thing about working at a big company is that "we have internal tech conferences that are bigger than some conferences that I've been to," where people can talk about specifics without guarding. Matty says a healthcare customer's conference is bigger than every DevOpsDays combined, and that Matty can't go to the talks.

Enterprise DevOps and the Security Door

Matty says "let's all temporarily mourn the term enterprise DevOps and be glad it died," since it was really DevOps without the culture. What Matty sees now is that the entry point into an organization has shifted from ops or dev to security and compliance, since InfoSec people love it. Bryan says governance and compliance are hard at scale, and Cloud Custodian showed there is a marketplace for it, though compliance isn't Capital One's business. Bryan also notes that being in tech at a financial company means not needing to know how the company makes money, and Matty says at Chase the line of business only became clear on updating a resume: $1.3 trillion a day in wires went through the systems Matty managed.

The Changing Role of Ops

Bridget asks about Bryan's Twitter thread with Alice Goldfuss and Kelsey Hightower on ops. Bryan says there are two Bryans, the worker and the person, and the worker can do things wrong without being a bad person. Likewise, "Bad ops teams are teams that don't want to get better," as distinct from overloaded ones, and they need to go extinct. The advice is that you're responsible for your output, and if you can't effect change, "you shouldn't be in a place where you can't make positive change." Bryan says velocity is increasing, and if ops sits on the sidelines saying it's too hard, someone younger will take the job. Bryan is over 40 and expects to keep doing this because times change, from Sun pizza boxes to other people's hardware in data centers. Bryan says to change the solution: if the dev team throws things over the wall, use the SRE book's production readiness tenets and tell them you have production requirements as well as business requirements. Bryan says it's all about being the adult and owning the idea of change for the positive.

Closing Advice

For anyone at a large enterprise, Bryan says to distill a problem to the basics: you're not curing cancer, so solve one little change, then combine them. Startups and enterprises are similar, with a few more rules because more money is on the line. Bryan adds that a big enterprise gives women and people of color more opportunity to move up, and Bryan sees more Black VPs and women at the SVP level than ever before. Bryan leaves listeners with a question: what did you do yesterday, and how can you make today better, and to do something for someone you hadn't yesterday.

Bridget and Matt chat about devops in a large enterprise with Bryan Liles (Capital One).

Check Outs

Matt:

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

GOTO Chicago - Matt and Bridget hosting an entire day of Arrested DevOps Live! May 1-2 - $75 off with discount code "arresteddevops"
Velocity San Jose - discount code "ADO2017" gives 20% off for Gold, Silver, and Bronze passes.

Open CFPs

lots of DevOpsDays

Image credit

Startups with Charity Majors & Nicole Forsgren Mar 04, 2017

Show notes

Bridget talks startups with two returning guests who both started companies after the live episode at DevOpsDays Minneapolis the previous July. Charity Majors is co-founder and CEO of Honeycomb, which is building an observability tool for when monitoring or APM runs into a wall, after Parse and its acquisition by Facebook. Nicole Forsgren is co-founder and CEO of DORA, with Gene Kim and Jez Humble, and before that was at Chef and in academia. Nicole opens with a warning that if you ask about the research, the talking will start.

The Crossroads

Nicole says the July conference found Nicole at a crossroads, having left academia just before tenure and spent about a year and a half at Chef, while DORA had been a side project that was getting big. Other companies were approaching Nicole, who didn't know whether to join a bigger one or run the startup. Bridget told Nicole to talk to Charity, whose advice was to decide what Nicole wanted, and if taking a job, to be transparent about the other project and promise a company only a year. Nicole says being about a year out hadn't been articulated until then. Charity says it was clear Nicole was in love with the thing. Nicole took over DORA, committing to Jez and Gene, who had been trying to convince Nicole for months.

Charity says that at the time of the last conversation, about six months had been spent heads down after a rough co-founder breakup, without knowing whether what they had was worth doing. Around Minneapolis, they decided it was real, and time to push the baby out of the nest. "The first 2 months that you were showing your product to people, you should be humiliated. You should be embarrassed." Now they have their first paying customers and are working through roughly 800 signups from Charity's Twitter feed. Charity says what's hard is to shut up and listen, and that hearing Bridget pitch Honeycomb to others was painful and one of the most valuable moments of recent months. Nicole says the same about the channel partner program.

Why a Startup

Nicole says the startup did the hunting. People kept asking to compare themselves with the industry using the State of DevOps data, and some things can only be measured by people, not systems, so Nicole and colleagues built it. Nicole swore never to do a startup and is completely risk-averse. Charity had always been an implementer, not an ideas person, and had assumed a startup would come at some point since it seemed like a missed opportunity in Silicon Valley. The core truth, Charity confesses, is disappointment with the roles offered coming out of Facebook and a successful startup: employers wanted a proof period as an individual contributor or a manager of three people. After a career of saying titles don't matter, Charity found they do, and "it was rage." Bridget is also angry all the time, and it fuels the work. Charity also notes pedigree counts for a lot in Silicon Valley, and that Charity was never going to be more fundable than right then.

Charity says "Nobody's a startup person," roughly 100% of people, but once in a while something smacks you, and it isn't always the idea. Sometimes it's the team, and the idea comes later, and everyone retcons it as if they always knew.

Teams and Balance

Nicole says the partnership with Jez gelled, and "individuals don't make software, teams do," so sometimes when a person leaves, you take the team. Charity says when your strengths and weaknesses mesh with someone's, it's magic. Charity's co-founder Christine says little but is right every time, so Charity copies Christine on everything. Nicole describes the balance of the three founders between focus and creative openness. Charity adds a rule: "Christine and I never freak out at the same time." They joke about whose turn it is to freak out and who's going to be calm in the next conversation. For the first six months it was mostly Charity, who hadn't expected to be CEO, and then Christine couldn't carry them any longer.

Hiring for a Startup

Charity says the risk profile is different: at a small startup each hire is one of two or three people and materially shapes the product, which Charity says is true of everyone they've hired, up to seven or eight. You're building a family, technical skills can be taught, but caring about what you care about is hard to tease out and matters more than anything. Charity has never simply interviewed and hired, and who Charity wants to sit next to eight to ten hours a day matters. A big company needs a filter, Charity says, but Charity criticizes Facebook for saying they couldn't lower the bar while having five women in production engineering out of 350, and admitting they found no correlation between interview scores and success. Bridget says a diverse interview panel prevents unconscious bias. Charity says "Great teams can fight well," like couples, and Nicole adds it's how well you resolve conflict. Charity favors transparency below ten employees, including cap tables, and shielding people only from chaos that would distract them.

Funding and Staying Lean

Charity says they took $2 million because it was offered on very good terms, are extremely fiscally conservative, and are racing to profitability. When people celebrate raising $100 million Charity sees failure to execute with what they had, and says giving up equity and control is often the unmentioned part. They are raising enough to get comfortably to break-even, so they can walk away from terms. Nicole says DORA is revenue-funded with no investors, which is the best way if you can do it but limits growth, so Nicole uses contractors for accounting, design and copy editing and is feeling scaling challenges. Bridget mentions Honeycomb becoming an ISV partner with Pivotal, and partnerships as a way not to hire.

Charity says startups need generalists who are fine changing their minds, and that you don't hire into a role until you've done it enough yourself to know what makes someone successful and have six months of work that justifies it. Charity adds that success shouldn't be measured by headcount but by what you deliver divided by the number of people, and is irritated by VCs and acquirers asking how many bodies Honeycomb has, while turning away world-class engineers.

Advice for Joining

Charity says the younger the startup, the more ownership comes with it, and you should look for founders aligned with where you want to be. Charity dislikes founders with a "them and us." People should ask far more questions, since many don't know to ask how many shares are outstanding, and look for ways to enrich the relationship before signing, perhaps with a trial period, or by taking friends who know out for drinks. Nicole's advice is to know who you are. Nicole is a big-company person who likes titles and transparency, and jokes about the nickname Crusher of Dreams, earned by spotting things that won't work and needing to be high enough to say so, with a filter that drops in front of the mouth. Charity adds that in your first 10 years you should push yourself, trying big and small companies, and "Running towards being uncomfortable is, I think, generally good life advice." Nicole says give it six months, and Charity says learning to tell good uncomfortable from bad uncomfortable is part of growing up, and that after staying a year and a day at two startups, Charity would now cut it short in a week.

Bridget chats about startups with Charity Majors (Honeycomb) and Nicole Forsgren (DORA).

Check Outs

Charity:

Honeycomb has a public announcement/launch on April 11
@goturbine for splitting traffic
also check out buoyant.io

Nicole:

DORA has an ROI white paper coming out soon
State of DevOps Report releases June 7
Book with Jez and Gene; early release is summer/fall

Bridget:

Systems We Love - coming to Minneapolis March 16th!

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

GOTO Chicago - Matt and Bridget hosting an entire day of Arrested DevOps Live! May 1-2 - $75 off with discount code "arresteddevops"
Velocity San Jose - discount code "ADO2017" gives 20% off for Gold, Silver, and Bronze passes.

Open CFPs

lots of DevOpsDays

Microsoft Redux with Liam Bennett, Brandon Olin, Reuben Dunn, Glenn Sarti, and Chris Hunt Jan 26, 2017

Show notes

Almost two years after the first episode on DevOps in Microsoft environments, Matty and Trevor bring back a panel that Matty bills as the largest in the show's history. Trevor's microphone fails almost immediately, so Trevor mostly appears through Matty's paraphrase and a checkouts segment near the end. The panel is Liam Bennett, who works at a managed hosting provider on Windows workloads across AWS, Azure and GCP and released many of the community Puppet modules from time at OpenTable, Brandon Olin, a systems engineer at Columbia Sportswear, Reuben Dunn, DevOps practice lead at Freedom in New Zealand, Glenn Sarti, a senior developer at Puppet specializing in Windows, and Chris Hunt, a Windows platform engineer at Ticketmaster who runs a DSC implementation approaching 1,000 nodes on a pull server. Matty works at Chef, after a pre-vendor career mostly as a Windows sysadmin. Liam's cold-open line is that "we're in this age of just kind of madness."

What's Working

Chris says DSC works, with the caveat that it's challenging. Brandon says it's automation pipelines and ChatOps to expose tasks to groups who don't have the skills or access. Glenn says PowerShell has been an absolute saver, and Reuben adds PowerShell and Chocolatey for distributing everything from desktops to test environments. Matty notes that a robust shell and package management are what the Microsoft world lacked, and that the old automation in VBScript was really macros. Brandon says they don't like to talk about the VBScript days, and Chris says it took about 10 years before they had a way to package and distribute modules.

Open Source Comes to the Windows World

Matty asks about sharing code in a historically closed-source platform. Liam says many organizations needed Microsoft to make the first move, and that "it required Microsoft to make that first move," because after Ballmer stepped away in 2014 the company got permission to change, and that gave everyone else permission. Chris says Microsoft had a history of building its own version and killing an open source project, and now seems to contribute instead. Glenn says that having worked for banks, military and government, contributing outside the firewall is still a big challenge. Brandon says using open source is a lower barrier than contributing back. Matty says it's often lawyers worried about setting precedent for other IP, and about telling apart plumbing from what's different. Chris says permissive licenses like MIT have helped, since companies now separate plumbing code from business code.

The .NET Developer Stack

Matty asks whether anyone deploys with Microsoft's stack. Reuben's team uses TeamCity, not VSTS, and objects to building differently on a server than locally, and to the one-pane-of-glass developer experience and right-click publish: "their definition of done, when they're finished is just a package," which is fine for a monolith but makes ops people mad in distributed computing. Matty recalls the stack being built around a developer with a codebase spraying it somewhere, config transforms and a per-seat license of $8,000 to $10,000 for every sysop who needed to deploy, so everything ended up manual.

Liam says culturally "you win most of these places over just by making it easier." Once you beat the Microsoft tool once or twice, people begin to question the rest. Liam sees TeamCity and Jenkins increasingly, and the source control part of VSTS dying quickly as Microsoft itself moves to Git, with the Windows core apparently 40 billion lines of code. Matty ties it to the old "Microsoft is the answer, what's the question" attitude, and to Microsoft's own not using SCOM to monitor microsoft.com. Matty adds that the company has improved by decoupling, and that its leadership and engineers get it, while middle management and sales might not.

Bringing Puppet and Chef to Windows

Glenn says some people used Windows in the dark days and never want to touch it again, and some new ones love Visual Studio, and Glenn has even met a Linux admin who loves PowerShell. The bigger shift is teaching people to model architecture, since Windows admins are used to next, next, next, finished. Matty compares it to Microsoft's own Office for Mac lesson: a tool should fit where its user's comfort zone is. Glenn says Chef and Puppet need to do a better job onboarding Windows people.

Matty recounts Jeffrey Snover's line that Linux is a document-based operating system and Windows is API-based, which is why IIS, being document-based, is the easy example. Bootstrapping is easy on Linux with SSH, while enterprises turn off WinRM, and securing it means running a command and setting registry keys, not dropping a file. Brandon says onboarding matters because if Windows admins hit roadblock after roadblock, they'll give up. Glenn praises the Docker for Windows installer as the best of both worlds. On DSC, Chris says it's "delightful for about one server, and then after that it starts to go downhill," and Brandon says you need a management platform, which is where Chef and Puppet come in. Matty quotes Snover that DSC is a printer driver and Chef or Puppet is Microsoft Word.

Maturity, Culture, and Table Stakes

Liam says initial usability of the tools is pretty good now. The roadblock is the second and third step of maturity, managing a whole fleet, and that story isn't told well. It depends on the company: if config management is in an internal IT organization that doesn't think of IT as its business, the jump feels epic. Matty says it's Conway's Law, and that in a low-trust culture people argue with math, as Adam Jacob says. Matty also notes that what vendors consider table stakes, like testing infrastructure code, isn't. Brandon says quite a few admins still use ".bak as their source control system," and one colleague with 30 years in IT said they were learning their job all over again.

Chris thinks Linux configuration management came from developers and Windows from ops. Matty has seen many Windows admins shred PowerShell and many Linux admins who want a click-button tool, and "being bad at command line knows no operating system." Liam calls this a golden age of managing Windows, with config management, Terraform and the PowerShell gallery, yet just as people reskill, the industry pulls the rug out with containers and .NET Core on Kubernetes. Liam sees mostly the other end, legacy apps never written for this, and cites that apparently 30% of Windows workloads on AWS are 32-bit.

Advice and Daily Drivers

Matty's advice is not to boil the ocean or fall into analysis paralysis: don't design high availability before you have one node under management, write some code and improve later, and avoid yaks. The principles, like continuous delivery practice and thinking of infrastructure as code, carry over when tools change, and they are harder than learning any syntax.

Asked for their daily drivers, Brandon uses an old Mac Pro running Windows 10 through Boot Camp, and PowerShell is the indispensable tool, Chris a corporate Lenovo with Visual Studio Code, Glenn a MacBook Pro with Windows 10 and no macOS, with Hyper-V and VS Code, Liam Ubuntu with Atom and a Windows VM, and Reuben a Surface Pro 3. Matty notes it is ironic that VS Code's Git integration is better than Atom's. The checkouts include Liam's pick of Terraform and a forthcoming book on it, Brandon's Operation Validation Framework and Pester, Reuben's Serilog and Seq, Glenn's Neo4j and Flow Perth's hack days for nonprofits, and Chris's GitKraken.

DevOps Cafe w/ Jeffery Snover - Linux is docs based, Windows is API based

Check Outs

Liam

James Turnbull Terraform Book
Black Mirror - My latest show to binge watch

Brandon

Infrastructure testing with Pester

Reuben

Paramore, aka Brighter - My goto reference implementation for understanding how to write robust “microservices” that gives you options
https://serilog.net/ & https://getseq.net/ Logging is the “new” debugging...
Phil Haack - Be the scientist

Glenn

neo4j Graph database- Works on Windows too!
Free OReilly ebook on graphdatabases
Flow Perth create unique events in the technology community bringing skilled volunteers and not-for-profits together

Chris

I’m a big fan of GitKraken. I can actually do some useful things with Git without spending a couple hours reading man pages.
termeter - A Go app for “rendering” ascii graphs in the console.

Trevor

Blue Raspberry Portable mic I’m looking at getting, tired of lugging the Yeti around (much like the 17” laptop I abandoned for my Surface)
Factorio - super fun collaboration game
pvpgn open source classic Battle.net + Westwood server, got it running- trying to stand it up inside habitat now

Matt

Maybe weird to talk about Apple-only app on the Microsoft show, but I’m now enamored with Bear Writer - just a nice markdown notes thing.
I’m kind of obsessed with Golang now - the go Fundamentals Pluralsight course by Nigel Poulton was pretty dang rad

devopsdays Cuba 2016 Jan 17, 2017

Show notes

Bridget and the show's audio editor, Joe, who joke that if this podcast were the Beatles, Joe would be Ringo, talk about the first DevOpsDays in Havana, Cuba, and share audio recorded in October at the closing session. The event was organized by Rudy of DevOpsDays Ghent as a joint venture between the University of Ghent and the University of Information Sciences (UCI) in Havana. Bridget and Joe joined fellow DevOpsDays organizers from Belgium, Patrick, Bernard and Rudy, and Mike from Dallas, whose fluent Spanish translated for the group, which Bridget says came in handy when explaining vegetarian food. Joe, who was the host for the event, needed some ancient high school Spanish too.

Bridget says the university is worth a visit: it has a giant Android lab and its own Linux distribution, and it sits about a 30-minute drive out of Havana on an old Soviet military base, along a road that was once used as an airstrip. Joe warns the audio sounds like a big echoey room because it was one.

The Closing Session

The recording begins with Hugo, an organizer, closing the fourth day, an intense few days in which Hugo saw people learning a lot, and says of the Ignite talks that "everybody was on fire." Hugo thanks the invited speakers, including Patrick, whom Hugo calls one of the founding fathers of DevOps, which is a rare opportunity for Cuba, and who answers "Two words: thank you." Hugo also thanks Bernard, Bridget, Mike for making everything understandable, Joe for the podcast, the local team, and a sponsor that covered all the expenses. Bridget adds that none of it would have happened without Rudy, who has spent years building the Belgian and Cuban cooperation, and leads a standing ovation.

Attendees then line up, in Spanish, with Joe translating in the episode. One says the event was great, interesting, instructional and intense, and that some of what was discussed will make life better and some will make it worse because they'll want to rush to implement it, but that the best part is the friendship. Another says the best part is the sharing of information, knowledge and experience among professionals. Maria Lina thanks the invited guests for dedicating their time and knowledge. A representative of UCI says the event helped identify bottlenecks and important things to do. Bernard says it is a pleasure to have been invited to be in Cuba, with "good weather, good food, good company, good conference."

Patrick and the Future

Bridget asks Patrick Debois, who started DevOpsDays in Belgium in 2009, what was different about this one. Patrick says in other countries people are saturated and wonder if they can learn one more thing, but here people started sharing from the first exercise, with the same enthusiasm as in 2009, when everyone left to take it home to their friends. Patrick says the retirement from DevOpsDays became official two years earlier, and that Bridget is now in a good lead, helping people organize elsewhere.

Bridget asks Enrique what comes next, for the conference and for DevOps in Cuba. Hugo answers, and Joe translates, that that the survey results are coming, everyone wants to do it every year and to keep developing the cultural movement of DevOps, and the idea is to share, and to work with all companies to advance.

Ignite in Spanish

Joe, a fan of the Ignite format, says the Cuban participants had never heard of it before they were supposed to give them, and took to it like fish to water. Ignite is a five-minute talk with 20 slides that advance automatically every 15 seconds, which is difficult for first-timers. Bridget says English-speaking Ignite speakers manage about three sentences per slide, while the Spanish speakers were probably putting about six or seven. Joe says they were going a mile a minute, and even catching every fourth or fifth word, they were really good. Joe singles out Henry, the "on fire" speaker, whose Ignite was one of the lead-offs, and Maria Elena, whose evening talk on telenovelas was very funny.

Bridget says of the year's conferences on five continents that this one stands out as significant. The show ends with the banana stand line in Spanish, and Bridget's note that in Cuba the bananas are really plantains.

Bridget and Joe discuss their experiences at devopsdays Cuba and share audio from the closing session.

Dramatis Personae

Rudy Gevaert @rgevaert
Patrick Debois @patrickdebois
Bernard Grymonpon @wonko_be
Mike Rosado @MikeRosTX
Enrique Carbonell Muela @kikicarbonell
Genry Leyva González @genrylg
Marialina Ballesteros Hernández @MarialinaBall

Further Viewing

A few videos of devopsdays Cuba have made it to youtube. They can be found [here.] (https://www.youtube.com/watch?v=bTMwHcjpMf4&list=PLobspijdw3822IEFotAHz_vWrc0nTqwxn)

Special thanks to Mike Rosado for his help in preparing the translations for this episode.

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Open CFPs

lots of DevOpsDays
ChefConf 2017 - closing on January 18, 2017
Monitorama - May 22-24 - until Feb 1

2016 Year-End Extravaganza Dec 31, 2016

Show notes

Matty, Trevor and Bridget record a hosts-only year-end episode in December 2016, and most of it is a long tangent about podcasting itself. The agenda, such as it is, covers favorite episodes, site and listen stats, what each of them did in 2016, and what they think the year meant for DevOps. The show notes are thin and Matty knows it. The cold open is Bridget: "Oh my God, we're not going to start talking about the process of podcasting again. Moving on."

The Cold Open Supercut

The episode starts with a supercut of the year's cold opens, which Matty describes as the lazy route: in the first year-end show Matty tried to make a highlight reel and had to listen to every episode, but now "you only have to listen to the first 5 seconds of our show." Matty also says the cold opens only started this year, and that Joe, the new audio editor, picks them and keeps surprising Matty.

Why the Audio Isn't Better

Listeners keep asking for better audio quality, and the hosts agree Hangouts is a poor choice but the "lowest common denominator": nobody has ever said they couldn't use it. They've tried other tools at least half a dozen times, including TriCast and Zencastr, and Trevor says that "every time we actually go to do it, something goes catastrophically wrong," usually with three hosts and a panel of guests. Matty says the show is unusual, since most podcasts have one host or a host pair in a room, and a panel show with few repeat panelists is "0.00001% of podcasts out there," so the software isn't built for it. Bridget's takeaway for anyone thinking of starting a podcast: "software is terrible." Matty also notes that two-person episodes are the easiest to edit, and that pre-recorded sponsor pre-rolls replaced the hosts reading them live.

This was also the year of cross-podcast episodes released on other shows' feeds, with the Goat Farm and Software Defined Talk, and a DevOpsDays Dallas super-episode with Software Defined Talk and Food Fight. Matty floated a roundtable of podcast hosts talking about how they make their shows, and every host who replied said nobody would care.

Favorite Episodes

Bridget's favorite is the first episode of 2016, on platforms, which set a bar the year managed to live up to. Trevor picks the DevOpsDays Dallas episode and one recorded in Singapore, where people "are getting all the concepts that we're talking about now, but they didn't have to go through all the pain." Bridget compares it to skipping landlines and going straight to cellular.

Matty's picks:

Personal brand: the episode that reached friends and family outside the usual audience.
Who Owns Your Availability: recorded within hours of the left-pad incident after Bridget said it had to happen that day, with a guest who was about to board a plane to Japan and was told to do it now.
CareerOps: an episode that came entirely from a listener's idea about disaster recovery for your career, which Matty uses to repeat that guests should arrive with a topic. "we have a long list of people and a short list of topics."
Operationalizing open source: an experiment where Michael Hedgepeth hosted and the hosts were panelists, after a proposal that ran ten paragraphs.

Matty adds that the show gets "one episode a year to be self-indulgent," and this is it.

The Numbers

On the numbers, Matty notes that the website had about 16,000 unique visitors, almost half of traffic comes from search, and Twitter accounts for 6 percent. The number one search term is Arrested DevOps. 1 percent of traffic came from being listed in the Hugo gallery, and another podcast turned out to be using the show's Hugo theme, which Bridget discovered through a Google alert on a name. Matty then wrote a Hugo theme for podcasters. Listens were 232,200 against 204,001 in 2015 with about the same number of episodes, so Matty reads audio popularity as roughly flat. The most listened-to episode was Application Configuration with Adam Jacob and Tim Gross. The most watched video was Bridget's fireside chat with Bryan Cantrill, at almost 2,000 views, then the containers and security episode with Jess and Ben Hughes at about 1,200, then the Jeffrey Snover episode recorded in 2014. Matty credits big personalities and topics people are hungry for, and that if every episode were a chat with a personality the show wouldn't have lasted, and without any it would have burned out.

Matty also tells the story of a blog post that put a personal site on the front page of Reddit, and after Bridget asks, agrees to take down a post a commenter had asked to have removed.

What Each of Them Did in 2016

Matty traveled little, lost status on everything, spoke twice (Pink16 and a CloudBees Jenkins conference in Chicago), attended one DevOpsDays, Chicago, which Matty ran, and got married. Trevor spent months leading a data center transformation in Asia-Pacific, saw Hong Kong, Singapore and Tokyo, and says staying on Singapore hours "broke me in some way," which is when Trevor understood burnout. Trevor spoke at the PowerShell Summit in Singapore and at ChefConf, moved to Los Angeles and joined Chef in the same role Matty holds. Trevor also recounts applying for an evangelist job at the lowest point of the burnout, and Matty adds that it was mature of a hiring manager not to hire for the wrong role at the wrong time. Bridget's advice is not to compromise on what you'll be happy doing and where you want to live.

Bridget gave about 25 talks, visited DevOpsDays in London, Toronto, New York, Detroit, Havana, Philadelphia, Madison and Sydney while running Minneapolis, and with Joe covered five continents. The plan had been to travel less, which became "the gamification of poor life choices" once the airline status arrived. Matty says the remote-work life has made travel something to dread, and that roles built on being in the room with customers make it hard to avoid. Trevor calls elite status "the worst wonderful thing in the world."

DevOps in 2016 and DevOpsDays

Matty's read of the year: "this was a year when we stopped talking about doing shit and we just started doing shit." After the empathy talks of 2014 and the thinking of 2015, people in the open spaces at DevOpsDays Chicago were talking as peers who were all doing it, instead of sitting at the feet of thought leaders.

Bridget says DevOpsDays grew from about 22 cities in 2015 to 42 cities on 6 continents in 2016, with first-time events in places like Istanbul, Porto Alegre, Raleigh, Kansas City, Philadelphia and Cape Town, and Moscow, Beijing and Zurich planned for 2017. Matty joined the core team as web team lead, doing the work behind the scenes of the site so that updating it is easier, while deliberately keeping the Git requirement: "we want to make it delightful but not easy." Bridget has notes on the mockup colors, asking only for better contrast.

Matt, Trevor, and Bridget chat (at length) about podcasts, podcast recording, and podcast recording software. Oh, and the highlights of 2016 if they get around to it. (Don't miss the supercut of all 2016's cold opens, which was edited by Joe, even though Matt takes credit for it!)

What were some of your favorite episodes?

Bridget: So many great ones! Kicked off the year with Andrew Clay Shafer and Kelsey Hightower, and we kept going at that pace!
Trevor: Soon-to-be-finished Singapore PowerShell Summit Episode
Matt: Personal Brand episode, Who Owns Your Availability (Bridget says “omg, left pad, we need to have this episode NOW”), CareerOps

Let’s talk numbers

Website

16K visitors to the website
46% of traffic comes from search
16% comes from referrals, mostly Twitter (Twitter is 6% of all traffic)
1% of our traffic comes from the hugo site itself (our site generator)

Episodes

232,200 listens in 2016 (204,001 listens in 2015)
Most listened-to episode in 2016 was Application Configuration with Adam Jacob and Tim Gross
Most watched YouTube video in 2016 was Bridget’s Fireside Chat with Bryan Cantrill

Website updates

Didn’t change much, but we list episode numbers now (maybe we’ll add dates sometime)

Other improvements

Matt did a session with Daniel J. Lewis of the Audacity to Podcast (we were a featured podcast eval in Podcasters Society) and we got a whole bunch of ideas on how to improve the show - some of which we have already started to implement). You can check out our outstanding issues on GitHub
Welcome to Joe as our main audio editor!

What happened with you in 2016?

Matt

Didn’t travel much - only spoke once twice - once at Pink16, and once at a Cloudbees conference in Chicago
Only went to one devopsdays - Chicago. I’m failing!
Hey, I got married

Bridget

28 talks. Give or take. A few of those were at devopsdays - I made it to London, Toronto, New York, Detroit, Havana, Philadelphia, Madison, Sydney - and of course ran Minneapolis.
Counting North America Joe and I hit 5 continents this year. And I was going to travel less. Instead, all the airline status. Rethinking next year. Maybe get into webinars?
Devopsdays: growing! New core team! New cities!

Trevor

I spoke 3 times this year, but got to attend so many more conferences than before
I was literally on the other side of the planet to my usual place of existence for the first time.
I moved to LA
I work for Chef now!

Check Outs

Bridget

Systems We Love - videos at https://twitter.com/SystemsWeLove/status/809117528374972416

Trevor:

[Westworld](http://www.startrek.com/article/klingon-mug-and-collector-lapel-pins-ready-to-beam-up Klingon Blood Wine Mug)

Matt:

There’s this thing called Minecraft I “discovered” via my kids.
Also reminder about code.org and everyone is talking about Hour of Code
It’s been super retro time for Matt as he re-read a bunch of the Dragonlance books, and discovered a bunch he hadn’t read, including The Soulforge.

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Open CFPs

lots of DevOpsDays
Velocity San Jose until Jan 10th
ChefConf 2017 - closing on January 18, 2017
Monitorama - May 22-24 - until Feb 1

photo credit 1, photo credit 2

devopsdays Sydney 2016 Dec 06, 2016

Show notes

Bridget records a live panel at DevOpsDays Sydney, held in the Sydney Masonic Centre, as a co-production with Software Defined Talk, whose Matt Ray guest hosts. Matt Ray moved to Sydney in July to build a footprint for Chef and do evangelism, and says Chef sponsors as many DevOpsDays as it can to build community. The panel is Katie McLaughlin, a first-year organizer who ended up emceeing the whole event, Lindsay Holmwood, who spoke first that morning, Matthew Jones, an organizer who runs the Melbourne meetup, and Mick Pollard, who goes by Aussie Linux and is attending. It's Bridget's first time in Australia, and Bridget wants a photo on one of the building's many thrones.

Where Sydney's DevOps Community Came From

Lindsay and Mick started the first ever DevOps meetup, in February 2010, and Matthew says Sydney's DevOpsDays in 2010 was the second one outside Ghent, making it the longest running. Mick says the reason was that at the end of 2009 Mick worked at a startup that missed its next round of funding, and everyone was fired one morning. With no name in the industry and no community events behind them, Mick did the hard slog of job hunting and decided no one should have to. So the meetup would be inclusive, not just for practitioners but recruiters and anyone else, and helping people find jobs is the only reward Mick needs. Lindsay says they're actively inclusive of recruiters, on the condition they don't cold-call the members, and sees them as allies.

Katie says the event has skipped beer at the first-day activities, with bowling instead, for the past two years, which helped attendance the next morning. Lindsay gave an opening talk at the 2013 Sydney event, and says this year's attendance was spectacular by comparison.

A Conference Where Everyone Participates

Matthew likes that half of DevOpsDays is open spaces, shaped by attendees and not organizers. Lindsay says there are no silent witnesses, and people who participate most enjoy it most. Matt Ray, as a vendor and former organizer, says a table isn't the point, since you have to show up and talk. Lindsay says at corporate events the contrast is stark, and that at DevOpsDays a vendor person is just an actual person.

The challenge is selling open spaces in Australia to people who haven't been. Mick passed along feedback from someone asking "why am I giving you money to come to a conference that has no speakers." Bridget says about 85% of any room has never been to one, so the stock template now says attendee-suggested breakout sessions instead of open. Katie calls it "a supercharged hallway track and some talks," and says the conference emphasizes that talks are recorded so you don't have to attend them all, and people travel to meet people. Mick heard a non-technical attendee say they felt very involved.

Topics: Lambda, Business, People

Lindsay's morning talk was about building teams as complex distributed systems made of humans. Lindsay came for the technology in Ghent in 2009, when people wanted agile systems administration, and found that "none of the technology problems are the hard ones in our industry. It's actually people." Matt Ray says the talks break into technology like containers, and how to do DevOps with people in the enterprise, like getting your boss to listen. Lindsay says functions as a service is a fundamental game changer, and expects the talks in two or three years to be about different technology. Matt Ray says last year and the year before were about containers, and "The people problems never go away." Matthew notes recurring topics of DevOps and the rest of the business, including an Ignite about finance and DevOps, and security. Katie says language communities are picking up on culture, hiring, diversity and inclusion, and that a linguist gave the closing keynote, and adds that organizer burnout is real, which is part of why Katie volunteers.

Australia and the US

Matt Ray doesn't sense the same fear of competition in Australia as in the States, where Amazon and Walmart are torching retail. Lindsay pitches changing technical practices to government by saying you want to go faster to deliver value and meet user needs, and that "going fast and being safe, they are not mutually exclusive." The Puppet survey stats show fast teams recover from failure faster, and Matt Ray calls it compliance at velocity. Lindsay says the government's Digital Transformation Agency supports hundreds of apps with a team of two. Matthew says Australians are open at meetups, but US and European companies moving in are bringing an NDA culture, which Matthew is mindful of as an organizer.

Where DevOps Is Going

Mick says more effort is going into what developers deliver and less into the operating system: "The OS layer is really more a utility these days," like a tap you turn on without wondering where the water comes from. Katie, who is newer to ops, says there is still gap filling and legacy work, since not everyone is running containerless Lambda functions. Lindsay says operations will consolidate, with fewer of those jobs, so learn at least one other programming language, since you can help developers who now can get things running easily. Matt Ray adds that there are more places things can go wrong, so operability wisdom still matters, and that "there's always gonna be servers somewhere." Matthew says the future is about people, and adds that everyone talks about business value but should remember the customer who pays you. Bridget ties it together with the electricity generated on a river in Appleton, Wisconsin, and says you decide what to abstract away, and which to pay someone else to do.

Bridget and special guest host Matt Ray of Software Defined Talk chat with Matthew Jones, Lindsay Holmwood, Mick Pollard, and Katie McLaughlin at devopsdays Sydney 2016.

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Open CFPs

DevOpsDays Baltimore - closing on Dec 12, 2016
ChefConf 2017 - closing on January 18, 2017
Velocity San Jose until Jan 10th
Monitorama - May 22-24 - CFP opening soon

Our TOPPODCAST Picks

Follow Us

Stay Connected

Organizing a First devopsdays

Foreground, Background, and Deep Work

Incident Commanders and Backups

Keeping Questions in Public

Roderick & Arthur

Aaron

Community & Event Stuff

Upcoming conferences

Open CFPs

Context Is Not State

Managing Up During an Incident

When an Outage Becomes a Disaster

A Plan Before You Need One

Please Don't Be Me

Dialects and Postmortems

Jeff

Mark

Community & Event Stuff

Upcoming conferences

Open CFPs

Measuring the Squishy Stuff

Small Teams, Moving Needles

Four Principles of Software-Defined Culture

Reliability

Operability

Outcomes, Not NGINX

Responsibility and People

Observability and Debuggability

Closing Advice

Nicole

Tim

Community & Event Stuff

Upcoming conferences

Open CFPs

The Nineties Sucked, and Then Everything Closed Up

A New Proprietary Era

Lambda, State, and the Borg Paragraph

Open Source Doesn't Die

Doing It Right Now

Integrity

Closing Advice

Community & Event Stuff

Upcoming conferences

Open CFPs

Start With Why

Strategy, Practitioners, and Constraints

Cloud Custodian and Why Enterprises Matter

Enterprise DevOps and the Security Door

The Changing Role of Ops

Closing Advice

Check Outs

Matt:

Community & Event Stuff

Upcoming conferences

Open CFPs

The Crossroads

Why a Startup

Teams and Balance

Hiring for a Startup

Funding and Staying Lean

Advice for Joining

Check Outs

Charity:

Nicole:

Bridget:

Community & Event Stuff

Upcoming conferences

Open CFPs

What's Working

Open Source Comes to the Windows World

The .NET Developer Stack

Bringing Puppet and Chef to Windows

Maturity, Culture, and Table Stakes

Advice and Daily Drivers

Check Outs

Liam