Arrested DevOps

Latest Episodes:

Tis The Season...For Scaling! with Rob Cummings and Matt Curry Nov 13, 2015

Show notes

Bridget hosts this one solo and talks holiday scaling with two people who have been through a lot of peaks. Rob Cummings has spent 10 years at Nordstrom and, since January, supports the operations teams for nordstrom.com. Matt Curry is Director of Platform Engineering at Allstate, and before that spent 8 years at PayPal, or, in Matt's words, "8 delightful holiday seasons of scaling." The twist is that a holiday spike is a known quantity, and the stories are mostly about what people still get wrong when they know it's coming.

Predicting a Spike You Already Know About

Nordstrom has two big peaks a year, the anniversary sale in July and Cyber Monday, with elevated load through the holidays. Bridget asks whether the prediction is a Magic 8 Ball, and Rob says it's "more magic than I would like to admit to." In practice the forecast is worked out with product management as a percentage above last year plus a safety margin. Their habit had been to start testing for Cyber Monday right after the anniversary sale. Since Cyber Monday is the bigger peak, that left too little time to get the kinks out, so they now test for the next peak of the year from the start.

At PayPal the forecasting was more rigorous. Matt says Cyber Monday and the second Monday in December, which eBay called Green Monday, were the biggest days, with a smaller spike in March when people listed the gifts they didn't want. The capacity team used R and other forecasting algorithms to know weekly volume within about 5%. By March, April and May the number barely moved, and the planning went on from there.

Testing Peak Load

Rob describes two methods. They model the traffic in an internal lab, and the models differ because Cyber Monday is mostly anonymous shoppers while the anniversary sale is registered checkout, which hits different systems. They also do what Rob calls "testing in production, because what could go wrong?" That means ramping production-shaped load against production during a non-peak hour and stopping the moment they hit a breaking point. The gap is register checkout and new account signups, which they can't yet test that way in production.

Matt describes PayPal's Holiday Canary Program. Applications got flagged when response time climbed sharply for small increases in throughput, and the flagged ones got canary tests where the team pulled nodes out of traffic or changed load balancer settings to push more load at one host. The bigger challenge, Matt says, was always the giant shared backend resources.

Getting Teams to Care and Not Gating Them

Rob says everyone at Nordstrom cares about the anniversary sale, but Cyber Monday had a culture of assuming it would be fine because anniversary just went fine, so this year took some flag-waving. Rob also changed how performance testing worked. It used to happen at the very end of a dev sprint, right before release, so every problem turned into a question of whether the perf environment or the new code was at fault, and they shipped anyway to find out. Now performance testing is not a gate. Engineering teams are expected to use the perf lab themselves for risky changes, with a full-on test reserved for big complex features.

Bridget asks how you keep speed from letting an index on a query slip through. Matt says PayPal's database team watched new queries and schema changes in staging and ran SQL explains, and that features almost never went in live. They went in off and were turned on over time. Feature flags "are awesome, and they can be terrible if you don't manage them well," Matt says, having seen a feature start corrupting cookies, where turning it off did not put things back to normal. Matt's view is still that restoring service fast matters most: "failure is always going to happen. You just want to make sure the customer doesn't know it's happening."

Sending a Slice of Traffic Somewhere New

Rob adds another trick as more of Nordstrom moves to public cloud: send a portion of traffic to the new infrastructure. They routed 10% of product page traffic to it, and performance and add-to-bag rates were worse than legacy. They scaled back to 1%, the team diagnosed the problem and shipped a fix, and the traffic went back up to 50% with performance better and add-to-bag rates where they wanted them. "Having that variable, that slider is handy."

Bridget points out that this works because they measure business outcomes, not only response times. Rob says that has been a culture change, with heavy investment in real user monitoring so they see what the client sees, and it has found anomalies nothing else would. Rob noticed that performance degrades at night when people go home to slower connections. Matt says PayPal could see measurable differences in conversion and cart abandonment based on client time, and Rob confirms Nordstrom sees a link too, though for anything short of a bad performance issue the effect is small.

Code Freezes

Rob's answer on freezes is "it depends." Legacy systems that were not built for continuous delivery still freeze, with extra rigor on any fix that has to ship. The newer systems on public cloud are not frozen.

Matt says PayPal called it a moratorium, and moratorium day was the day all of operations threw a giant party. Matt's reasoning is that "the last release and the first release of the year are always the 2 worst": everyone crams features in before the freeze, and then the pent-up features go out at the start of the year. Bridget says that sounds like an argument for small batches, and Rob says Nordstrom sees the exact same behavior. Matt adds that as a payment processor PayPal owed merchants predictability, since the merchants have the same holiday peak.

Bridget asks about the difference between releasing code and making a breaking change. Matt says a PayPal checkout touched something like 80 services, and you never know how the most minor change will interact in production. Matt's example was an eBay seller who became a buyer and ended up with 10,000 addresses, because every address they had ever shipped to became one of theirs, and the system was deduplicating in memory. It "didn't work very well."

Bridget objects that if devs aren't pushing code, entropy and third parties can still break things. Matt agrees, but says a freeze narrows what you have to look at. Rob says incident data on frozen systems shows fewer breaking incidents, and adds that "it's a lot easier to explain to our business partners when a third-party device fails than when we touch something and it broke."

Peak-Day Operations

Matt says PayPal ran all-day bridges on every peak day, with everyone in the command center or NOC and hourly checks that everything was green. Rob says Nordstrom's third parties staff up for the peak days, open proactive incidents and review systems, and Matt says the worst that happened with a merchant was a threat to wire PayPal off their checkout.

Matt also says capacity is a sensitive topic around the holidays, because everyone wants to fix every problem by adding hardware, and that can make things worse and cause cascading failure. Matt is honest about the people side too: it makes the CTO warm and fuzzy to see a room full of people watching monitors. Rob says in a 1,600-person technology org some teams are further along with ChatOps and automated monitoring, and others still need the all-day call.

Incident handling itself doesn't change much. Rob says on-calls sit in a room together so escalation is faster, and teams that are not normally on-call get pulled in and escalate sooner. It is the see-something-say-something mentality of "let's just overreact to everything, at least on these couple days." Rob has been pushing the panic button sooner in recent weeks to keep teams practiced, because they only do this a couple of times a year.

Lessons and a Horror Story

Matt's takeaway is that "heroism isn't scalable," and that if you already have a process for finding your risk, you should assess it all year instead of right before the holidays. PayPal also hit years where a software architecture constraint meant no amount of infrastructure would help even at 20% CPU. So they set the target at double what they expected to hit, to force the engineering teams to fix the architecture.

Rob's horror story is from Rob's first year at Nordstrom, a few months in. The whole site came down for the entire anniversary weekend under load. They had never done performance testing and were still on bare metal, so scaling wasn't really a thing. It initially looked like a denial of service attack, and then, "no, no, it's just our customers trying to buy things." Cloud helps, but Rob says the catch is the services that still live in Nordstrom's data center: "I'll tell you what you can't provision on demand, and that's bandwidth." That means dealing with telcos and, sometimes, construction equipment.

Beyond Retail

Matt says Allstate's worry is a disaster where people need claims and the systems aren't there, which is much less predictable than a holiday. They aren't in public cloud yet, but they run platform as a service, which makes it easier to move workloads and forces them to be more metrics-driven. The cultural work, like rethinking least-access security when job functions change, is ongoing.

Rob says Nordstrom is investing in continuous delivery, infrastructure as code and lean continuous improvement, with plan-do-check-act cycles, and that the customer mobile teams are their unicorns. To spread it, they needed senior leadership bought in, shared weekly demos, and a dedicated team of practitioners who assess whether a team is ready, run a workshop and measure afterward. On measurement, Matt talks about the quality of tests and code coverage on CI servers and about cycle time, and Rob says cycle time is the big one for Nordstrom, where a VP set a goal of reducing it by 20%.

As they wrap up, Rob says the most exciting part of the move to public cloud is teams taking ownership of their systems, so it's "not an ops problem anymore, it's all our problems." Rob wants what Matt has, which is a platform that abstracts some of that responsibility. Matt says of Rob's public cloud developer empowerment, "I want what he has."

Checkouts

Rob:

Matt:

Bridget:

Bose QuietComfort 20 Acoustic Noise Cancelling Headphones (really work)
Lizone Extra Pro 26000mAh External Battery Charger (charges laptops!)

ITIL Eye for the DevOps Folks with Steven Boyd Oct 31, 2015

Show notes

Steven Boyd is a certified ITIL expert and a federal employee at the U.S. Patent and Trademark Office, running the service desk's problem management and major incident processes. Trevor opens the show by noting that "DevOps can be a swear word depending on who you talk to," and Matty comes in with a theory: DevOps is the natural merging of Agile and ITIL. This conversation is mostly Steven correcting what the DevOps crowd thinks ITIL is, and Matty and Steven finding out how much they agree.

What ITIL Is, and How to Say It

The pronunciation debate goes nowhere fast. Steven says the spelled-out ITIL is the common one, but that people in the DoD community say it like the word idle, which Matty had never heard. Trevor has never been exposed to it at all, so Steven gives the short version: decades of best practices for IT service management, originally collected inside a UK government organization to cut costs and improve efficiency. The current 2011 version is a repackaging of those practices. Steven stresses that it is "a descriptive framework, not necessarily a prescriptive."

Matty is ITIL V3 Foundation certified, and came to it backward. When Matty took the workshop, the realization was that it was the thing the team at Bank One had been doing the whole time. They just hadn't known they didn't invent it.

Change Management Is Not the Whole Framework

Steven walks through the certification ladder. Foundation gives you the vocabulary and an overview. The intermediate certificates split into a lifecycle track (service strategy, design, transition, operations, and continual service improvement) and a capabilities track. The expert level, Managing Across the Lifecycle, is the one that covers the ability to actually create internal processes based on the framework.

Steven's diagnosis of the ITIL backlash is that when someone says ITIL, they usually mean the change management process inside service transition. Matty adds that for most people in ops, change management is the piece they actually touch, through change requests that get rejected while nobody knows what's going on. Steven's explanation for why it is the only piece people know: "Because that's how it was sold." When an organization implements a piece of it badly and brands the result as ITIL, the whole framework takes the blame.

Risk Management, Not Risk Aversion

One listener question asked how ITIL's risk aversion squares with fail fast, fail small. Steven pushes back on the premise. ITIL is about classifying changes and documenting them so risk can be assessed, not avoided. Once an organization understands a risk and accepts it, the change becomes a standard change with standing approval, so it doesn't go to the CAB every time. The documentation also gives you traceability when you have to work out which change led to an incident.

Matty takes that in the automation direction: a standard change could be standard because a set of automated compliance, security and test checks are all green, rather than because someone wrote down which buttons to push. Matty wants humans in the loop only where something needs human judgment. That leads to separating risk from impact. Something likely to break that touches one thousandth of your users for five seconds may be fine, while something very unlikely to break that would "set the whole building on fire" may not be. Steven agrees the tolerance is specific to the organization, but says it only works if change management is wired to incident and problem management. Without that flow you have silos of information, "and now you're not making a real risk assessment."

Trevor asks what a CAB is. Steven says Change Approval Board, then corrects that later in the episode: it's the Change Advisory Board, and the advisory part is the point, since the board is meant to receive information back from operations.

Blameless Postmortems and Problem Management

Matty ties this to blameless postmortems: people will make mistakes, so the question is how to improve the system. If a change passed every automated check and still caused a problem, the answer is to fix the system, not to tell Trevor the code was awful.

Steven says the major incident process ended in what the DoD calls an after-action report, which Steven treats as akin to a blameless postmortem and as the trigger into problem management. Steven's problem management process starts with a preliminary analysis to scope the investigation and decide whether there is a viable business reason to pursue it. Steven argues that problem management is not only about prevention. It is also about minimizing and mitigating the impact on users, and most organizations invest in change management instead because vendors tell them that is where their incidents come from. In a fail fast, fail small shop you will have incidents anyway, so the question is how quickly what you see in service operations gets fed back into development and design.

Matty describes personal CAB experience as mostly making sure nobody changed a thing at the same time as someone else. Matty would rather trust an automated before-and-after showing that things weren't broken and still aren't, applied in a repeatable way. Matty quotes Mark Burgess: "every time someone logs interactively into a system, they compromise everybody's understanding of that system."

Roles, Functions, and How the Records Connect

Matty relays a question from Dustin Collins, who had said a shallow understanding of ITIL is that it helps define roles, and who pointed at the failure mode in cross-functional teams: "if everyone owns it, no one owns it." Matty adds an anecdote Matty calls infamous or apocryphal, about a speaker at Etsy asked how people make sure the follow-ups from a blameless postmortem get done. The answer was that they just do, which Matty says doesn't scale.

Steven says ITIL does address this, through the RACI matrix and through the distinction between roles and functions. People like to sit in one silo, but ITIL says an analyst can also hold a role in the change management process. Steven had seen incident teams believe problem management was somebody else's job and that documentation was for the designers.

When Matty sketches how an incident flows into a problem and then into a change, Steven stops Matty with a clarification: "one type of record cannot turn into another type of record." Incidents can trigger a problem, and resolving a problem may require a change, but the records stay separate and are tied together. Matty describes the bridge Matty's team built at Apartments.com between the Agile backlog tooling and change requests, so releases tied into changes and a problem could land back with a product owner as a defect. Steven says ITIL won't dictate that as long as you know your inputs and outputs, and that defect management is a big part of service transition, because without it you can't trace what you see in production back to something testing appeared to resolve.

Follow the Framework, Then Extend It

Matty raises zealotry: saying there is one ITIL way, and that anything else is doing it wrong, is the one thing that would actually be wrong. Matty also doesn't want people to decide they will never have a change board because they had a bad one three jobs ago.

Steven agrees on the zealotry but disagrees a little on cherry-picking. Steven says you shouldn't subjectively pick which parts of ITIL to implement, and you should deviate from the framework only when you are piloting something or you have a process that is better than the basic approach. "It's more of a guide," Steven says, and as long as you stay consistent and document your records, you can modify it. Matty clarifies that the point was extending it and building bridges, not skipping parts. Steven agrees that ITIL gives you the bare-bones best practice and that a better process only creates more value for users.

CMDBs in a Volatile World

The last listener question asks how a CMDB fits when infrastructure is volatile and what is worth documenting. Matty's position: "any CMDB that requires manual updates is about as valuable as the bits that it's written onto." If you treat infrastructure as code with something like Chef or Puppet, the tool can populate the CMDB with accurate information, which Matty prefers to a discovery crawl that takes six days and is stale on arrival.

Matty also warns about capturing too much. At the bank, each VM had a CMDB record tied to its host, and since VMware could move VMs between hosts all day, every migration became a change control process because it touched the CMDB. That made automatic load balancing impossible. Steven's answer from the ITIL side is short: document whatever creates value. For problem management, that means enough data to scope an issue, because if you can't scope it you can't assess its impact and urgency, and so can't prioritize it.

Steven's line from the DevOps DC talk, which Matty calls the pull quote for the episode, is a question: would you accept DevOps in a box? If not, why would you accept ITIL in a box?

Matty wanted to spend the last stretch on incident, problem and root cause practices like ChatOps and blameless postmortems. Steven's answer was "We don't have time for that today." They wrap on the point that both ITIL and DevOps are built around continuous improvement, and Steven calls the two symbiotic.

Agile Change and Release Management at the #1 Online Rental Site in the US - Matt's talk about ITIL and Agile
Chef Style DevOps KungFu

Checkouts

Trevor

Yoshi's Wooly World

Creating DevOps Communities and Events with Andy Burgin, Dustin Collins, and Nathen Harvey Oct 14, 2015

Show notes

Why Start a Community or Event

The topic came from Andy Burgin, who runs the LeedsDevOps meetup in the UK and works as a senior DevOps engineer at Sky Bet, and Dustin Collins, who runs the Boston DevOps meetup and is a developer advocate at Conjur. Nathen Harvey of Chef joins them. Nathen says an internal event or community makes sense for the same reasons colleagues don't join external ones: fear of sharing trade secrets, and the practical problem that meetups are after work, when "what you really want to do when you're done with work is go to family or go to pub." A lunch and learn during office hours avoids that.

For external meetups, Dustin says one-way conversations don't help much with hard things like DevOps, since participation is what surfaces issues you hadn't thought of. Andy adds that many people can't get to big conferences, so something local gives them somewhere to engage. Nathen says organizers and participants both have to make sure newcomers feel welcome, because "you're only a newcomer the first time you go." Andy started LeedsDevOps about two years ago because the city's meetups were language-specific and none spoke to Andy's ops work, apart from talks on tools like Vagrant. Andy had also used open source for years without giving back. Dustin took Boston over from an organizer who quit, partly for selfish reasons: to improve at public speaking and organizing, and to meet people.

Nathen's advice is "to not start a meetup group," since it's hard work, and if you're the right person you'll do it anyway. Otherwise, give a DevOps-flavored talk at the PHP group and see whether a few people light up. Dustin found some meetups hated it, but a few resonating people is enough. If the room says "Dustin, you're not allowed to come back," Nathen says, that's permission to start your own. Matty describes suggesting that a suburban organizer run events under the Chicago DevOps group instead of splintering, since it's a lot of work.

Doing It Alone, and Finding Help

Dustin ran Boston alone for eight months, spending six to eight hours a week, until a regular attendee who's very logistically minded offered to handle venues and sponsors. Andy has run Leeds alone for two years by being "essentially lazy": early on Andy took the first two speakers and the most convenient venue, and now the group's reputation lets Andy pick sponsors, with a backlog of speakers. Andy's early marketing was a web page and a Twitter account, following everyone who followed other user groups in the city, and "being quite British and polite, people would follow me back," which brought 100 relevant followers in five days. Andy sees the role as facilitator: "creating the thing and letting the thing happen."

Nathen remembers the first meetups as throwing a party and forgetting the invitations, and says the first DevOps meetup Nathen ran had no topic at all, only a discussion of what the group should be. Nathen ended meetups with checkouts so that everyone shared something and understood "this is not my meetup, this is our meetup." Matty says of the DevOpsDays Chicago kickoff, about 24 people came to the first organizer meeting and about 10 were in the organizer photo seven months later. Trevor says to manage your expectations of volunteers.

Dustin warns that if you start a DevOps meetup with your ops friends, it's easy to end up with an ops meetup: Dustin got about 24 ops people, "and I mean guys," and a 20-minute sidebar on ZFS versus Btrfs. Recruit a developer as co-organizer, Dustin says, and bring in security and business people. Andy says finding speakers beyond Docker talks is hard, and that Windows speakers took extra effort. Dustin adds it's hard to know what people will like: a Python meetup Dustin attended featured a talk about a machine that pricks your finger to test for the flu, with no Python on screen, and the audience loved it.

Local Celebrities, and Mixing It Up

Matty admits to the echo chamber of inviting the usual suspects. Nathen says if you bring in an outsider, also have a local speaker, since "we should also be building local celebrities." Dustin says to vary formats: long talks, short talks, open spaces and panels, or "it's that time of the month again." Matty says to shake things up while it's going well, since Matty's Chicago events fill within 48 hours and always with the same people. Dustin has speakers start basic and go deep, so both newcomers and experts stay engaged. Nathen says DevOps topics are wide, which is why open spaces work.

Trevor's Azure meetup drew mostly developers, and adding infrastructure talks shrank it. Matty says you can't control your audience, and jokes about putting the dev talk second. Dustin's controversial tip is to bring in a business person with a thick skin to tell it like it is, which happened once as a heckled stand-up and gave technical people a shared enemy.

Hosting at Your Company

To a Twitter question about hosting a meetup at a corporation, Nathen says if you want your employer to host, remind them that they're probably hiring, and a meetup brings people through the door. At Custom Ink Nathen would offer an office tour after each meetup, with the aim of getting people excited about working there without pitching them. Trevor adds that it gives your own team speaking practice. Andy says speaking at other groups promotes your own and is how Andy learned what DevOps really was: Andy had thought it was about Nagios and Chef, "and boy, was I wrong."

A New Segment

Matty announces a podcast recommendation of the episode, and starts with The Food Fight Show, which Matty says was a key learning source when beginning with DevOps. Nathen says the idea of checkouts was borrowed from the Ruby Rogues.

Matt spends the entire episode claiming that Nathen was famous for being on ADO11, when in fact it was ADO14.

Check Outs

Nathen

If you’re at a big conf, find the locals and do a mini meetup onsite
DevOpsDays Podcast
Public post mortems

Andy

Dustin

ContainerDays NYC - Oct 29-30 - Docker Docker Docker
Downtown NYC Tech is meeting Oct 28 - Boyd Hemphill/Docker in production
CMXHub - Good articles/videos on community building
Jenkins Job DSL - Conjur repo w/ jobs for example

Trevor

Agents of Shield Season 3
Surface Book, Nexus 6P <3 gadgets

Matt

Classy Little Podcast
Doctor Who LEGO set (Bridget told us about it, how did she know about it and we didn’t?)
CHICAGO CUBS CHECK THEM OUT YO!!! BUZZ SAW!!!

infrastructure as code with Joshua Timberman, Eric Sorenson, and Robyn Bergeron Oct 01, 2015

Show notes

What "Infrastructure as Code" Means

Joshua Timberman of Chef, Eric Sorenson of Puppet Labs and Robyn Bergeron of Ansible join Matty and Trevor for a panel on treating infrastructure as code. Joshua has been at Chef about seven years with a systems administration background. Eric started by running an ISP "when 14.4K modems were the new hotness," used CFEngine for a long time, and now does product management at Puppet. Robyn was a sysadmin from 1996 to 2000, then Fedora project leader at Red Hat for two and a half years, and has been at Ansible for a month: "systemd is not my fault."

Matty's working definition is that infrastructure is versioned, modularized and testable in an automated way. Eric says at AtomicCon in Portland, seven of ten talks opened with their own definition, and the most compelling, in Eric's view, was that if a natural disaster destroys your infrastructure, you could rebuild it in a new place "using just the contents of a version control repository." Joshua says the original line was Adam Jacob's. Robyn reads the many definitions as a sign that everyone's is shaped by their experiences and should stay open to evolving. Eric adds that seeing scripts checked in, with commit logs you can read like archaeology to divine intent, can be revolutionary for people who have never had it.

Unit Tests, Integration Tests, and "Is Nginx Installed?"

Eric says test-driven configuration management didn't exist when Eric started, and credits the Chef community for pushing it. Trevor says people are amazed to find they can check that "the thing I did actually was the thing I did." Joshua likes unit tests because you're testing inputs across platforms, since real environments mix SmartOS, Linux and Windows, and tests catch regressions when someone breaks the most popular platform. Matty says sysadmins are used to testing but not to automating it. Eric quotes an AtomicCon line that everybody has a test environment, and some people are lucky enough to have it be different from production. Robyn's own is "via Twitter when everything goes down."

A Reddit post Matty flags argues that unit tests alone aren't enough for configuration code. Eric agrees they're necessary but not sufficient, and says the state of the art has moved to integration-style tests that spin up machines and assert how they interact, using tools like ServerSpec. Matty borrows a framing from a colleague: signal in is unit tests, signal processing is whether the tool works (the vendors' job), and signal out is acceptance tests, which are "is what I said what I meant." Matty would have newcomers write the "is nginx installed?" test because it teaches the pattern and guards against regressions, and Joshua has come around to 100% coverage of resources so that an accidental deletion during refactoring is a conscious choice.

Matty tells, going by memory of Paul Reed telling it, of Firefox shipping a build that broke MLB.com on the first day of a World Series, and a regression test that has run in every build since. Eric says acceptance tests are your codebase's scar tissue, and that they can slow releases as they accumulate. Joshua says skipping tests just moves the cost, because "you're gonna have to refactor the entire world anyway."

Crawl, Walk, Run

Matty's advice is to start doing it right from day one: even a pipeline with no tests, so the only way anything reaches a server is through source control, because habits are hard to break. Why bother, Matty asks as devil's advocate, when you could keep playbooks on your laptop? Joshua says that works for one person, but real professionals need to recover when a quick change goes wrong. Eric says it's fine "as long as you don't have any coworkers or any customers," and Robyn says otherwise you're making other people have bad days.

Robyn stresses baby steps, since a run of failures demoralizes people, while small successes add up to minutes, and then to a week not spent patching fires. Matty's version is crawl, walk, run: learn to install Apache before you manage the storage array, and pilot with people open to change the way Agile transformations do. Trevor types "trickle-down DevOps" in the chat.

Aligning Infrastructure Code With Application Code

Trevor finds it's often easier to let the infrastructure tool deploy the code, having spent two weeks fighting Octopus over environments. Matty says the application is just another configuration point, but that if you already have a working Capistrano setup you should keep it, and that aligning the two takes a high level of trust. Robyn says it takes trust, transparency and simplicity, and that a group insisting on a tool nobody else has a month to learn is the start of an unhappy relationship. Trevor says don't force everyone onto two tools at once.

Eric describes high-functioning teams where application developers build versioned native packages, like RPMs, and not an obscure tarball or WAR file, so the application sits alongside the OS packages, and calls that what empathy means in practice. Matty notes that people used the empathy talk at DevOpsDays to say the community was all feelings, and points to a line in the Continuous Delivery book that having a very skilled person do mundane tasks is the surest way to ensure error, short of sleep deprivation or inebriation. Eric jokes about a world where Jordan Sissel never had to write FPM, and adds that the tool came out of a low-trust environment where Eric couldn't ask the team to deliver something installable.

Empathy Includes Sales and Marketing

Robyn says people at open source companies see trust and transparency on both sides, in the community and in the workplace. Trevor says marketing colleagues at 10th Magnitude were upset by a tweet at a DevOps event asking why companies and their salespeople were there. Robyn's answer is that it's so they can develop empathy for the people using the tools, or it's "a gigantic feedback loop." Robyn adds, "You're either an open source software company or you're not," and that saying you're all about DevOps but not talking to people leads them to expect a unicorn and get a box of Kleenex.

What's Different Now

Asked what's changed since they first treated infrastructure as code, Joshua says "everything": you can't be a sysadmin without understanding how to write some kind of code. Eric says culture, with developers building monitoring in as a first-class thing, and tooling, since there are now test frameworks even for Bash. Robyn says open source used to be "no free hippie code" at work and is now acceptable. Matty says the shift is enterprises like Target and GE sharing their stories, after years of being told "you are the 5th person to come in here today and tell me that I should be like Netflix," and Robyn says companies now let employees share theirs, including the failures, in part to keep them.

The Reddit post referenced in the episode:

Having a difficult time wrapping my head around test driven infrastructure

Check Outs

Joshua:

Policyfiles! Webinar coming soon, or already happened!
ChefDK 0.8.0, especially if you’re starting to work with Policyfiles
Fat Scotch Ale - Silver City Brewing get it at SEA TAC! :D

Eric:

Dark techno/dnb from Grey Area in the UK
Ansible :) especially people using Ansible in conjunction with puppet and chef - @ me on twitter!

Robyn:

Eric: Lots of those peeps :) I am happy to help connect folks.
Ansible 2.0… soonish!
Fedora 23 beta: GO GET IT
SPLATOON

Trevor:

Welcome to the Dungeon boardgame
Rocket League

Matt:

Pac-Man 256 iOS Android
The Martian (audiobook)
Felicia Day’s audiobook, []“You’re Never Weird On The Internet (Almost)”](http://www.audible.com/pd/Bios-Memoirs/Youre-Never-Weird-on-the-Internet-Almost-Audiobook/B00XUTQ692)

Cognitive Neuroscience with Courtney Nash & Lindsay Holmwood Sep 23, 2015

Show notes

Systems of Technology, Systems of People

Trevor and Bridget talk with Lindsay Holmwood and Courtney Nash about how human brains shape the way we build and run systems. Lindsay moved from engineering into management a few years ago and has spent about three years reading everything within reach about how people and groups work and about cognitive biases. Lindsay is three weeks into a role as infrastructure and platforms lead at the Australian government's Digital Transformation Office, whose remit is to build "clean, fast, simple, and humane services." The platform is the easy part, Lindsay says, and the hard part is helping teams learn to get the most out of it. Non-Australians can be hired with visa sponsorship.

Courtney, an editor at O'Reilly and Velocity conference chair, was a cognitive neuroscientist in training, a year from finishing a dissertation on learning and skill acquisition, before leaving to work at Amazon. Velocity struck Courtney as operations and therefore not a fit, until finding people who realized that "when you have to start talking about systems of technology, you have to start talking about systems of people." Courtney and Lindsay met at Mountain West RubyConf in 2013 after Lindsay's talk on escalating complexity, which took 60 or 70 hours of research and was emotionally taxing because "people actually die."

Fast, Slow, and Priming

Courtney warns that if someone says their product is deep-learning AI, most of that is "just not true," and that the brain tricks us into thinking we make rational decisions. Courtney explains the Stroop effect: it takes longer to say the word blue when it's displayed in green than in blue, because the brain systems for reading words and reading colors conflict. Priming can produce similar friction, for instance in how quickly you associate a woman with the word physician versus software developer. The interesting part, Courtney says, is becoming aware of where those conflicts sit.

Lindsay has given the DevOps Field Guide to Cognitive Biases talk three times. Lindsay describes Kahneman and Tversky's two systems, a fast one and a slow one, with the brain optimizing for speed over accuracy. The guest notes that Thinking, Fast and Slow has one of the lowest completion rates among Kindle books because it's dense, and admits to stopping after a couple of chapters. Smell, Lindsay says, bypasses the fast system, and Lindsay tells of a New Zealand safety company that has workers mount a handkerchief with a loved one's perfume on their equipment to prime them to slow down. Courtney adds that smell takes a different route in the brain than the other senses. Lindsay's gentler recommendation is Dave McRaney's You Are Not So Smart.

Biases Can Be Retrained

Courtney says the good news is that brains aren't fixed. Courtney had "a wicked reaction time" on the Stroop test only from practicing it to embarrass people. Lindsay says biases come from underlying heuristics and can be countered. Lindsay says that in a blameless postmortem you can appoint a devil's advocate to argue for the old view of blaming a person, which gets people to spend more time justifying what actually happened.

Lindsay's other example is the Dunning-Kruger effect: the less you know about something, the better you think you are at it, and you're also worse at recognizing genuine skill in others. The guest says it's what "basically drives DevOps," with ops thinking developers can't run things and developers sure they could keep it up if only they got in there. Even minimal training in a topic improves your ability to self-rate, and being embedded in another team is enough. Lindsay also claims the effect mostly shows up in the West and doesn't replicate in Southeast Asia.

Three Biases That Get in the Way of Empathy

Courtney says empathy has reached buzzword status, and likes that, and recommends Indi Young's work on it as a skill. Lindsay names the three biases that most affect it, as the guest sees it. The fundamental attribution error is judging someone's action by their character, so the driver who cuts you off is an idiot, and not a person swerving for a possum or a wombat. The better-than-average effect: Lindsay says a study in Australia found 70% of people rate their driving above average. The halo effect is when a great impression of a person or brand colors your reading of everything they do, which is how you get a cult of personality.

Normalcy Bias, Alert Fatigue, and Anomaly Detection

Bridget asks about Lindsay's talk on the psychology of alert design. The talk's main takeaway is the normalcy bias: before something bad happens we discount it because it never has, and during and after an event we're slow to recognize it. Lindsay cites research on tornado sirens finding people need confirmation from five to seven sources before acting. Courtney relates it to alert fatigue: the brain filters and finds patterns, so if alerts fire without anything bad happening, it stops reacting. Bridget adds the temptation to raise the threshold on an alert that's always going off.

Lindsay was working on an anomaly detection startup until a month earlier, chaining open source tools together to mimic newer findings on how the brain does pattern recognition, like "a basement full of college students that are just looking at a single graph all the time." Lindsay says statistical techniques built on normal distributions don't fit web operations data, and plans to open source the work. Courtney says most so-called AI is really good pattern recognition, which isn't nothing.

Getting a Talk Accepted

Courtney asks Lindsay about early Velocity rejections. Lindsay says the talks were procedural and purely technical, and got better once the work drew on bigger ideas from management, leadership and psychology books, which Lindsay reads instead of Hacker News. The Field Guide talk is, Lindsay says, "a blatant ripoff" of Sidney Dekker's Field Guide to Understanding Human Error, and it worked.

Courtney's advice, borrowed from Scott Berkun, is to give a talk about something you love, hate or are very good at, and to treat the CFP as the proving ground for the idea. Lindsay tries out 5 or 10 minute pieces at local meetups, taking notes the way comedians do. Bridget says speaking at meetups also gets you video, which program committees will watch for unknown speakers. Courtney agrees big stages aren't where to start: "it's terrifying."

Check-Outs

Lindsay recommends Patrick Lencioni's books, McRaney's two books, Kahneman's, and QF32, a Qantas pilot's account of landing an A380 and the team dynamics involved. Courtney recommends Ramez Naam's Nexus trilogy and Neal Stephenson's Interface. Bridget recommends the indie film Advantageous and two Velocity Santa Clara keynotes, by Laura Bell and Astrid Atkinson.

ChatOps Extravaganza with Jason Hand, Sasha Rosenbaum, and Peter Burkholder Sep 10, 2015

Show notes

Recording Live from DevOpsDays Chicago!

ChatOps is used by many teams and companies as the main communication tool for day to day chat, and their most important activities. In fact, ChatOps may be taking the place of email in the workplace for internal communication for tech teams as it helps communication during DevOps activities like deploys, code pushes, etc. This episode discusses best practices (if there are any) of ChatOps and how to make sure you are getting the most from your team communication tools.

Asynchronous vs. Synchronous Communication

25% of the work week is spent managing your inbox. You can actually increase productivity by moving to Sync communication…we think.
Sasha: ChatOps creates an enormous amount of noise while at the same time makes communication grouped and searchable.
Discussion suggests it is up to the user to mediate that noise, but is it the user, the culture, or the conversation itself that dictates the role of chat? Tivo is given as an example of user mediation: you recorded a shit ton of stuff, and watched only what you wanted.

The panel comes to the conclusion that important decisions should lean away from ChatOps, and into a more formal, permanent form of communication. “Important things will ‘re-bubble’ again,” but the chatroom is not the place if a team consensus is needed, especially if the team is remote.

Create a culture where ChatOps is used in the way you need.
Risky to go "Super Pendulum Swing" in one direction or the other.

What is ChatOps good at?

Solving the communication problem.
- Brings everybody into the same experience. Even if you are across Europe, or accross the room, you are having the same experience.
Great for in the moment Q & A.
- Even with one on one questions, if the answer is shared in a public channel, the information is given to all on the team which moderates the need for repeated questions, and increases team efficiency. You need to be constantly pairing. If you direct message someone, you are keeping that information from the team. "If you are not working in your chat tool, you are not collaborating."
Shared History
- Makes communication searchable, and organized by topic, or at least team.
Rooms should be broken down to their smallest parts. Topics, Meetings, Projects, they should all be open spaces for all departments.
Getting messages/alerts from integrated tools is perhaps one of the most important features of ChatOps in DevOps: Jenkins, Github, Travis, etc.

What’s the Problem?

There are just too many messages. But they are necessary messages.
Internal ChatOps tool is almost useless when you are a consultant and you are all working on different clients.

Problems with Adoption?

In your organization, if you are considering chat tools for different purposes, use benchmarking and measurements to monitor your usage and data in each tool (in this case, chat vs. email)
Matt uses RescueTime (https://www.rescuetime.com/) religiously. His current rate of email vs Slack: 4 Hours in Slack, 48 minutes in email.
Sales, Support, etc. prefer email, but that will not change until their tools are integrated with Slack as well.

How to make use for permanent communication?

Have you adopted ChatOps for sustained messages and conversations that need to be kept?
Pinned Items are like the refrigerator...it's emptied at the end of the week.
If it should live longer than a week, then it gets moved to the wiki, google doc, or the most appropriate space for the info.

How do you practice Chat-Zero? (Comment with your answer)

How do you go through every message in your chat?
How do you know what is important?

Last Thoughts:

Peter can’t wait for computers to be smart enough to interrupt us only when appropriate. “Who is the least invested in their work right now? Let’s notify that person.”
If you are considering it, do it. But be careful what you use, and how you use it.
Jason: It's new tech, but its the old problem. ChatOps is just the newest efficiency on the line.

Sign up for the Banana Stand for the latest ADO news.

Podcast Me Maybe with Kyle Kingsbury Aug 15, 2015

Show notes

Matt, Trevor, and Bridget catch up with Kyle Kingsbury about his research on failure in distributed systems, lighting rigs controlled by code, consent and representation, and more.

Call Me Maybe: an exploration of failure in distributed systems using the Jepsen tool.

Monitorama 2015 talk on Riemann

Bridget announces that she’s joined Pivotal; Matt claims this is to diversify the podcast so it’s no longer Chef employee, Chef partner, Chef customer.

Community & Events

Lots of open CFPS on devopsdays.org

Chef Community Summit is Oct 14 and 15 in Seattle. Matt & Trevor will be there.

Matt & Trevor: at DevOpsDays Chicago August 25 & 26!

Bridget: at VMWorld the first week in September.

Check Outs

Kyle:

Subnautica!

Bridget:

Sense8 on Netflix
Bosh tutorial

Trevor:

Alphabet Announcement. Google pointing themselves towards the evil Silicon Valley "Hooli" company.
Dragonball Z Resurrection F
Herkimer NY, and herkimer diamonds

Matt:

Amazon Echo
Helicarrier LEGO set

Building An Ops Team with Charity Majors, Patrick McDonnell, and MCR Jul 29, 2015

Show notes

Charity Majors (Parse/Facebook) is an “Accidental Computerer.” She was the first infrastructure hire at Parse, which was acquired by Facebook in April 2013. Charity handles all of the backend operations and DBA work, and manages a team of 7 engineers.

Patrick McDonnell is a web operations manager at Etsy. He made the transition from individual contributor to management a few years ago.

Mike Rembetsy (MCR) is the VP of Technical Operations at Etsy. He was one of the first ops people at Etsy when he joined in 2008, and has helped grow the team to over 50 engineers since then.

Charity points out that your first question should be whether or not you actually need an ops team. She says, “There are a lot of places out there that think they need traditional operations engineers, when all they really need is someone to really care about their infrastructure… You should have genuinely hard operations problems before you even start looking to hire engineers.”

Once you do start down the path of building an ops team, MCR notes that you have to maintain it. You’ll need to grow the team, grow the individuals, grow the culture, which, if your company is in a startup phase, can be distracting to the overall goals.

“The glue that holds a team together is how well people interact with one another, how well they respect each other,” says MCR. “That’s what you build good ops teams on, and frankly, that’s what you build any good team on.”

As a growing company, we suggest you follow this process:

understand/establish your mission
communicate that mission clearly to the interviewing team
find people who can fulfill that mission
understand the fact that technical skills are great, but at the end of the day, it’s about the chemistry and culture of your team

Trevor agrees that cultural concerns are absolutely an issue, and asks for suggestions on how to interview for that.

MCR explains that at Etsy, they split their interview questions into technical questions and cultural questions about how the interviewees handled particular situations, as well as other skills.

Charity notes that good ops engineers are good at learning things, but some people freeze up when they’re put on the spot. She suggests providing interviewees with at least 50% of the questions beforehand, so they have a chance to prepare the preliminary information before interacting with her in a formal interview. “You want people to bring you the self that you’re going to be working with on a day-to-day basis,” she says, “not the self that is freaking out and wondering how they’re being perceived.”

Transitioning into the topic of management, Charity brings up the point that if, as a manager, you’re still responsible for key pieces of the infrastructure, you’re holding your team back in their technical development.

One of the jobs of a manager, MCR says, is to help motivate your team, and then step aside and let people get things done. In doing this, you let them succeed, as well as fail, and grow as a result of those experiences. He references Daniel Pink’s book, Drive, and the three pieces of science that motivate human beings: autonomy, mastery, and purpose.

“A manager’s role is a facilitator,” says Patrick. “Everyone should be doing what they think they need to do. It’s my job to remove obstacles that come in their way and make sure I can smooth things out if need be, but really, it’s to encourage and allow people to reach their full potential.”

Etsy has two separate career paths: one for individual contributors and another for management, which allows for the honing of particular skills specific to the end goal.

Charity agrees wholeheartedly with this approach, and says emphatically, “Management isn’t a promotion. It’s a career change.”

In addition, manager and leader is not synonymous. Some people are much better leaders than they are managers, and vice versa. It is also possible to be a leader while continuing as an individual contributor. In fact, Patrick points out that in some circumstances, you might actually lose influence when moving into a management role if you’re already a leader amongst your peers.

Bridget asks the panel, “What’s your best advice for someone in the position of building an ops team?” Patrick: “Focus on hiring good people. Hire people that you like. Hire people that you trust. Hire people that, maybe they need to do a little more research, maybe spend a little more time on StackExchange than the next person, but you know that they’re going to get the job done in the way that you need to get the job done.”

Charity: “Build your networks. Go to meetups, talk to people. Don’t just talk to the popular kids. Reach out to diverse communities and diverse crowds, and go meet people who are doing cool and exciting things, who are slightly off the beaten path. The more people you know, the better you’re going to be at hiring.”

MCR: “Make sure that you address conflict. Make sure you create a safe place for the people you do hire, to have open, honest conversations with one another. There’s nothing more toxic to a team than people chatting behind other people’s backs.

Check Outs

MCR: – Open Source Utility for ELK – 2012 Velocity Talk from Dr. Richard Cook: How Complex Systems Fail Charity: – Guided Meditation for Adults: “Breathe in strength, breathe out bullshit.”

Patrick: – Dr. Christina Maslov’s talk at Velocity: Burnout in Tech – 5-minute burnout self-assessment

Bridget: – DevOpsDays Minneapolis talks

Trevor: – Batman Arkham Knight – GitKraken

Upcoming Events:

DevOpsDays at #alltheplaces

eating sushi with andrew clay shafer Jul 18, 2015

Show notes

Andrew Clay Shafer (@littleidea).

Coming to you live from DevOpsDays Minneapolis, Matt and Bridget sit down with Andrew Clay Shafer in front of a live audience to talk about the growth of DevOps, explain some commonly heard but not always understood terms, and more (after a brief detour on why episode numbers on podcasts are obnoxious, and why this episode is titled "Eating Sushi with Andrew Clay Shafer").

Don't know who Andrew is? He suggests you Google him, but then goes on to give a little bit of his background: he's been involved in software development and technology for almost 20 years. After rooming with Luke Kanies, founder of Puppet Labs in college, Andrew got interested in operations and system administration.

O'Reilly's Velocity Conference was also influential in Andrew's growth, and Andrew reiminisces about his presentation on Agile Infrastructure in 2009. John Willis speaks up from the audience, and strongly suggests going to look at the slides.

Matt takes a moment to explain how Arrested DevOps started: "I started listening to John and Damon on DevOps Cafe and understood about 5% of what they were talking about... There are these things that we, as part of this community, tend to know, and what we try to do with this show is break it down for the people who don't have the tenacity or stubbornness that I did."

Along that vein, Matt asks Andrew to expand upon the "wall of confusion" idea that was referenced in his 2009 talk, and has become a commonly-used (but not always understood) term in DevOps lingo.

"It's a jargony way to talk about the different incentives that exist between developers and operations," says Andrew. "There's a transition that happened as software became service-oriented, versus shipped on CDs, where the servers now become this critical part of the value chain, and if you deemphasize the system administration and operation of those servers, then you don't actually have software. In the middle of these two worlds, where in one, systems administrators were for keeping the printers and the mail server up, to where they're a critical part of the value chain in the new world, there are broken IT practices that don't make sense when you're trying to manage a service. It means recognizing that the best way to optimize a system isn't to just throw random stuff onto production servers, and then make it ops problem, but to recognize that the infrastructure itself has become an application, and that you can manage these things as an application."

Andrew points out that as much as he enjoys the attention (and who wouldn't?), he was simply in the right place at the right time, and the right people listened to him. He connected dots to take advantage of tools and practices that were already used to manage software process, and bring that into the infrastructure and operations works.

Bridget asks, "You mentioned Agile, and you mentioned Scrum. I've heard you say that Scrum is a disease. Can you give us your thoughts on where that sort of stuff is going?"

"My personal opinion is that Scrum's impact on software development is net negative," Andrew says. "I think it's particularly bad when people try to adopt it in operations. It's really susceptible to problems when you have any interrupt-driven work whatsoever." He suggests pursuing kanban and chatops, making work explicit and visible -- tools that allow both you and your management to understand the full context and value of the situation.

The conversation transitions into talking about how to make actual changes to your operations and infrastructure teams, rather than always jumping through hoops to make the necessary changes to keep the pages up or the apps running smoothly. The answer isn't to simply communicate how difficult it is to manage a system -- upper management won't understand the pain, and therefore won't listen to the complaints. You have to involve the rest of the team so that they understand what you're going through first-hand. You get empathy from suffering. This all plays back to Conway's Law, as Andrew points out: "If you believe Conway's Law is true (as I do), then you understand that your org structure (who communicaties with whom, who reports to whom, etc.) determines the outcome of any decision."

Bridget brings up the point that this is the essence of dogfooding: requiring not only your engineers to be in the code, but your employees to be using the products that you're creating, so that there's a general understanding of why things work the way that they do, and a buyin for the necessary changes.

Bridget asks Andrew to expand more on what he thinks we are (and should be) optimizing for, which he touched on briefly during his talk at DevOpsDays.

Andrew counters that in order to do that properly, we need to first frame the context, which is a problem that plauges DevOps, Agile, and many other systems with which people are trying to transform their companies -- you can't do something prescriptive until you have enough context to understand where you're starting from. For example, the diet and exercise program you'd give to someone who's relatively healthy and active is very different than someone with a different set of circumstances. By the same token, you can't prescribe a solution to an infrastructure problem without first investigating the roots causes and understanding what the foundation is.

However, if you model the world as everything is an agent trying to maximize some function, then the basic premise of your decisions is cause and effect. "Looking at the way people behave, and how this plays out within organizations, you might have very different patterns of interactions and patterns of health." Andrew continues, "Therefore, what you'll tell people do is very different from context to context."

Despite all of the different scenarios, Andrew argues that there are three things you can always do:

Understand the incentives that people are motivated by
Align the incentives with behaviors
Radiate information to help people make different choices

Wondering what the Nash equilibrium and Pareto efficiency game theories are? Here are a few links:

Nash equilibrium: Wikipedia
Pareto efficiency: Wikipedia
Andrew's slides on Leading a Learning Organization

Career Devops with Jeff Hackert Jun 25, 2015

Show notes

Jeff’s background includes 30 years of experience working in people management. Specifically in developing the management skills, and careers of Software Engineering teams. He now works at Chef as the Director of Learning Experiences.

What is Career Development

Just thinking about career development in the workplace is not very common. Jeff discusses statistics around the specifics of Career Development and implementation strategies. For example, in one study, more than 60% of respondents said that Career Development was of “little to no” importance to their current employer. Less than 5% of employees at some organizations surveyed receive any career feedback from their current bosses.

Jeff: There are different approaches people usually take as they grow into their careers. The first is a “go with the flow” approach which can cause issues when leadership skills are not developed in response. A second approach is more proactive in which you plan out and describe your career and where you would like to go in the future.

What do you do?

The panel discusses their own careers and their attempts to summarize “what you do,” their weaknesses, and their strengths. Especially if the description is to be seen by the entire organization. Can you brag? Should you be vulnerable? For some, listing strengths as an engineer can actually be more difficult than listing weaknesses.

Jeff: “It’s a huge act of vulnerability to say who you want to be in an organization”

Jeff: “Every Engineer is responsible for their own career development […] and you are 100% responsible for the career development for every engineer on your team”

Can you be the Director of Flowers?

The group discusses the merit of titles and how relevant and useful they are within an organization. Jeff: “Titles don’t matter, and they absolutely matter.”

Ultimately, titles should be descriptive of what you want to do in that position and are indicative of positional authority more than anything else.

Management is not for everybody.

"It's Not a Promotion - It's a Career Change" — Lindsay Holmwood (http://fractio.nl/2014/09/19/not-a-promotion-a-career-change/)

Matt: “I don’t like managing people that’s not my thing.”

Jeff: Performance management is not the same as Career Development. Often, when Software Engineers get promoted to managers, coding becomes a secondary responsibility and People Management becomes a priority. For some, this is not the trajectory they want for their careers, and that’s ok. Providing context for feedback and guidance is a great way to identify these mismatches between current position and where someone wants to be.

Jeff: Using analytics "Project Oxygen" from Google describes 8 qualities of a good manager. ( http://www.nytimes.com/2011/03/13/business/13hire.html )

Jeff: "The Beatle Book" - Ken Schwaber ( http://www.amazon.com/Ken-Schwaber/e/B001H6ODMC )

Trevor: Doing an emotional check-in to describe your current emotional state is a powerful management and communication tool (from the Core Protocols by Jim McCarthy - http://www.mccarthyshow.com/online/)

Jeff discusses the usefulness of check-ins on every level within an organization.

You know you have a good manager when: 1. They expresses real concern with your career development, separate from quality conversations. 2. They create opportunities for you to realize your goals. (or at least get closer) 3. They have your best interest at heart.

…and then there’s the checkouts:

Our TOPPODCAST Picks

Follow Us

Stay Connected

Predicting a Spike You Already Know About

Testing Peak Load

Getting Teams to Care and Not Gating Them

Sending a Slice of Traffic Somewhere New

Code Freezes

Peak-Day Operations

Lessons and a Horror Story

Beyond Retail

Checkouts

Rob:

Matt:

Bridget:

What ITIL Is, and How to Say It

Change Management Is Not the Whole Framework

Risk Management, Not Risk Aversion

Blameless Postmortems and Problem Management

Roles, Functions, and How the Records Connect

Follow the Framework, Then Extend It

CMDBs in a Volatile World

Checkouts

Trevor

Why Start a Community or Event

Doing It Alone, and Finding Help

Local Celebrities, and Mixing It Up

Hosting at Your Company

A New Segment

Check Outs

Nathen

Andy

Dustin

Trevor

Matt

What "Infrastructure as Code" Means

Unit Tests, Integration Tests, and "Is Nginx Installed?"

Crawl, Walk, Run

Aligning Infrastructure Code With Application Code

Empathy Includes Sales and Marketing

What's Different Now

Check Outs

Joshua:

Eric:

Robyn:

Trevor:

Matt:

Systems of Technology, Systems of People

Fast, Slow, and Priming

Biases Can Be Retrained

Three Biases That Get in the Way of Empathy

Normalcy Bias, Alert Fatigue, and Anomaly Detection

Getting a Talk Accepted

Check-Outs

Recording Live from DevOpsDays Chicago!

Asynchronous vs. Synchronous Communication

What is ChatOps good at?

What’s the Problem?

Problems with Adoption?

How to make use for permanent communication?

How do you practice Chat-Zero? (Comment with your answer)

Last Thoughts:

Sign up for the Banana Stand for the latest ADO news.

Community & Events

Check Outs

Kyle:

Bridget:

Trevor:

Matt:

Check Outs

Upcoming Events:

What is Career Development

What do you do?

Can you be the Director of Flowers?

Management is not for everybody.

Related Podcasts

1

2

3