Arrested DevOps

Latest Episodes:

A Year of ADO Jan 01, 2015

Show notes

How It Started

For the year-end episode there are no guests, and Bridget Kromhout, the newest host, asks Matty and Trevor to start from the top. Matty says the idea started as a DevOps 101 blog, a friend came up with the name Arrested DevOps, and podcasts had a gap for people just starting to learn about DevOps. The mission statement was to be the show for people whose boss "read about DevOps in the in-flight magazine and now I'm supposed to do it." Matty learned from DevOps Cafe, then Food Fight and The Ship Show, and admits to still not always following what John Willis is talking about. Bridget points out that Matty is now name-dropping Willis and Damon Edwards after swearing off name-dropping in episode 1, and asks if Matty has met John Allspaw at Velocity. Matty hasn't been to Velocity yet.

Trevor tells how they met at the new Azure meetup in Chicago, running into a "really loud, knowledgeable guy" who turned out to be Matty. Matty says follow-through has never been a strength, so a partner was needed for accountability, and that editing the first episode made Matty think "are you ever gonna let Trevor talk?"

By the Numbers

Matty cautions that podcast stats are guesswork, since they only count audio downloads, and that Matty halves them for personal use. The most downloaded and most viewed episode is 14, How to Eff Up DevOps, which Matty calls the Pete Cheslock effect (Bridget: "like the Colbert bump"). Older episodes keep growing, which Matty takes to mean new listeners are working through the back catalog. Matty says an episode gets about 2,500 to 3,000 audio downloads and 200 to 300 YouTube views, and that when the Minneapolis episode wasn't posted to YouTube, some people wrote asking what happened, since YouTube is how they follow the show. Every episode except the two live ones was recorded on a Google Hangout, "which is ironic, I guess, that our truly live episodes were not live streamed."

Topic First or Guest First

Matty admits part of the reason the podcast exists was to get Jez Humble on. After the tenth episode Matty tweeted, and Jez replied that the hard part isn't getting Jez on the show, it's getting Jez to shut up once on. Jez was episode 15. Trevor says the approach is topic first, then who can talk about it, and jokes that Matty likes famous names. Matty admits to being prone to the echo chamber, and says Trevor finds people Matty has never heard of, which is good since some listeners aren't in the DevOps community at all.

Listener feedback shaped some episodes. People said the show was heavy on culture and light on tools, which led to the Git episode. Matty says the audience is causing an identity crisis, since it's meant to be the 101 show but people Matty knows listen. What matters more to Matty are the messages from people who say "I didn't even know where to start," or one who wrote that they'd drunk the Kool-Aid, their organization hadn't, and the show made them feel it could be done. That's "way more awesome" than knowing an expert listens.

Favorite Moments

The episode plays an audio supercut of the year's lines, including "tools are easy, people are what's tough." Bridget loves that in episode 1 Matty wouldn't read the John Vincent quote because Matty didn't want to say the word shit on air, and that by the Etsy episode "everybody is swearing like Chef employees." Matty says the swearing used to get a spring sound effect, until that stopped. Trevor says they originally avoided the explicit tag so as not to turn listeners away, and that Matty eventually decided that was a fruitless effort. Bridget also likes Trevor's line in episode 2 about never stopping learning, and Bridget's own MST3K-style response: "so you're like a shark?"

Bridget's favorites are the Minneapolis episode with Patrick Debois, the conferences episode with Jason Dixon and Pete Cheslock, and hosting the enterprise one. Matty says that one was a comedy of errors, with hotel Wi-Fi, a guest in Budapest, and an Azure network incident that day. Trevor's favorites are episode 17 on asking for help and the Etsy story about telling a senior VP that the VP was being a jerk. Matty's is episode 14, partly for a running backchannel in a Google chat, and partly because Matty thinks a guest is different from a host, and it was fun to watch Nathen Harvey simply contribute. Matty regrets the awful audio from Matty's cheap headset.

Behind the Mic

For anyone thinking of starting a podcast, Matty says the cost ranges from nothing to thousands a month. They record on Google Hangouts on Air, which pushes to YouTube, then Matty pulls the audio and MP4, converts to AIFF, and either edits it personally or sends it to Mandy Moore (@therubyrep), who edits it and returns it via Dropbox, usually within 48 hours. The MP3s live on Amazon S3 and the site, which serves the RSS feed, runs on Azure. Sponsors pay for the post-production and microphones, and for the first few months it was funded out of Matty's pocket. Matty notes that when editing, everyone seems to say "um" more toward the end because boredom sets in, which is the argument for outside editing.

The only thing stopping them from publishing more often than twice a month is time. A third co-host helps, since until Bridget neither could skip an episode, and Julian Dunn had to stand in at Minneapolis.

The Retro Goes Missing

Trevor says the original structure was a mock agile framework, which is why the closing segment is called checkouts and there was a retrospective at the start. They dropped the retro because they didn't have anything to say, and Matty says the current work can't be discussed, like customer visits. Matty now misses the retro, since shows like The Ship Show and Food Fight let you know what the hosts are up to, and asks listeners whether they care.

Where the Show Fits

Trevor says other podcasts rarely get a listen, and Matty says the show fits in the same place as before, alongside The Ship Show, whose hosts seem to hack Matty's Google Docs and do the show ideas first. Matty says the show is not what was envisioned, "and that's cool." Trevor says people have told Trevor they're glad to have an intro point, and a sysadmin friend prefers The Ship Show for the technical detail, and that for Trevor it's "nice to provide what I needed then."

Bridget asks Matty whether the show has changed Matty's cynicism. Matty says the optimism was forced, starting when Matty told the team "let's just pretend it will work," and selling new ways of thinking has kept it up. The new Facebook search surfaced Matty's 2011 posts, where Matty was "a cynical asshole" about DevOps, including a coworker's comment that it was "great for tiny companies with no customers and no future." Bridget: "Yes, tiny companies with no customers and no future, like Etsy and Facebook."

Both hosts changed jobs during the year, Matty to Chef and Trevor to 10th Magnitude, with about a week of overlap. Matty says the show improved networking, and at least once a month a prospective customer says "you do Arrested DevOps, right?" Trevor says 10th Magnitude was picked partly to learn from Matty, and that being on the show is now seen as an asset there.

What's Coming in 2015

Ducy is the field correspondent, and might pop into a Hangout from wherever Ducy happens to be. Planned topics, with the promise that they won't promise when, include a promise theory episode with someone who lives in Minneapolis, blameless postmortems, DevOps jobs, a Microsoft and PowerShell episode, Docker, eventually consistent distributed systems, and building teams. Matty says the previous longest episode was episode 2, at one hour and seven minutes, and that this one will beat it.

Check-Outs

Bridget saw the giant interactive Google Android billboard in Times Square that features DramaFever, and recommends Lara Hogan's Designing for Performance. Trevor recommends the Apollo guidance computer rewritten in JavaScript, the Human app now on Android, and the Netflix series Black Mirror. Matty recommends the Guardians of the Galaxy Honest Trailer, the expanded Facebook graph search, the @PeteChessBot Twitter bot, and the Accompli mail app.

The Database: The Elephant in the Room Dec 09, 2014

Show notes

Why the Database Is Different

Grant Fritchey, the self-described Scary DBA and an enterprise DBA for about 20 years, and Jonathan Hickford, a product manager, both from Redgate, explain why databases are the elephant in the room of continuous delivery. Grant's answer is data persistence. Releasing software is straightforward: "You take an old piece of software, you throw it away, put the new piece of software in, you're done." You could do the same with a database, "but then the phone starts ringing because the business is freaking out because there's no more data left in the system."

Jonathan adds that data comes from several places, some generated by users and some configuration data from development, and often flows in the opposite direction from the code in a pipeline. A database may also be read by several applications and BI jobs, so a change has to be safe for all of them. Grant says data migrations for something like a new non-null column run against a live system and can cause downtime, and that part of the push for NoSQL comes from the difficulty of database deployments.

Reviewing the Script the Day Before Is Too Late

Grant hates the common DBA habit of looking at the script the day before it goes to production, which means the first run happens in production. Matty has seen DBAs who ran the developers' scripts without reading them and says that adds no value. Grant, self-deprecating, says the hard way taught that on Thursday you can't stop a Friday deployment: "What you have to stop is bad development," which means going to the scrums and standups and knowing what's happening six months, six weeks or six days before, not six hours.

Jonathan says the core of continuous delivery is fast feedback, so a data problem should be found in development, and production deployment "should be boring." Grant says that line is getting stolen: people always ask how your plane flight was, and you don't want an exciting one, "because you don't want an exciting production deployment."

Tooling and Source Control

Grant says step one, though it sounds bad from a tool vendor, is tooling, since you can't get a database under source control by hand. You need differential scripts, alter scripts and migration scripts that move data, not create scripts rerun over and over. Grant's objection to ORM tools is that they only produce 1.0-style code, such as create table, when existing data has to persist. Jonathan says once the database is in source control you can repeat things, and test a refactoring against realistic data to find out it would take three hours over millions of rows. Grant adds that even knowing it takes three hours lets you schedule for it.

The Myth of Rollback

Matty says knowing you can't go back and are always going forward is a reason to test deployments continuously, since the old habit of testing your backout never happened anyway. Grant says only two rollbacks really work: restoring or undoing a snapshot right after the release, or redeploying forward. A script to roll back pieces is "such a lie because the data comes in over time and ain't nobody wants to get rid of it." So the deployment process itself has to fix mistakes.

Grant says test environments should be as close to production as possible with cleaned data, for example without emails or personal details, and that sometimes it's fine to refresh QA when it burns down. Matty describes a former 3.5 terabyte data warehouse multiplied across more than 20 environments and being told there was no time to make test data. Jonathan says you don't need every environment to be a copy: one for data size, one for data complexity, and lighter ones for fast changes. Jonathan suggests testing sharding and replication earlier, and prefers "the short, fat pipeline rather than the long, thin pipeline."

Blue-Green for a Database?

A question from IRC asks how to blue-green a database change. Jonathan says it's possible with patterns, such as splitting a column by letting the application or data access layer handle both versions and migrating slowly at quiet times. Trevor asks about schema changes and Grant's answer is one word: badly. Grant has done it with triggers keeping two copies in sync, and calls it a "massive undertaking and fraught with horrific danger," like Indiana Jones. Jonathan says if a DBA says the database isn't the place to tackle this, they're probably right, and offers versioned schemas as an alternative.

Matty adds that in agile shops the throwaway shim, like a bridge you won't need once the stream is gone, is a hard sell to a product owner with a user story waiting.

How DBAs Feel About DevOps

Grant says it's a mixed bag. The camp that upsets Grant most says everything's fine because they've always done it that way, and Matty's reply is "We have always been at war with East Asia." To reach them you have to document their pain: how long a deployment took, whether there was downtime or data loss, how long recovery took, and the same for the developers who hop through hoops before production, and take that to management. Others are ready, and Jonathan says at DevOpsDays and other conferences someone always asks in the Q&A "what are you doing about the data?"

Grant describes the approach as that of a "completely lazy bastard" who takes advantage of what development teams have already worked out about source control, labeling and branching: "it's down to more labor than thought." Grant supported 10 development teams by automating builds and deployments, letting developers write their own T-SQL and reviewing for 15 to 20 minutes a day per team. Jonathan says the best visits Jonathan has made had a DBA, a developer and a sysadmin in the room.

Common Tooling, Testing, and Meeting in the Middle

Matty worries about silos created by tools, like data developers stuck on a different source control system. Jonathan says vendors are becoming more agnostic and that using one system lets you make atomic commits containing both database and application changes, which Jonathan calls probably a prerequisite for continuous delivery. Grant says without everything versioned together, people had to ask which database version went with which application.

On testing, Jonathan mentions tSQLt and DBUnit, and Grant says you don't write a test for a table. What matters are tests on destructive changes, that a migration moved the same number of rows and the updated data looks as expected, for confidence and boring deployments. Jonathan adds monitoring after release. Grant says DBAs need to learn tools like TeamCity, Jenkins, Octopus and PowerShell, because "we can only automate all of that. So what is your job again?" and they need to meet developers in the middle, not stand "athwart the bridge stopping you from going to production." Matty says sysadmins need to get smarter about data too.

DevOps in the Enterprise Nov 19, 2014

Show notes

DevOps Is DevOps Is DevOps

This is Bridget Kromhout's first episode as official co-host. Bridget is an operations engineer at DramaFever with a past as "a BOFH at places many hundreds of times this size." The panel is Michael Ducy of Chef, Ross Clanton, who leads part of operations at Target, and Steve Pereira of MyPlanet in Toronto, who helps big companies get started with DevOps and Agile and runs the DevOps and OpenStack meetups there.

Michael's position is "DevOps is DevOps is DevOps": the principles of continuous improvement are the same everywhere, and an enterprise just has more, sometimes political, nuances to dance around. Ross agrees, whether you're an enterprise, a startup or a unicorn, and adds that a large company like Target carries the technology debt it built up over the years, and a mindset shift across a large organization is a challenge that's "rewarding, it's frustrating, it's all kinds of things." Steve calls DevOpsDays "this magical camp where we all sort of get away from our jobs."

DevOpsDays Inside Target

Ross and Heather Mickman, a colleague, started an internal DevOpsDays at Target almost a year earlier, and have held three, quarterly, with a big annual one coming in February. The tagline is Connect, Share, Learn, and the aim is to get people across silos together by choice. The first drew 160 or 170 people, with Michael and an external speaker from Nordstrom, and Michael's talk "The Goat in the Silo" is still discussed by engineers there. Michael says enterprises don't realize they have enormous tech communities inside them that they haven't used, and that when Ross and Heather spoke at DevOpsDays Minneapolis about Target's transformation, it changed people's assumptions about the company.

Do Enterprises Just Need the Automation Part?

Matty says enterprise DevOps is often read as CAMS without the C, on the assumption that they don't need the hippie-dippy culture, just good automation and stats. Matty adds you can't cherry-pick, and that the traditional DevOps community talks about empathy so much this year because it feels it already has automation figured out. Ross says culture is the most critical part, since it amounts to mindset, and the rest follows as an outcome. Ross heard at the Enterprise DevOps Summit a mix of top-down and bottom-up approaches, and the theme that everyone is now shifting to optimize for speed.

Steve says many of those companies' transformations took three to six years, and that culture is like steering a giant tanker, and reports hearing that many started because they were in a mess and had no other option. Steve hopes that won't continue, now that there's data. Michael disagrees: the body of knowledge was already there, and real change comes only from "oh my God, I have to," so "there's going to be a lot more bloodshed." Ross's own moment came from reading The Phoenix Project, with a sense of having lived its characters. Ross says the way to make DevOps stick in a big company is to find each person's pain points and help them past those, which is a localized and time-consuming approach, and not to inundate them with concepts like promise theory.

The Conference Bridge Problem

Bridget tells of meeting three people from Thomson Reuters at DevOpsDays Ghent, who mostly wanted to hear about team chat. Their pain point was incidents, where everyone sits on a conference bridge and each new arrival needs a summary, so they hear it "like 14 times in a row." Bridget showed them Slack, and two of them downloaded it at the table. Ross says persistent chat has created a lot of aha moments in some groups at Target too.

Sharing, Open Source, and Auditors

Matty says a few years ago it seemed only Netflix, Facebook and Etsy were doing DevOps, but the larger companies just weren't sharing, and some places told employees not to tell anyone how they did it. Ross says most enterprises don't fully embrace open source and rarely let people present or blog externally. Target's public tech blog was only three or four months old, and it's starting to contribute to open source, though "it's going to be a slog."

Steve says one of the best threads at the Enterprise Summit was audit and compliance, with auditors on panels explaining how working with them within DevOps principles benefits a company. That matters, Steve says, because the fear that devs will run wild on production has kept many from taking DevOps seriously.

Culture Isn't a Checklist

Steve says the culture side is what everyone finds most intimidating, and that sharing is how someone at the bottom gets the conversation with leadership started. Bridget recalls Steve's Ignite about DevOps as a MacGuffin, and Michael's slide of an overly process-driven dystopia, where holding too rigidly to something like CAMS becomes dogma too. Michael says tools and even processes are easy, and that "the organizational management issues are the real challenge." Michael adds that the summit consolidated a lot of enterprise information in one place, and proposes an enterprise DevOps podcast. Ross is game, suggests "Arrested Enterprise DevOps," and Bridget looks forward to listening "from a safe distance."

One Piece of Advice

Michael's advice is that it's "probably going to piss a lot of people off" and you need to find allies. Ross says be very patient, and describes a Venn diagram that colleague Jason Walker draws on whiteboards, with three circles, collaboration, empathy and experiential learning, and DevOps at the center. Steve's is that sharing is how you start. Matty's is that you probably already do some of this under other names, and that with a transformation you should do a pilot and stack the deck for results. Trevor's is to advocate your problems and say what hurts, since you'll find everyone shares the same hurts. Bridget says you can't do everything at once, so if all you manage is getting ops into scrums, that's a start, or adding someone who knows what's going on to the change advisory board.

Checkouts

Ducy

The Goat Farm podcast

Ross

All the great talks from the Enterprise DevOps Summit, DOES14
Target Tech Blog (post on flashbuilds)

Steve

ALL THE TWEETS!
Debian snapshots
Amazon Lambda/CodeDeploy/Containers (in case it’s not covered in the AWS recap)
My recap doc of DOES14, pretty rough but shareable

Bridget

devopsdays.org - for a devops near you, or to be inspired by talks at past ones so you can hold one inside your org
Lots of great talks last week at DevOpsDays Vancouver, and one you should definitely watch is Stephanie Van Dyk, a Google SRE who worked on the healthcare.gov rescue.

Trevor

Worm Robot
Big Hero 6
Barbiefail

Matt

Yak Shaving Expert t-shirt
Customizing Chef by Jon Cowie
Have you tried turning it on and off again supercut from the IT Crowd

on-call_burnout_podcast — **git 101 with Emma Jane Westby** Nov 13, 2014
Show notes

Microsoft is open-sourcing .NET and creating the CLR for Mac and Linux .Net core5 is the new framework for 5, you can ship your own version of the app.

There is now a free version of Visual Studio — Visual Studio Community for open source developers and students. In fact, it is all happening out in the open on github.

Emma Jane is a long time listener of ADO! She has been teaching version control for many years with specific emphasis on the communication behind version control in teams. She has since switched to distributed version control such as git. Her aim in her teachings is to create resources that make git ”less painful” than it currently is.

Distributed vs Centralized Version Control

Emma Jane:

In distributed VC the DB that contains the changes, exists on the local system and I can have multiple connections to multiple DBs with other versions.

Centralized is all in a single DB, locally.

How is Distributed VC relevant to DevOps?

Matt: Many people hold the theory that you cannot have “The DevOps” without distributed version control. It implies communication through teams, so what is the validity of that statement.

Emma Jane:

Git is not the only VC option out there, but it is the most popular currently.

You need to assess your team and your project, along with the related expertise and community support, and go with the one that fits your needs.

As soon as you say “can’t” someone will prove you wrong.

Testing

Matt:

The whole basis of git-flow is based on the fact that you can’t trust your contributors. Especially with open source. It sends a message that says “I don’t have to test my shit, because you’ll do it for me.”

Emma Jane:

If we are talking about testing, you need to have full coverage testing of whatever your product. Many testing frameworks allows for 99.9% accuracy on the tests, but that .1% causes you not to trust your tests. This makes it really hard to get reliable CI into the dev process.

For that matter, Devs shouldn’t trust themselves when it comes to pushing code, you should always rely on testing because everyone is going to make mistakes, and humans might not catch them.

Git allows you to have control over the pushed code.

Trevor:

There should be no permissions. All developers should have the same permissions and the flow should go through QA.

How do you learn git?

Emma Jane:

All kinds of people are interested in learning git. But mainly:

Someone who is on subversion and wants to change to git

Someone who has been told to use git, but they don't know how to run command line tools.

CTO or management types that know they want to use git, but they’re not really sure where to go from that decision.

In order to identify how your team will most efficiently use git, draw out your team flow and identify where efficiency is being blocked. Is re-basing causing problems? Is a PR sitting out there for too long? Use those as discussion points with your coworkers.

You cannot introduce creativity when you are just told to memorize commands.

Emma Jane: - Use Interactive Add! It allows you to split up your diffs into different commits. So you don't end up committing a huge chunk of features that should most-definitely not be committed together.

How do I get set up?

Look for the right git-flow based on the type of deployments you are going to be using to release the software. Are your deployments feature based? or time based? How important is a rollback?

Your code should always be deployable in a CD framework. You are only rolling forward, you have one master branch, and feature branches, how can you have correct and fast CD if you have multiple branches before the CD process starts.

Your git setup should be directly related to your infrastructure. The git releases and flows of a team of 1 is going to be massively different than the git flow of a large team for a Could Provider.

Things you should know (about git):

Rebasing:

Rebasing allows you to recombine how your commit chunks are strung together. It takes all the commits of a branch and

Great for when you are adding too many commits.

Git Bisect

You can take out commits individually, and assess if the commits are in a working state.

However, if you do not have full commits, for example, commits when you are just thinking about something, it will be much harder to assess the state of the commits individually.

Source of Emma's talk about Git: http://github.com/emmajane/gitforteams Post version: http://24ways.org/2013/git-for-grownups/ Recording: http://prague2013.drupal.org/session/git-makes-me-angry-inside

Emma's rant about storing the history of your project: http://gitforteams.com/resources/evolution-social-coding.html

GitHub conversations: http://guides.github.com/introduction/flow/

Check-outs
Emma

Kaleidoscope mergetool - because of image diffs

Sketch training for techies - because of impostor syndrome for drawing

GNOME wins - go open source!

Matt

Teamocil - generator for tmux sessions

The NoPhone - kickstarter for a “technology-free alternative to constant hand-to-phone contact”

“It’s not a promotion, it’s a career change” - blog post by Lindsay Holmwood

Trevor:

Sandisk SSD

Android L: Inbox

Rosetta Probe

managing systems in the cloud Oct 15, 2014

Show notes

What "Cloud" Means Here

The episode opens with the show reading its first glowing iTunes review, titled "Two Chimps and a Mic," which gives five stars because the reviewer "had no clue what they're talking about." Trevor's only correction is that there are two microphones. Then Tom Limoncelli, co-author of The Practice of Cloud System Administration, explains why the book exists. Tom's first book came out in 2001, and system administration has since shifted from help desk and machine administration toward service administration, including what Tom learned at Google.

Tom says the word cloud "has been taken by marketing people and kind of destroyed" (the publisher wanted it in the title, so the book's first use of the word is to explain what they mean by it). What they mean is distributed computing, systems so large that work is divided among many machines, as opposed to the 1990s pattern of buying a bigger and bigger server, which stops scaling because machines only get so large. Trevor asks for horizontal versus vertical scaling, and Tom admits to keeping a cheat sheet on Tom's palm: vertical is one machine getting bigger, and horizontal is dividing the work across 8, 80 or 800.

Data Over Hunches, and Analog Systems

The first skill change moving to Google, Tom says, was that monitoring became far more important, because in a big distributed system no one person understands everything, so you have to instrument it and make decisions from data. Tom tells of the second or third week there, when Tom offered a solution from senior sysadmin experience and a very junior sysadmin pulled out a graph showing that the change Tom had made moved the line the other way, so they should do the opposite of what Tom said. Tom says it was humbling and "such a better way of doing system administration." The second change is that at large scale downtime is more visible, and you get good at sensing a sick system and fixing it before it becomes an outage, not at responding to outages.

Matty asks about servers as interchangeable cogs, and Tom says "the components don't matter, the system matters." RAID is Tom's small-scale example of decoupling component failure from service failure. In a very large system, Tom says, computers aren't up or down but degraded: "Gmail is never up or down. It's running at 90% up or 80% up. It's an analog thing."

Your Architecture Is Wrong, With Details

Trevor's clients are finding their software is all-or-nothing, and Tom says that's why sysadmins need to be at the architectural level, not saying "we don't use that." Even in the most distributed system there's a dirty secret: "there's always that one thing that can't be replicated," like a lock server. When Matty cites Jez Humble's "your architecture is wrong," Tom says a sysadmin who says that gets asked how it should be done, and often doesn't know. So the first third of the book is distributed architecture explained for sysadmins, "like a boot camp for the architectural decisions," and the second part is operating large systems, with on-call and monitoring.

The book's "surprise ending" is an assessment system, twelve questionnaires you rate from 1 to 5, and if you redo it monthly in a spreadsheet you get a heat map of improvement, going from red to green. It can be done per service, and rolled up across teams so a CIO can see where to move resources. At Stack Exchange, Tom says, the Director of Engineering used it to focus on the top-priority service.

Twenty-Seven Handoffs

To find where to start, Tom uses a lean analysis to find the bottleneck, or gets silos into a room to talk about the pain they cause each other. On one project the team analyzed one iteration and found 27 handoffs across 15 teams, and "no one had ever drawn a picture" of it. They sat down with each team and walked through the graph, and found many unnecessary dependencies and undocumented ones that explained delays. During a break, someone from one team cornered Tom, embarrassed to admit they'd never realized their delays caused five others. Tom says "an executive standing in front of everyone saying we have to break down the silos" has never broken one, but sitting down with another team builds empathy. In a sorting example, one team always handed the database down unsorted and every team after them spent two hours sorting it, when it could have been sorted once.

At Stack Exchange, Tom adds, the first disaster recovery failover drill took 10 hours, then 5, and now 1 or 2. In one drill they filed about 30 bugs, and half were closed before it ended because developers watching operations do the steps built a button that did it in one click. Matty says you can't have "the empathy project"; Tom agrees the goal has to be improved service or uptime, and that empathy is the route. Tom's illustration is continuous integration, where the point isn't a two-hour build but the confidence to push constantly, and Tom compares software to a car company that warehoused every car for a year before selling it.

Hipsters, Unicorns, and Damn Humans

Matty says jargon can make the field intimidating, and mentions the blog post Matty wrote on hipster DevOps. Tom says technology always has the top 5% inventing things and someone has to push it to the other 95%, and is glad the DevOps community has stopped acting as if enterprises would never understand. On J. Paul Reed's closing talk at DevOpsDays Chicago about ending the unicorn talk, Matty says the unicorns themselves say "we're not unicorns," and that the concept of "the other" propagates the myth that DevOps only works at unicorn companies. Tom points to Todd Underwood's LISA talk debunking the idea that Google's problems are unique, and says every company has the biggest problem, "those damn humans." Tom cites The Forbin Project, where a computer concludes the real problem is people.

Coping Mechanisms and "Bring Us the DevOps"

Tom's time management book for sysadmins, which Matty has on Matty's shelf, is a list of coping mechanisms, since "no human actually has good time management skills." Tom kept it around 120 pages, with the top three tips in the first 20, because "always front-load."

Asked what to do when the boss says bring us DevOps, Tom says to find out what they mean, since after a month you may hear "I don't see any difference" because they wanted something else. It's like "bring me the cloud," which to executives usually means getting a server in 15 minutes and not waiting 6 months, and to consumers means their photos are backed up. If they don't know, cynical Tom says fix your biggest pain point and call it DevOps, while non-cynical Tom says turn business priorities into measurable results, automate measuring them, and work on projects that move them.

Check Outs

Tom

Stack Exchange is open sourcing its monitoring system called “Bosun”. Look for the presentation at Usenix LISA http://www.usenix.org/conference/lisa14/conference-program/presentation/brandt

Matt

pester busser for test kitchen by Jay Mundrawala (discussed in Matt Wrock’s post which I will link to because it is a long url)

Riffsy ios8 keyboard for animated gifs

Trevor

http://www.jeremymorgan.com/blog/programming/the-great-unicorn-hunt/
Borderlands the Pre-Sequel is out :D

devopsdays chicago Oct 09, 2014

Show notes

Day 2 of the First DevOpsDays Chicago

Matty and Trevor record in front of a room on day 2 of the first DevOpsDays Chicago, the second in the Midwest after Minneapolis, and Trevor's first DevOpsDays. Joining them are J. Paul Reed, who is "defecting from The Ship Show" for one episode, Michael Ducy, who is named the show's "special field correspondent" for events the hosts can't attend (Michael's reaction: "I'm the Walter Cronkite of DevOps now"), Jason Hand of VictorOps, and Mark Cornick, an Ignite speaker who has been on a self-described DevOpsDays tour this year.

Mark's summary for anyone who hasn't been: it's highly focused on learning, full of ways to participate, and inexpensive, since Mark hasn't paid more than $100 for a ticket. It's usually two days, with invited talks in the morning and an open space in the afternoon, and the sponsors are "just here to be part of it on the same terms as everyone else."

The Midwest's Turn

Michael says the Midwest tends to lag in technology adoption, going by data from RedMonk, with New York a little faster and San Francisco "so far in the future." Over the last two years, though, DevOps meetups have taken off, with Minneapolis one of the largest in the country, and Michael thinks the Midwest is doing a lot of cool things without getting together to talk about it. Paul says that on the coasts people are building "Airbnb for your cat," while these cities host industries that are the backbone of the country, like Target in Minneapolis and the futures exchange in Chicago. Michael lists the enterprises that turned up this week, including Allstate, Kroger, McDonald's, Kohl's, and Crate and Barrel.

Paul tells of talking to a Crate and Barrel attendee after having ordered wine glasses two weeks earlier. When Paul mentioned the checkout hadn't gone smoothly, the attendee said the feedback was useful, since they'd changed how they handle cookies, and handed over a test credit card number to try again. Jason adds that at both Minneapolis and Chicago, a large majority of the room raised a hand as first-timers.

Converted by Open Spaces

Matty, one of the organizers, admits to having sold Chicago short during planning. Matty proposed a blameless postmortem open space, thinking nobody knew about them, and heard that it "got really advanced really fast": "nope, dummy, we know that." They sold out at over 300 people and turned some away, and day 2 turnout was nearly the same as day 1. At the evening event, at least half a dozen people told Matty they'd thought open spaces sounded dumb and changed their minds.

Jason was skeptical too, and now open spaces are Jason's favorite part. Mark, a self-described introvert, says one of Mark's first open spaces was Tom Duffield on enjoying tech conferences as an introvert, and that "as a result, here I am": an Ignite talk followed, and now a spot on this podcast. Justin, a developer from Chicago at a first DevOpsDays, expected someone talking at the audience and then drinks, and instead found topics too small for a full talk but worth discussing. Matty notes that badges say participant, not attendee, because the conference is meant to be interactive.

The 37th Floor and the Arcade

The venue was on the 37th floor of the Sears Tower, which Justin still calls Sears and Paul calls "the name that shall not be spoken," and Paul says it's the highest DevOpsDays anyone's been to. The hashtag was #DeepDishDevOps, and the evening event was an arcade with Dig Dug, Joust, and Teenage Mutant Ninja Turtles, which Paul says is a lot harder when you're not 10 years old and crowding around the machine.

How Chicago Got a DevOpsDays

Matty says the email to Michael went out in spring of the year before, asking why Chicago didn't have one. Michael says the Chicago meetup was one of the first in the country but "a little flaky at times," a remark an organizer has ribbed Michael about, and that in Minneapolis, at a social mixer with four people standing around, Bridget Kromhout put a hand up and said "I'll organize it." Nobody in Chicago had been passionate enough to take it on. Matty asked Patrick Debois for the names of people who'd shown interest in 2013, held a kickoff meeting and sent a SurveyMonkey survey, and ended up with a group in maroon shirts who mostly didn't know each other. It was, Matty says, very flat.

Matty thanks Minneapolis for its 18-page postmortem document, and says you have to make it your own, though it's nice "when someone else has made a couple mud pies." Early on there's little to do and at the end there's a lot, so around six weeks out Matty asked for a reaffirmation of a loyalty oath they had never actually taken. Shannon, another co-organizer, compares it to planning Shannon's own wedding, and says what stands out is that everyone is a volunteer with no ulterior motive. Aaron says organizing gave more than attending would have. Organizers weren't allowed to give talks, first because of conflict of interest and then because they'd have no time, and they ran an on-call rotation on an email address, an idea borrowed from Minneapolis. Michael adds that Amsterdam went all out with headset walkie-talkies.

Compliance, and DevOps Equals 42

Michael says the talk to look up is the compliance one, since there's "a lot of fear and uncertainty when you talk about moving faster" and the community rarely discusses audits. Paul liked that it defined the auditors' vocabulary, given that operations people speak their own jargon at everyone else: an auditor asks whether you have a control for something, and ops people have no idea what that means.

Steve Pereira, a consultant visiting from Toronto, says Paul's talk "DevOps == 42" was a highlight, because Steve had expected a talk claiming DevOps was the answer and got one about not knowing the right question. Paul says Matty came up with the title, that the schedule link went to no abstract, and that it seemed so appropriate that the abstract stayed empty.

Check-Outs

Michael learned about the twelve-factor app methodology in an open space on hybrid cloud, where most of the developers hadn't heard of it. Paul recommends the #DeepDishDevOps hashtag and the Giordano's pizza served at lunch. Trevor recommends a terminal library that plugs into the Jimmy John's API so you can literally "sudo make me a sandwich," and the local Daisy Cutter beer from Half Acre. Steve mentions rollout.io, which is about pushing hotfixes to a mobile app without waiting on App Store approval. Matty's is DevOps Against Humanity, a Cards Against Humanity-style deck of DevOps jokes started by Bridget Kromhout, played at the after-party where a few people won Raspberry Pis.

Conference Love Sep 24, 2014

Show notes

First Conferences and the Hallway Track

Jason Dixon of Librato, who started the Monitorama conference, Bridget Kromhout, an operations engineer at DramaFever who helps organize DevOpsDays Minneapolis, and Pete Cheslock of ThreatStack discuss why conferences are worth attending. Their first ones vary widely. Bridget's was LISA '95 as an undergraduate. Matty's was COMDEX in about 1996, which was really a trade show, and then there was no other until TechEd 2008. Pete's was Surge in 2012, because earlier employers didn't see the value of sending people. Jason's was probably LinuxWorld in New York in the early 2000s.

What you get that you can't get another way, Pete says, is the hallway track, the people chatting between sessions, which at many conferences "is actually the most interesting one." Jason isn't there to learn from the talks, since there's plenty of content online. Jason goes to meet the developers behind open source projects and have the conversation comments can't give you. Bridget's example is standing in a hallway at Velocity Santa Clara in 2013 and asking an AWS employee about Hadoop and MongoDB problems. The employee turned out to have written the MongoDB on AWS white paper. Bridget went home and re-architected everything, moved to SSDs and removed unnecessary sharding, and it cut enough off the Amazon bill that Bridget is "pretty sure that paid for the conference plane ticket."

Networking Isn't a Dirty Word

Engineers hear networking and, "if it's not dealing with routes and subnets," tune out, Pete says. But Pete can reach out to old contacts from years ago and ask if they've done something, or know someone who has. Pete pitches conferences to managers on recruiting: tech recruiters often receive 20 to 30% of a first-year salary, so instead of paying a recruiter $10,000 or $20,000, send engineers to conferences. Pete met Jason at one, they stayed in touch, and when Jason was looking, "you kind of jump the line." Trevor says meetups do the same job, and that's how Trevor and Matty met, at an Azure meetup.

Start Your Own, or Sit at a New Table

For people who don't know anyone, Jason's answer is admittedly useless: start your own conference, which is what Jason did to get the speakers and interactions Jason wanted. The market, Jason thinks, is moving toward small, community-focused conferences. Matty likes that as making the thing you want. Bridget went to Velocity Santa Clara knowing only one person, and "sat at a different table at lunch every day," looking for people with blue and pink hair, most of whom turned out to work for Etsy. Pete notes everyone at lunch is there to meet everyone else.

Matty, who describes being very introverted, tweets to the conference hashtag a week or two before and follows it, so the first introductions have already happened virtually. Bridget says if you're not on Twitter, stop listening and make an account, since badges often show handles. Jason put avatars on Monitorama badges because people know the avatar and not the face, and admits to feeling like an outsider at a front-end conference where everyone knew each other. Trevor mentions imposter syndrome, and Bridget's tip is that "pretty much everyone loves to talk about themselves," so ask polite questions and listen.

Leaving the Echo Chamber

Bridget, Jason and Pete all find events through Twitter, and Pete worries that it's an echo chamber. At DevOpsDays Boston about 10% of the room had been to one before, and Bridget says over 80% in Minneapolis were first-timers, with a later poll at 75%. Pete found a language-specific event, Mountain West Ruby Conference, was a change of scene. Pete also spoke at an Agile conference and polled the room on company size, finding about 90% worked at companies of 10,000 or more, "a room full of Scrum Masters," attacking the same problems at a scale none of the panel can imagine. Bridget points to the Enterprise Scale DevOps conference for that audience.

Swag

Pete has too many conference bags now that messenger bags are too much for Pete's back, and had to work hard to get a Monitorama hoodie from Jason. Jason isn't anti-swag but likes practical stuff, and as an organizer wants people to think about the landfill footprint: Monitorama's badges and lanyards are biodegradable, and tries to talk sponsors out of excess swag. Bridget says DevOpsDays Minneapolis gave speakers small battery packs, and that DevOpsDays New York skipped t-shirts and used the money to build a well in Cambodia. Pete's suggestion is to offer donating instead of a free t-shirt at checkout, because "who needs more stuff really?"

Conference Etiquette

Bridget describes a conference as a space where you're at work but not really at work, with bad Wi-Fi, alcohol, and people you don't know well, so professional boundaries get fuzzy. Bridget's rule: have fun, but ask "would I like these people to try to recruit me for their next startup?" Bridget also says being welcoming means not making assumptions about people, like assuming a woman must be a recruiter or an older person is trying to sell you something, and instead asking open-ended questions. Trevor has seen people ask a third party whether someone is a recruiter or an engineer, when the answer is "go talk to them."

Conference Stories

Bridget got Patrick Debois to fly from Belgium for the closing keynote of DevOpsDays Minneapolis, and Andrew Clay Shafer, who couldn't attend, introduced Patrick over a Google Hangout, so a "giant floating head" appeared on screen as a surprise. Matty spent an hour talking with Jeffrey Snover at ChefConf and offered Jeffrey the opinion that PowerShell was as readable as Perl.

Jason tells of an OSCON booth demo of a failover firewall on OpenBSD, where Jason gave the whole spiel to someone who, as Jason tells it, turned out to be the creator of FreeBSD, which went unnoticed until the people behind started laughing. Jason also says Monitorama's diversity effort included inviting a speaker on the subject, and that the audience's reaction moved Jason. Pete's story is a sushi dinner in San Jose after Velocity where the person across the table turned out to have written a project Pete depends on, and Trevor's is being asked "are you that guy from the podcast?"

What was the first tech conference you attended?
What are things you get from a conference that you cannot learn other ways?
How can I maximize my value out of attending a conference?
If I’m going to a conference where I don’t know anyone, how can I still have a good time?
You both have planned conferences. What are some of the things that go into organizing that people might not be aware of?
What is your favorite conference story?
How do you learn about new events to attend?
How do you pick which events you know about to go to? There are a lot, and it can be hard to narrow down when you only have a 1-2 conference limit from an employer, or your own resources.
What was the coolest piece of “swag” you got from a conference?
Does the swag at a conference weigh in for you at all? I hear a lot of noise around the big Google events because everyone knows they are walking away with hardware.
Lets talk about conference etiquette, and discuss some of the points in Bridget's article

Notes:

Introversion and Tech Conferences by Tom Duffield

Check Outs

Jason

Some of my favorite talks, first a couple from John Rauser:
- Look at Your Data
- Investigating Anomalies
And another recent one from Kyle Kingsbury at Strangeloop:
- Jepsen II: Linearizable Boogaloo

Bridget

Local meetups on http://meetup.com (and this is the Minneapolis startup I mentioned: http://congruence.io)
MonkeyLectric Bike Lights: http://www.monkeylectric.com

Pete

FPM - http://github.com/jordansissel/fpm
Deb-S3 - http://github.com/krobertson/deb-s3
STM Aero Backpacks - http://www.stmbags.com/catalog/laptop-backpacks/aero-small-laptop-backpack/

Trevor

Borderlands the PreSequel: http://www.youtube.com/watch?v=wpgMBivKR-w trailer video is hysterical. Whether you’re going to play or not.
Jetbrains free for students *(.edu)
Netflix spoilers http://spoilers.netflix.com/spoil-yourself

Matt

Tech Douchebags podcast - http://5by5.tv/tdb
Spoonium (containers for Windows) - http://spoonium.net/
Columbia Treadlite 10L Backbpack - http://www.amazon.com/Columbia-Treadlite-Backpack-Black-Size/dp/B0058XJXZW (hattip to Ryn Daniels @beerops)

Something About Security With Ben Hughes Sep 09, 2014

Show notes

Security, the Etsy Way

Ben Hughes of Etsy is one of the senior network engineers there, though Ben's team looks after infrastructure, about a thousand machines and a hundred laptops, and leaves networking to a NetOps team. The guest fell into security after dropping out of school on discovering 2600 and Linux, and at one point had around 1,500 accounts on the Unix system at Ben's high school. Ben came to Etsy from Puppet Labs, learning about DevOps Days there.

Ben's current work includes Python on Midas, a host-based intrusion detection system for the Mac that Etsy released with Facebook, and Go to trick Logstash into using different TLS ciphers. Next week Ben is at an incident response conference, because for breaches "the days of it being an if are long gone."

Blameless Phishing

Matty recalls Ben's point about phishing: the question is what you do once someone gets phished. Ben says Etsy focuses on people first. You can't shout at a recruiter for opening a PDF, since that's their job, and "blameless postmortems work for blameless phishing campaigns." What matters is that people tell the security team when something screwy happens, and "the more you chastise people, the less they will tell you." So when someone reports a phish, they say thank you and give them an Etsy gift card. Being able to respond matters more than believing you have full coverage: "Nothing's 100% in security, despite what some people will try and sell you."

Scary Clouds and the Trusted Network That Isn't

Matty raises the belief that you can secure your own stuff better than a cloud provider. Ben says that if you bring the "armadillo" security model of the last 20 or 30 years into the cloud, "you're going to have a bad time," because unless you build a private network, there's no trusted internal network at all. Matty passes on a Microsoft data center manager's remark that after the US government Microsoft is the most attacked entity on the internet, so who better to learn from.

Ben's caution is about outsourcing security to someone else. Without a complete overview of your organization and your threat model, "you're going to be hit by a surprise," and security vendors sell silver bullets using fear. Etsy is big enough to build its own tools, as Netflix and Square do for their particular problems, and those tools "you can't buy commercially."

Don't Throw It Over the Wall to Security

Ben says throwing something over the wall to security at the last minute is as bad as throwing it over to operations, and everyone who's had a bad time says they should have talked to security sooner. Given budget to allocate, Ben would spend it on a security team early, not on products or pen testing, because until you map what you're defending and from whom, you can't build defenses. Etsy's first security person was one of its toolsmiths who took on security on the side, and Ben says you can find people interested in security in operations, development or QA.

For developers, Ben's number one request is "please stop turning off TLS verification," and the age-old rule of never trusting user input: "never trust users," in the nicest possible way.

Proxies and the People Who Route Around Them

Matty rants about transparent HTTPS proxies that decrypt traffic, which break a gem install because the certificate doesn't verify, so people turn verification off. Ben gives the benefit of the doubt, since there are good reasons to inspect web traffic, and suggests trusting the proxy's CA cert in the gem config. The guest says people will route around anything that stops them doing their job, with SSH tunnels, OpenVPN or tunneling IP over DNS: "However you need to get out of a network, you will." So "we can either work with these people or kind of be avoided by these people."

Trust famously doesn't scale, Ben says: at 50 people you know everyone's name, and at 10,000 you don't trust other buildings. Etsy defaults to trusting everyone, monitors everything, and is transparent about what the security team is doing, which is how you get trust back. The guest sums up DevOps as "why don't we work together?"

Secrets, Passwords, and Breaking Things on Purpose

For a listener's question on secrets, Ben says Chef encrypted data bags "turn your encryption problem into a key management problem," and points to Nordstrom's Chef Vault, which Matty says ships with ChefDK. The guest would also like Square to release its resident-only in-memory file systems built on FUSE. Ben's main advice is to get to where you can change a secret and nothing breaks: once sharing works, randomly change some passwords, "it'll break a ton of stuff, and you'll have a terrible day," but you'll know next time credentials leak, which they do. "Pastebin is a treasure trove of other people's mistakes, and GitHub Gists is just a shopping cart full of logins."

Personally, Ben uses a password manager with around 600 credentials and two-factor on everything. Matty asks front-end developers to name password fields password so managers can find them, and mentions a six-word Diceware passphrase, learned by making 1Password prompt for it every time for a couple of days.

What People Get Wrong

Ben's first misconception is that HTTPS is too slow, and the rebuttal is istlsfastyet.com. Ben's second is the media's fixation on zero days, which "99.99999% of organizations do not need to worry about," when the unpatched Linux kernel and users' four-character passwords are bigger problems.

Trevor asks how to start security where there's no team. Ben says find the person who stays up late on IRC, give them time, and get leadership to treat security like backups. Some security is better than none, and "smaller incremental gains is always the way to do it." To sell it, Ben says you can do it "not out of fear, but out of this is the responsible thing to do," and customers pick the vendor that's more secure. Matty realizes there's no padlock in a mobile app to tell you whether traffic is encrypted, and Ben says the ideal is for platforms to show one only when certificates are validated. The guest tells of a game site that showed a picture of a padlock next to an HTTP URL.

The Villain in The Phoenix Project

Matty asks whether InfoSec is still the right name, given the band Information Society, and Ben says it's still used, that Ben's own LinkedIn said Security Monkey for a long time, and that hacker versus cracker is settled. On The Phoenix Project's security manager, who's cast as a villain, Ben says many see security as Big Brother, and that Etsy's team held a hackers party watching the film Hackers, taught people to pick locks, and ran a security hack week. If you're the villain, Ben says, "start being nicer to people because you work with them."

Ben also thinks security needs to buck up its own ideas: the military jargon, like the phrase threat intelligence, which "brings me out in a rage," makes the field seem aggressive. The guest wants less preaching that it's 100% or nothing, noting that even CPUs ship with errata lists, so "computers are broken from the ground upwards" and it doesn't all have to be doom and gloom.

What exactly do you security folks do all day?
So let's talk about ZOMG SCARY CLOUDS
I'm a developer. What do I need to know to help me be pals with InfoSec?
Ops people know security, right? Or not?
Common security mistakes/misconceptions
How can InfoSec be better buddies with other functions?
Are you tired of Matt calling it "InfoSec"? Does it remind you of Information Society?

Check Outs

Matt

zsh and oh my zsh - http://github.com/robbyrussell/oh-my-zsh
zmojii plugin
all of the chef 12 things!
st. tom waits prayer candle - http://www.etsy.com/listing/177742614/saint-tom-waits-prayer-candle

Ben

http://cybersymposium.isis.poly.edu/symposium/ The NYU “Bridge to cyber security: a women's Symposium” which is a two day event aimed at introducing women of all ages and points in their career to security.
(if you’re gonna talk oh-my-zsh, you should really talk prezto - http://github.com/sorin-ionescu/prezto)
http://gauntlt.org/ by James Wickett (of Signal Science) and Mattjay (of Whitehat security) (and others) is pretty amazing CI/CD tool for security testing your code.
http://twofactorauth.org/ list of places that do and don’t have two factor authentication. cough iCloud drama cough

Trevor

http://www.strengthsfinder.com/home.aspx Strengthsfinder- recommended to me by Kay Johansen at FlowCon, interesting analysis of leadership strengths
Amazon Instant Video is F&$king finally on Android (still needs Chromecast support built in, but screencast works just fine)
Rampage the boardgame. http://boardgamegeek.com/boardgame/97903/rampage

dev to ops Aug 28, 2014

Show notes

Ops by Necessity

Aaron Blythe of Cerner, and John Smyth and Nate Burleson, both senior consultants at 10th Magnitude, all started as developers. Aaron's first taste of ops was deploying a Rails e-commerce application solo: Aaron learned Bash on the fly, decided there had to be a better way, and went from Capistrano to Chef. John says the move into pure operations came because an application was performing badly on SQL Server, and that the split between the two "wasn't so clear-cut." Matty says John has talked about doing this for years, but "we used to just call it working."

Nate says the Unix sysadmins from early in Nate's career taught Nate Perl tricks, and were good programmers who knew the command line and automated their jobs. In Nate's telling the division between dev and ops is modern and artificial, though sysadmins did tend to keep programmers away from the systems, "for fear that we didn't know what we were doing, which was true." Trevor says the move begins with necessity: something doesn't work, you find it isn't your software, and if nobody else can change it and you can, "you become the vector for that change." Trevor's own start was manually changing permissions on the wwwroot folder and blowing up IIS.

What a Developer Brings to Ops

John says everything is layers of abstraction, and the deeper you know them the better you can troubleshoot. John's development background started with learning assembler and the hardware by programming it. Aaron says the exchange goes both ways: the development team teaches ops code reuse, and ops has a vast knowledge Aaron never had when the development environment was set up for Aaron, who couldn't have built it from scratch.

Code Review for People Who've Never Done One

Matty quotes Adam Jacob on cultural change: see what happens to sysadmins when you tell them they have to do a code review before an infrastructure change. Aaron says the development teams Aaron works with spend about half their time reviewing or testing someone else's code, which ops colleagues find hard to believe, and that with Lean, "you got to slow down to be able to speed up."

Matty admits not really knowing how to do a code review, and the guests explain. John says there's no right way but there's probably a wrong one: reviewing a sample of someone's code for style, or reviewing commits before they're committed. Trevor says review is subjective without agreed standards, like clean code conventions, and that it's mostly accountability. Trevor describes a project with 100 design patterns "because the person who wrote the code had no one to be accountable to." John says "there's no better way to make sure the code is readable than to make sure somebody reads it other than you," and that with Chef they check attributes are used consistently and things are decomposed. Aaron says tools now let you comment inline on a single line, like a stray rm -rf, and that review is a learning experience for new hires, who are told to grab a line and ask what it does.

Trevor likes pair programming for the same reason, and Aaron has paired with an ops person, staying late and going from whiteboard to keyboard to system, and says "when you hit that magic, it's pretty sweet."

Surprises

Nate was surprised by the immaturity of tooling in the Windows ops space, including source control and code review, since Nate had found the most technically adept people tended to be in ops. Trevor's surprise was how far over the line toward ops Trevor had already gone: building and configuring machines, adding users and firewall rules, because nobody else was in that role.

Incentives Without Friendship

A Twitter question asks about incentives for bringing dev and ops together. Aaron says there aren't concrete ones, and that the two groups sit on campuses 15 miles apart in Kansas City, so Aaron's team goes to the ops campus one day a week and ops comes to theirs two days a week, though no manager said they had to. They do it because they "want to be physically in the same location and humanize each other."

Matty says metricizing collaboration is tricky, and John says formal structure doesn't dictate relationships, since teams that are separate but whose people work together have accomplished the goal, and a joint team "doesn't prevent somebody from being a dick." Matty adds that collaboration doesn't require friendship. Matty and John worked together for years at a bank without socializing outside work and collaborated fine.

Learning the Other Side

Trevor's advice for a developer who wants to understand ops is to talk to an ops person, and Matty calls that not really advice, asking how. Trevor: "Like a human being?" John says the opportunities to collaborate are situational, and that when you have a problem you should ask questions and not pretend to know: "Nothing makes me crazier than when somebody feels the need to answer a question despite the fact that they don't know what the answer is." Aaron asks what the most frustrating part of someone's job is, and is blown away by the answers, since Aaron didn't know anyone had to worry about those things. Trevor adds not to ask "to ask the question," but "because you care." Matty asks how John would react if someone interrupted during a fire to say it looks stressful, and John says the reaction would probably be that of a sarcastic ass.

War Stories

Aaron deployed an application on a Friday at 5 o'clock, and during dinner with Aaron's wife, Aaron's boss's boss said the CSS wasn't loading. It worked on one of the two nodes and not the other, so Aaron had to leave dinner and fix it. John's was a single storage array that periodically ground to a halt and took down a fully virtualized data center. Nate released a new version of an error-logging SOAP service the day before Nate's honeymoon, and it caused an infinite loop of logging its own errors: "the plane lands in Argentina, and my BlackBerry starts lighting up." Nate's lesson was to never release on a weekend or before leaving town.

Matty tells one about a coworker who was handed a decommissioned database server, and panicked when the storage array wasn't visible. Matty checked the SCSI cables and found that, in a hurry, the coworker had plugged the two RAID cards into each other and the two storage arrays into each other, and for at least a year Matty called the coworker Loopback.

You Suck at Technical Interviews

John Vincent - DevOps The Title Match

Learn Linux

Check Outs

Nate

ConEmu (Console Emulator):

John

Nueske’s slab bacon

Aaron

Jez Humble’s ChefConf talk

Trevor

TeamCity

Encourage extension Visual Studio

Matt

Lumosity Mobile

The Sysadmin Show Aug 20, 2014

Show notes

What exactly does it mean to be a sysadmin? How do you define “sysadmin”?

What is your favorite thing about devops? How does it change your life as a sysadmin?

Matt has a belief that sysadmins are inherently cynical. Is he an idiot?

What do you miss about “the good old days?”

Alex Howells asks: “How often do you think 'Fuck it all, I'm going to be a plumber or electrician?'"

Video of guys changing tires while car is on two wheels http://www.youtube.com/watch?v=5WLwRg3erm4

Amazon Certified Sysops Admin Associate http://aws.amazon.com/certification/certification-levels/certified-sysops-admin-associate

Check Outs

Matt

LOLRoot - self signed cert authority (lolroot is actually from snorby,org, which is now owned by threatstack. all roads lead to Peak Cheslock)
Bastard Operator from Hell - http://bofh.ntk.net/BOFH/
Registration is open for Chef Community Summit - ADO listeners can get 10% off their registration with the code ARRESTEDDEVOPS

Our TOPPODCAST Picks

Follow Us

Stay Connected

How It Started

By the Numbers

Topic First or Guest First

Favorite Moments

Behind the Mic

The Retro Goes Missing

Where the Show Fits

What's Coming in 2015

Check-Outs

Why the Database Is Different

Reviewing the Script the Day Before Is Too Late

Tooling and Source Control

The Myth of Rollback

Blue-Green for a Database?

How DBAs Feel About DevOps

Common Tooling, Testing, and Meeting in the Middle

DevOps Is DevOps Is DevOps

DevOpsDays Inside Target

Do Enterprises Just Need the Automation Part?

The Conference Bridge Problem

Sharing, Open Source, and Auditors

Culture Isn't a Checklist

One Piece of Advice

Checkouts

Ducy

Ross

Steve

Bridget

Trevor

Matt

Distributed vs Centralized Version Control

How is Distributed VC relevant to DevOps?

Testing

How do you learn git?

How do I get set up?

Things you should know (about git):

Check-outs

What "Cloud" Means Here

Data Over Hunches, and Analog Systems

Your Architecture Is Wrong, With Details

Twenty-Seven Handoffs

Hipsters, Unicorns, and Damn Humans

Coping Mechanisms and "Bring Us the DevOps"

Check Outs

Tom

Matt

Trevor

Day 2 of the First DevOpsDays Chicago

The Midwest's Turn

Converted by Open Spaces

The 37th Floor and the Arcade

How Chicago Got a DevOpsDays

Compliance, and DevOps Equals 42

Check-Outs

First Conferences and the Hallway Track

Networking Isn't a Dirty Word

Start Your Own, or Sit at a New Table

Leaving the Echo Chamber

Swag

Conference Etiquette

Conference Stories

Notes:

Check Outs

Jason

Bridget

Pete

Trevor

Matt

Security, the Etsy Way

Blameless Phishing

Scary Clouds and the Trusted Network That Isn't

Don't Throw It Over the Wall to Security

Proxies and the People Who Route Around Them

Secrets, Passwords, and Breaking Things on Purpose

What People Get Wrong

The Villain in The Phoenix Project