The InfoQ Podcast

Latest Episodes:

Streaming: Danny Yuan on Real-Time, Time Series Forecasting @Uber Mar 31, 2018

Show notes

On this week’s podcast, Danny Yuan, Uber’s Real-time Streaming/Forecasting Lead, lays out a thorough recipe book for building a real-time streaming platform with a major focus on forecasting. In this podcast, Danny discusses everything from the scale Uber operates at to what the major steps for training/deploy models in an iterative (almost Darwinistic) fashion and wraps with his advice for software engineers who want to begin applying machine learning into their day-to-day job. Why listen to this podcast: * Uber processes 850,000 - 1.3 million messages per second in their streaming platform with about 12 TB of growth per day. The system’s queries scan 100 million to 4 billion documents per second. * Uber’s frontend is mobile. The frontend talks to an API layer. All services generate events that are shuffled into Kafka. The real-time forecasting pipeline taps into Kafka to processes events and stores the data into Elasticsearch. * There is a federated query layer in front of Elasticsearch to provide OLAP query capabilities. * Apache Flink’s advanced windowing features, programming model, and checkpointing convinced Uber to move away from the simplicity of Apache Samza. * The forecasting system allows Uber to remove the notion of delay by using recent signals plus historical data to project what is happening now and what will happen into the future. * Uber’s pipeline for deploying ML models: HDFS, feature engineering, organizing into data structures (similar to data frames), deploy mostly offline training models, train models, & store into a container-based model manager. * A model serving layer is used to pick which model to use, forecasting results are stored in an OLAP data store, a validation layer compares real results against forecast results to verify the model is working as desired, and a rollback feature enables poor performing models to be automatically replaced by previous one. * “Without output, you don’t have input.” If you want to start leveraging machine learning, developers just need to start doing. Start with intuition and practice. Over time ask questions and learn what you need, then apply a laser focus to gain that knowledge. You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Check the landing page on InfoQ: https://bit.ly/2GJQbUo

Sander Mak on the Java Module System Mar 23, 2018

Show notes

Sander Mak and Wes Reisz discuss the Java module system and how adoption is going. Topics discussed on this podcast include Java modularity steps / migrations, green field projects, some of the concerns that caused the EC to initially vote no on Java 9, and a new tool for building custom JREs called JLink. Additionally, as Java 10 was recently released a short bit at the end was added to discuss some of the latest news with Java. Why listen to this podcast: • People quickly moved to Java 8 because of features like Streams and Lambdas. Java 9 has a different story around modularity and application architecture. Adoption is slower and more intentional. • Migrating large codebases to use modularity is hard. Many of the projects using modules are greenfield, and those large codebases that are moving now are most often using the classpath. • Jlink is a new command line tool released with Java 9. It allows developers to create their own lightweight, customized JRE for a module-based Java application. • Java version scheme has dropped the 1.* prefix. Future releases of the JDK will have the version number and follow the form *.0.1 (i.e. 9.0.1) • While the module system will likely show it’s benefit mostly for new development, many 3rd party libraries are moving to adopt modularity and removing their dependencies on JDK internal APIs. It’s improving the experience for teams adopting modularity. • There are no known open JEPS regarding the enhancement of the Java module system. • Java 10 has been released. The release features changes to the freely available Java versions, local variable type inference (var), experimental GRAAL JIT compiler, application class data sharing, improved container support/awareness, and others. More on this: Quick scan our curated show notes on InfoQ https://bit.ly/2DQ7ptx You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Check the landing page on InfoQ: https://bit.ly/2DQ7ptx

Jendrik Joerdening and Anthony Navarro on Self-Racing Cars Using Deep Neural Networks Mar 16, 2018

Show notes

Jendrik Joerdening and Anthony Navarro describe how a team of 18 Udacity students entered a self-racing car event They had very limited experience of building autonomous control systems for vehicles and had just 6 weeks to do it with only 2 days with the physical car. They describe the architecture, how they co-ordinated a very diverse team, and how they trained the models. Why listen to this podcast: - Last year a team of 18 Udacity Self-Driving Cars students competed at the 2017 Self Racing Cars event held at Thunderhill Raceway in California. - The students had all taken the first term of a three term program on Udacity which covers computer vision and deep learning techniques. - The team was extremely diverse. They co-ordinated the work via Slack with a team in 9 timezones and 5 different countries. - The team developed a neural network using Keras and Tensorflow which steered the car based on the input from just one front-facing camera in order to navigate all turns on the racetrack. - They received a physical car two days before the start of the event. More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2DykAiJ You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Check the landing page on InfoQ: http://bit.ly/2DykAiJ

Andrea Magnorsky on Paradigm Shifts and the Adoption of Programming Languages Mar 03, 2018

Show notes

On this podcast, we talk with Andrea Magnorsky, who is a tech lead at Goodlord on their engineering squads; she has a background in Scala, C#, and organised conferences. Today we’ll be talking about paradigm shifts. Why listen to this podcast: * A programming paradigm has a loose definition. It’s just about finding a way of doing things. * There are a number of different ways to think about problems - and different paradigms do this in different ways. * To shift paradigms, you have to un-learn some of your instincts. * When adopting a new paradigm if people don’t want to learn anything, then they won’t. * Multiple paradigms help you apply different ways of thinking about solutions to problems because solutions vary across languages. * Quick ways to start gaining knowledge and adoption for new languages are to use a new language as a test harness for your existing code. More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2oPFG71 You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Check the landing page on InfoQ: http://bit.ly/2oPFG71

Anne Currie on Organizational Tech Ethics, including Scale, GDPR, Algorithmic Transparency Feb 23, 2018

Show notes

On this podcast, Anne Currie joins the tech ethics discussion started on the Theo Schlossnagle podcast from a few weeks ago. Wes Reisz and Anne discuss issues such as the implications (and responsibilities) of the massive amount of scale we have at our fingertips today, potential effects of GDPR (EU privacy legislation), how accessibility is a an example of how we could approach tech ethics in software, and much more. Why listen to this podcast: - Ethics in software today is particularly important because of the scale we have available with cloud native architectures. - Accessibility offers a good approach to how we can evolve the discussion on tech ethics with aspects that include both a carrot and a stick. - Bitcoin mining power consumption is an example of something we never considered to have such negatives. - The key to establishing what we all should and shouldn’t be doing with tech ethics is to start conversations and share our lessons with each other. If you want to find out what every software developer, data scientists or ops should know about GDPR, download our free guide "Perspectives on GDPR": https://bit.ly/2FRvLnP More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2FtgdIy You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Check the landing page on InfoQ: http://bit.ly/2FtgdIy

Oliver Gould on Service Mesh for Microservices, LinkerD, and the Recently Released Conduit Feb 09, 2018

Show notes

This week on The InfoQ Podcast Wes Reisz talks with the CTO of Bouyant Oliver Gould. Bouyant is the maker the LinkerD Service Mesh and the recently released Conduit. In the podcast, Oliver defines a service mesh, clarifies the meaning of the data and control plane, discusses what a Service Mesh can offer a Microservice application owners, and, finally, discusses some of the considerations they took into account developing Conduit. Why listen to this podcast: - Service mesh is dedicated infrastructure that handles interservice communication. - There are two components to a service mesh: the data plane handles communication and the control plane is about policy and config. - LinkerD and Conduit are two open service meshes made by Bouyant. Conduit has a small memory footprint and provides a convention over configuration approach to service mesh deployment. - Adopting Rust (language used for implementing the data plane in Conduit) requires thinking of memory differently, and the best way to adopt Rust is to read other people’s code. More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2skWF61 You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Check the landing page on InfoQ: http://bit.ly/2skWF61

Theo Schlossnagle on Software Ethics and the Presence of Doing Good Feb 02, 2018

Show notes

This week's podcast features a chat with Theo Scholossnagle. Theo is the CEO of Circonus and co-chairs the ACM Queue. In this podcast, Theo and Wes Reisz chat about the need for ethical software, and how we as technical leaders should be reasoning about the software we create. Theo says, "it's not about the absence of evil, it's about the presence of good." He challenges us to develop rigor around ethical decisions we make in software just as we do for areas like security. With the incredible implications of machine learning and AI in our future, this week's podcast touches on topics we should all consider in the systems we create. Why listen to this podcast: - The ubiquitous society impact of computers is surfacing the need for deeper conversations on software ethics. - Ethics are a set of constructs and constraints to help us reason about right and wrong. - Algorithmic interpretability of models can be difficult to reason about; however, accountability for algorithms can be enforced in other ways. - Questions to be considered when writing software should evolve into: What am I building, why am I building it, and who will it hurt? - Ethics in software will take industry reform, deeper conversations, and developing a culture of questioning the software we’re building More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2BZAC4p You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Check the landing page on InfoQ: http://bit.ly/2BZAC4p

Chris Swan on DevOps and NoOps, plus Operations and Code Validation in a Serverless Environment Jan 19, 2018

Show notes

On this week’s podcast, Wes Reisz talks with Chris Swan. Chris is the CTO for the global delivery organisation at DXC Technology. Chris is well versed in DevOps, Infrastructure, Culture, and what it means to put all these together. Today’s topics include both DevOps and NoOps, and what Chris calls LessOps, what Operations means in a world of Serverless, where he sees Configuration Management, Provisioning, Monitoring and Logging heading. The podcast then wraps talking about where he sees validating code in a serverless deployment, such as canaries and blue-green deployments. Why listen to this podcast: * Serverless still requires ops - even if the ops aren’t focused on the technology * Even with minimal functions, the amount of configuration may exceed it by a factor of three * Disruptive services often move the decimal point * ML is the ability to make the inferences and AI is the ability to make decisions based on those inferences More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2Bff4jU You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Check the landing page on InfoQ: http://bit.ly/2Bff4jU

Architecting a Modern Financial Institution with Vitor Olivier, Thoughts on Immutability, CI/CD, FP Jan 12, 2018

Show notes

This week’s podcast features a chat with Vitor Olivier. Vitor is a partner at NuBank (a technology-centric bank in Brazil). This podcast hits on topics from several of Nubank’s recent QCon talks and includes things like: Nubank’s stack, functional programming, event sourcing, defining service boundaries, recommendations on reasoning about services, tips (or tweaks) on the second iteration of their initial architecture and more. Why listen to this podcast: - Property-based testing and Schemas (or Clojure.Spec)are complementary. - Clojure’s functional nature and Datomic’s features are a match for Nubank’s requirements. - A (micro)service needs to be able to create the full representation of the core feature it’s handling. - GraphQL is useful to abstract away the distributed system complexity from the mobile (or frontend) developers. - Nubank’s uses a combination of monitoring and sanity checks in real time at various level to keep systems consistent. - Once an invariant is broken, the system will try to fix it automatically. More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2mnqyfK You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Check the landing page on InfoQ: http://bit.ly/2mnqyfK

Charles Humble and Wes Reisz Take a Look Back at 2017 and Speculate on What 2018 Might Have in Store Dec 29, 2017

Show notes

In this podcast Charles Humble and Wes Reisz talk about Java 9 and beyond, Kotlin, .NET Core 2, the surge in interest in organisational culture, quantum computing and more. Why listen to this podcast: - Java had a big year with Java 9 shipping, Java EE going open-source and moving to Eclipse as EE4J, and IBM open-sprucing J9. From next year the platform will also be on a bi-annual release cycle with the next two versions (expected to be Java 10 and 11) both shipping during 2018. - Kotlin joined Scala, Clojure, and Groovy as a strong alternative language for the JVM particularly for mobile where it was buoyed by Google’s official blessing of it as a language for Android development at Google IO. - On InfoQ we also saw a big surge in interest around .NET linked to .NET Core 2, and at both InfoQ and at QCon San Fransisco we also saw an upsurge in interest around organizational culture with one of the culture tracks (the Whole Engineer) moving to one of the larger rooms. - We started to see Quantum computers emerging from the labs, with IBM making a 16 Qbit quantum processor available via their cloud for developers to play with, and the corresponding library available for Python on Github, - Another major trend from the year was the availability of machine learning libraries for software developers to build and train models Check the landing page on InfoQ: http://bit.ly/2ljlBVH Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq

Our TOPPODCAST Picks

Follow Us

Stay Connected

Related Podcasts

Links

Stay Connected