AWS re:Invent 2017

Latest Episodes:

ABD316: American Heart Association: Finding Cures to Heart Disease Through the Power of Technology Dec 01, 2017

Show notes

Combining disparate datasets and making them accessible to data scientists and researchers is a prevalent challenge for many organizations, not just in healthcare research. American Heart Association (AHA) has built a data science platform using Amazon EMR, Amazon Elasticsearch Service, and other AWS services, that corrals multiple datasets and enables advanced research on phenotype and genotype datasets, aimed at curing heart diseases. In this session, we present how AHA built this platform and the key challenges they addressed with the solution. We also provide a demo of the platform, and leave you with suggestions and next steps so you can build similar solutions for your use cases.

ABD318: Architecting a data lake with Amazon S3, Amazon Kinesis, and Amazon Athena Dec 01, 2017

Show notes

Learn how to architect a data lake where different teams within your organization can publish and consume data in a self-service manner. As organizations aim to become more data-driven, data engineering teams have to build architectures that can cater to the needs of diverse users - from developers, to business analysts, to data scientists. Each of these user groups employs different tools, have different data needs and access data in different ways. In this talk, we will dive deep into assembling a data lake using Amazon S3, Amazon Kinesis, Amazon Athena, Amazon EMR, and AWS Glue. The session will feature Mohit Rao, Architect and Integration lead at Atlassian, the maker of products such as JIRA, Confluence, and Stride. First, we will look at a couple of common architectures for building a data lake. Then we will show how Atlassian built a self-service data lake, where any team within the company can publish a dataset to be consumed by a broad set of users.

ABD319: Tooling Up for Efficiency: DIY Solutions @ Netflix Dec 01, 2017

Show notes

At Netflix, we have traditionally approached cloud efficiency from a human standpoint, whether it be in-person meetings with the largest service teams or manually flipping reservations. Over time, we realized that these manual processes are not scalable as the business continues to grow. Therefore, in the past year, we have focused on building out tools that allow us to make more insightful, data-driven decisions around capacity and efficiency. In this session, we discuss the DIY applications, dashboards, and processes we built to help with capacity and efficiency. We start at the ten thousand foot view to understand the unique business and cloud problems that drove us to create these products, and discuss implementation details, including the challenges encountered along the way. Tools discussed include Picsou, the successor to our AWS billing file cost analyzer; Libra, an easy-to-use reservation conversion application; and cost and efficiency dashboards that relay useful financial context to 50+ engineering teams and managers.

ABD320: Netflix Keystone SPaaS: Real-time Stream Processing as a Service Dec 01, 2017

Show notes

Over 100 million subscribers from over 190 countries enjoy the Netflix service. This leads to over a trillion events, amounting to 3 PB, flowing through the Keystone infrastructure to help improve customer experience and glean business insights. The self-serve Keystone stream processing service processes these messages in near real-time with at-least once semantics in the cloud. This enables the users to focus on extracting insights, and not worry about building out scalable infrastructure. In this session, I share the benefits and our experience building the platform.

ABD327: Migrating Your Traditional Data Warehouse to a Modern Data Lake Dec 01, 2017

Show notes

In this session, we discuss the latest features of Amazon Redshift and Redshift Spectrum, and take a deep dive into its architecture and inner workings. We share many of the recent availability, performance, and management enhancements and how they improve your end user experience. You also hear from 21st Century Fox, who presents a case study of their fast migration from an on-premises data warehouse to Amazon Redshift. Learn how they are expanding their data warehouse to a data lake that encompasses multiple data sources and data formats. This architecture helps them tie together siloed business units and get actionable 360-degree insights across their consumer base.

ABD329: A Look Under the Hood – How Amazon.com Uses AWS Services for Analytics at Massive Scale Dec 01, 2017

Show notes

Amazon's consumer business continues to grow, and so does the volume of data and the number and complexity of the analytics done in support of the business. In this session, we talk about how Amazon.com uses AWS technologies to build a scalable environment for data and analytics. We look at how Amazon is evolving the world of data warehousing with a combination of a data lake and parallel, scalable compute engines such as Amazon EMR and Amazon Redshift.

ABD330: Combining Batch and Stream Processing to Get the Best of Both Worlds Dec 01, 2017

Show notes

Today, many architects and developers are looking to build solutions that integrate batch and real-time data processing, and deliver the best of both approaches. Lambda architecture (not to be confused with the AWS Lambda service) is a design pattern that leverages both batch and real-time processing within a single solution to meet the latency, accuracy, and throughput requirements of big data use cases. Come join us for a discussion on how to implement Lambda architecture (batch, speed, and serving layers) and best practices for data processing, loading, and performance tuning.

ABD331: Log Analytics at Expedia Using Amazon Elasticsearch Service Dec 01, 2017

Show notes

Expedia uses Amazon Elasticsearch Service (Amazon ES) for a variety of mission-critical use cases, ranging from log aggregation to application monitoring and pricing optimization. In this session, the Expedia team reviews how they use Amazon ES and Kibana to analyze and visualize Docker startup logs, AWS CloudTrail data, and application metrics. They share best practices for architecting a scalable, secure log analytics solution using Amazon ES, so you can add new data sources almost effortlessly and get insights quickly.

ABD335: Real-Time Anomaly Detection Using Amazon Kinesis Dec 01, 2017

Show notes

Amazon Kinesis Analytics offers a built-in machine learning algorithm that you can use to easily detect anomalies in your VPC network traffic and improve security monitoring. Join us for an interactive discussion on how to stream your VPC flow Logs to Amazon Kinesis Streams and identify anomalies using Kinesis Analytics.

ABD337: Making the Shift from DevOps to Practical DevSecOps Dec 01, 2017

Show notes

Agility is the cornerstone of the DevOps movement. Developers are working to continuously integrate and deploy (CI/CD) code to the cloud, to ensure applications are seamlessly updated and current. But what about secure? Security best practices and compliance are now the responsibility of everyone in the development lifecycle, and continuous security is a critical component of the ongoing deployment process. Discover how to incorporate security best practices into your current DevOps operations, gain visibility into compliance posture, and identify potential risks and threats in your AWS environment. We demonstrate how to leverage the CIS AWS Foundation Benchmarks within Sumo to trigger alerts from your AWS CloudTrail and Amazon CloudWatch log when risks or violations occur, such as unauthorized API calls, IAM policy changes, AWS Config configuration changes, and many more. Session sponsored by Sumo Logic

Our TOPPODCAST Picks

Follow Us

Stay Connected

Related Podcasts

Links

Stay Connected

Our TOPPODCAST Picks

Follow Us

Stay Connected

AWS re:Invent 2017

Related Podcasts

1

2

3

4

5

6