Batching vs. Streaming - Scale & Process Millions of Measurements a Second • Adam Butler • GOTO 2020

This presentation was recorded at GOTO Chicago 2020. #GOTOcon #GOTOchgo http://gotochgo.com Adam Butler - Engineering Lead in John Deere’s Intelligent Solution Group ORIGINAL TALK TITLE Batching vs. Streaming - John Deere's Journey to Scale & Process Millions of Measurements a Second ABSTRACT The John Deere data platform receives and processes millions of sensor measurements per second from machines around the world. In this talk, Adam will discuss the various technologies and architectures John Deere has implemented and how they had to adapt to ever increasing volumes of data. Adam will explore their usage of Lambda/Kinesis, Spark and Flink...where they worked, where they didn't, lessons learned and how their current architecture [...] TIMECODES 00:00 Intro 03:31 Streaming machine state data (2014) 05:39 Streaming agronomic data (2016) 10:10 Batch agronomic Data (2017) 12:21 Streaming agronomic data (2018) 16:27 Streaming agronomic data (2020) 18:47 Beyond 2020 Read the full abstract here: https://gotochgo.com/2020/sessions/1327/batching-vs-streaming-john-deeres-journey-to-scale-and-process-millions-of-measurements-a-second https://twitter.com/GOTOchgo https://www.linkedin.com/company/goto- https://www.facebook.com/GOTOConferences #Data #Streams #Backend #Batching #Streaming #Lambda #Kinesis #ApacheSpark #ApacheFlink Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at http://gotocon.com SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1