Kafka-Confluent

• Confluent initially

  • Lots of data systems and data rich applications

  • Allows every developer to take action on what is happening in real-time

  • Data in Motion - category

• CIO of United States Postal Service (USPS)

  • 129 billion mail pieces over 164m places; 232k vehicles

  • 33,000 post offices

  • 115 PB of data

  • Started Kafka journey in 2016; 

  • Have their own data centers; 28,000 virtual and physical servers

• Q: COVID 19 Pandemic - Distributing Tests

  • CIO team responsible for setting up website for 

  • Synchronous - accepting the order, validating the address, making sure no duplicate order

  • Order placed on kafka topic - three downstream consumers and sent it to inventory systems; handled 8.7m orders per hour

  • Had to process orders super quickly

• Q: Other areas of Kafka use?

  • We need a digital representation of all of our assets; package scans, trucks, georeplication

  • Ingest 30 terabytes to create an application called Informed Visibility

    • Mesh of 400 Kafka topics

    • Powers 20 downstream business applications

    • Pricing, logistics, conditions of packages, conditions of post office

  • Geo-replication - we need to create caches for different regions; helps keep distributed caches in sync

  • We want to have the Kafka registry; enterprise topics so we can have everything cataloged; want to bring structure around data streaming

• Kafka History

  • 3 major releases every year; 22 major releases

  • Biggest features for growth

    • Replication 2016 - releasedreplication feature which helped for high availability for mission critical applications

    • Security 2018 - allowed customers and users to do authentication using an industry standard way; 

    • Connect 2019 - framework to allow community to build connectors for common data sources

    • kstreams; exactly once semantics 2019-20 - made streaming application building easier; allowed people to view data easier

    • Partition Scalability 2021 - allow for 10x scalability in metadata

  • Evolution over time

    • 200+ connectors - postgres, mongodb, 

    • Client libraries in a bunch of different languages including python, C#

    • Stream processing engines - beam, flink, hazelcast, spark

  • Open source project

    • 2200 contributors; 1500 reviewers

  • Users

    • 80% of fortune 100 are users

• Pinterest Chuyan Wang

  • 400m monthly active users

  • working at pinterest for 10 years

  • History - started using it back in 2012; needed unified and high throughtput log transformation platform

  • Right now - its powering hundreds of different batch and stream 

  • All the signals we develop for A/B experimentation

  • Advantage - ability to generate faster signals and recommend real time content to users

  • Data Architecture - use kafka for all data transportaiton - activity log or production CDC; batch and stream processing - push to S3 and use real time flink

  • 4,000 brokers; 3,000 topics; 50+ clusters; 500k+ partitions; 

  • Kafka can scale well both vertically and horizontally

  • Operating at that big scale - kafka is very stable - performance improvements and patches help a lot; instance hardware failure, traffic pattern change are biggest challenges; we used to be running kafka on top of magnetic disks - now we use NVme SSD, I3s or I4s - IOPS has a lot to work with high partition density environement

  • We choose to invest heavily around automation; we have a project called Array that is helps automate some things - burdens for automated broker replacement, worker upgrade, restart, workload balancing

  • We've built several tools at Pinterest and have open sourced them as well - two other projects - Singer and Secor - Singer is a high throughput logging agent - moving it from idsk ; secor kafka consumer that persists data to S3 with stronger consistency

  • Querybook - Pinterest big data IDE with a notebook style interface

  • Kafka in the future - data mesh - data discoverability and data proliferation - internal data by building a data catalog; data documentation; table of tiering to improve batch and analytical processes

• Kafka in the Cloud

  • Managing a distributed system is not easy; balancing clusters, upgrades, etc. 

  • Typically can translate into lower costs

  • Running Kafka in the cloud Brokers have local attached storage; low-latency real time experience - we typically need to use 

    • Overall storage cost could be dominating

    • We are sort of coupling bulk of data with the storage

  • Confluent cloud - 

    • Users care about low latency real time experience for most recent data

    • For historical data only care about good throughput but not low latency

    • Local hotset to allow for fast access to recent data; 

    • Remote access in external block storage; good throughput at lower cost

    • Decoupled bulk of storage with computation