Видео

Wade here. Thankfully, scale is something that Flink does very well. You can scale out your jobs, use tools to shuffle and rebalance data across a cluster, etc. Having said that, as you scale, you do have to be more careful. The state needs to be stored somewhere by Flink so you will need to ensure you aren't running out of resources. You can often alleviate this with careful distribution across a cluster. In general, try to avoid storing things that grow in an unbounded fashion. When you do have something that is going to grow, make sure you have a way to partition and scale it across the cluster.

Adam here. When you create the CDC topic and snapshot the table, set retention to infinite and set compaction to true. Your topic will be an eventually-consistent replica of your database table. Whenever you CUD a row in the DB, the latest full set of data will be propagated to the topic (think Debezium, with it's before/after fields). Then as a consumer you can simply materialize the whole topic if you want the whole set of data, or just select the fields your app cares about and discard the rest. Basically, you want to do whateve you can in your power to alleviate your consumers from having to "reconstruct" the data on their own, merging snapshots and deltas leads to a lot of complexity and work, replicated for every single consumer that wants the data. The tradeoff is more data over the wire with conventional message brokering - but the benefit is that it's a much simpler architecture. Storage is super cheap, especially if you're using Cloud storage as a backer for your Kafka topics. Note that this design is becoming even simpler with the adoption of Kafka replication without network. Eg; Confluent Freight topics use Amazon S3 as both storage AND replication, so that you don't pay cross-AZ fees anymore. Check it out here if you want to know more: www.confluent.io/blog/introducing-confluent-cloud-freight-clusters/

Wade here. CDC is entirely optional. If you already have a CDC system in place, it can assist you in this process. But, if you don't have CDC in place, and don't want to introduce it, you can write some code in your microservice to do this. Unfortunately, this isn't a problem you can solve in Kafka. That's in part because it's not a Kafka problem. It's a distributed systems problem that can exist in any system. You can encounter the exact same problem when saving to a file and sending an email. But the other issue is that in this specific instance, Kafka can't solve the problem because Kafka doesn't know about it. I.E. The situation outlined is what happens when Kafka doesn't get the message? How could Kafka solve the problem when it doesn't even know the message exists? To understand the problem in more depth, check out www.confluent.io/blog/dual-write-problem/

@@ConfluentDevXTeam I got your point. Thanks for the reply, Wade.

Wade here. Thanks. I'm glad you enjoyed it. On the visual side, I've started working a bit more with animations to try to assist in focusing the eyes where they need to be. It sounds like that has been effective.

Wade here. There's an implicit assumption in the video about order. I am assuming that your order of operations is 1. Write to the Database, 2. Write to Kafka. With that assumption in mind, if the write to the database fails, then presumably the operation aborts, and the write to Kafka never happens. This scenario is fine and we don't have to do anything. Now, if you reverse the order of operations and do the write to Kafka first, then we are back to the same problem and have to investigate the same solutions. You can find a deeper discussion on this here: developer.confluent.io/courses/microservices/the-dual-write-problem/. There's also a separate edge case that I haven't discussed. What happens if the write to the database times out? In that case, it's actually possible that the write succeeded, but the code will view it as a failure and the write to Kafka won't proceed, again resulting in an inconsistency. To be safe, it's best to deal with the dual-write problem upfront so we can avoid these edge cases.

Wade here. Thanks, glad you enjoyed the content. Though, in this case, although the Outbox will work, I feel that Event Sourcing might be a better solution.

Wade here. Thank you. Glad you enjoyed it.

Hi Wade. Thanks a lot for your videos. I think I have one thing that you probably could address. I read somewhere that, if you use CDC (i.e. Debezium), there is no need to store the events in the outbox table at all. It is sufficient to save and immediately delete them, because CDC only sees the changelog of the db, reacts to the insert and ignores the delete. Right?

No they flipped

Adam here. It seems you're missing some key elements: 1) A traditional RDBMS bundles processing with storage.In headless, the storage is completely separate from the processing and query. I do not know of any RDBMS systems that let you access their underlying storage (Eg: a B-tree) directly, but if there is, I would be keen to find out. 2) You don't have the risk of one query saturating your data layer and stalling other queries, like you do in an RDBMS. However, this relies on the fact that most data layers will be served via massive cloud compute storage (R2, GCS, Azure Blob, S3, etc), and not you running your own data layer on an in-house HDFS. 3) Client-server embeds business logic inside the server for the client to call. There is a tight coupling between the two. In HDA, there are no smarts in the data layer. It's just data that has been created by the upstream services. IF your server only provided raw data via a GET, and writes via a PUT/POST, but had absolutely no other functionality whatsoever, then you could equate it to a headless model. That's pretty much what Iceberg and Kafka do, with a bit of cleanup optimizations sprinkled in.

Wade here. Depends on the final implementation details of the system. They certainly could have an initial layer of basic checks that run synchronously (see other comments). This would be looking for obvious things that can be detected on a single transaction. But, the reality is that most fraud doesn't show up on a single transaction. A lot of fraud detection happens over multiple transactions. As a result, the nature of the problem dictates that they have to let through some fraudulent transactions because they can't detect it until they have seen a pattern develop. And yes, if the $$ amount is high, that could be a problem. That is why many banks have transaction limits. They have decided what level of risk they are willing to take and set the limit accordingly. There's a balance to be struck. The bank has to decide where they want to strike that balance. Would they rather have valid transactions fail (or timeout) or potentially let through more fraud? Both situations are costly. But which one costs more?

Wade here. You probably want to take some time and watch the rest of the video series here: developer.confluent.io/courses/microservices In particular, you might find the most recent video helpful (and the comments below it): developer.confluent.io/courses/microservices/case-study-asynchronous-events/ But the gist of it is that Fraud Detection takes time. You generally can't look at a single transaction and say "oh, that's fraud." You have to look at patterns of transactions over time. Eventually, you develop enough information that you can start to say "Ah, something weird is happening here" and at that point, you lock the account and prevent future transactions.

Wade here. The thing is, it may not be as simple as detect fraud or don't detect fraud. It's a balance. Yes, you obviously want to detect fraud as accurately as possible. But your bank also needs to be able to give people money when they ask for it. Financial systems, payment processors, etc, have very strict limits on how long they can take. If running fraud detection exceeds those limits, all transactions will be rejected, whether they are fraudulent or not. If that happens, the fraud detection is going to be largely irrelevant because the bank won't have any customers left. Having said that, in a real system, they can definitely have layers of fraud detection. Do the fast stuff in real-time, and leave the slower stuff for an async process.

@@WadeWaldron-Confluent alright. now i know a bit more about banking system

Adam here. I covered that here: ruclips.net/video/dahs6jpCYQs/видео.html Headless is the full decoupling of data access from processing _of all forms_, providing reusable data assets for anywhere in the company - not just for analytics use cases. To emphasize - a data lake, or lakehouse, or lake-like warehouse, are all analytics constructs first and foremost. They're also predicated on the notion that you must copy all the data into the data lake's bronze layer "as-is". From there, you add schemas, formatting, and structure, leading to a silver layer (another copy). Then you can start doing work on it. The problems: 1) You don't own the data. The source breaks, your pipeline breaks, and then you're forced to react to it, determine impact / contamination, reprocess data sets, etc. (Did this for 7-8 years myself, no thanks). 2) It's stuck in your data lake. All that work you did to convert it Source->Bronze->Silver is only usable if you use the data lake. _Historically_, leading data lake providers have been happy to provide you with the best performance ONLY if you use their query engines. Using an external engine (if even compatible) would lead to far worse performance. Data lakehouse/warehouse/househouse providers were more than happy to lock you in on the data front, because they made big $$$ on it. But happily, this is starting to change due to the adoption of open-source formats that you can run yourself in house - you can see it with the growing adoption of Iceberg (Databricks bought Tabular, Iceberg cocreators - Snowflake is investing heavily in Iceberg, open sourcing their Polaris catalog). Data lake providers _could_ decide NOT to adopt these open formats, but then they risk losing their business to those who have - so the result is that most players are letting go of their control over storage so that they can adopt Iceberg/Delta/Hudi compatible workloads that they may not have had access to before. If you want a quick mental shortcut for how this is different - a headless data architecture lets you plug your data into your data lake _at the silver layer_. The data is well-formed, controlled, schematized, and has designated ownership. But you can also plug that same data into a DuckDB instance, or you can plug it into BigQuery, or Flink, or any other Iceberg-compatible / Kafka compatible consumer endpoint. The idea is that you've decoupled the creation of the data plane from "analytics-only" concerns, and instead focused on building modular reusable building blocks that can power any number of data lakes, warehouses, or swamps, in addition to operational systems.

Adam here. I'm not advocating removing bounded contexts, putting all the data into a big mixed pile, and tightly coupling all your systems together. A headless data architecture promotes the data that should be shared into an accessible format - in this version, the stream or the table. If you already have EDA with well-formed and schematized streams you're halfway there. The table component is an extension, where you take business data circulating in your streams and materializing it into an Iceberg table - but note that we didn't shove it into some data lake somewhere for just the lake to use. It remains outside the lake, and is _pluggable_ into whatever lake, warehouse, operational system, SaaS app, or client that needs it. This pluggability forgoes copying, so that you don't need to build pipelines and copy data. The gist is that you focus on building a data layer that promotes access and reuse - something that comes for free with a Kafka-based EDA, but that has historically struggled for tables due to the general approach of dumping it all in a data lake to sort it out later.

Well, the team building your architecture could abstract it below the public API. If you query data from BiqQuery, make the system do all processing on GCP and so on. However, if you're trying to join/aggregate data from different clouds, then yeah, I guess you're out of luck. Or you could make a query engine that is architecture aware and takes into account where the data is, the potential egress/ingress, etc as cost for the query planner and then try to push down as many operations as possible, so that you only send through the internet the most compact and already processed data, instead of the entire projection.

Adam here. Inter-cloud costs remain a factor, but typically I wouldn't expect to see a single query federated across clouds WITHOUT taking into consideration data locality. For example, issue separate queries for each cloud, aggregate locally, then bring those results to a single cloud for final aggregation (basically a multi-cloud Map-Reduce - thanks Hadoop!). Speed also remains a factor, as you pointed out, due to CAP theorem. There is no free lunch, so if you're going with a global, multi-cloud, distributed data layer, then yeah, you should probably invest in some tooling to prevent your users from shooting themselves in the foot with a $50k per query bill.

That's the common quote, yes. However, in this case, Danica is correctly quoting from J.R.R. Tolkien. www.goodreads.com/quotes/229-all-that-is-gold-does-not-glitter-not-all-those

Hi, Gilles from Confluent here. You can add business domain metadata via the UI or the API, or you can use our Terraform provider. Check out my video to see how to do that: ruclips.net/video/Vg7k_3vlC3Q/видео.html.

Wade here. You are welcome. Glad you enjoyed it.

Wade here. Yeah, that's more or less my thinking as well. Simple rules that can be executed fast can be used for a quick sanity check on the transaction. Async events for a more detailed analysis. The key is to recognize that sometimes things need to be done synchronously, but that doesn't mean everything needs to be synchronous.

Wade here. Absolutely. Within our industry, there is a tendency to assign strong consistency to processes that aren't actually strongly consistent and don't really require it. That is a hurdle to overcome. And it's not to suggest that you can always overcome it. Sometimes, the business wants strong consistency, and you can't do anything to change their mind. However, we should avoid assuming that everything must be strongly consistent. Start with async events. Fallback to other techniques if required. And make sure the business understands the consequences. If making the operation strongly consistent results in a 10x increase in latency (as an example), is that really what the business (or customer) wants?

​@@ConfluentDevXTeam Totally agree

Wade here. Thanks. Glad you enjoyed it.

Wade here. I'm glad you got value out of it. I'm curious what aspect you found so important that you feel other orgs need to hear it.

Ephemeral means that there is no indefinite persistence of the events. It can vary from "all events are only in memory" (and therefore vulnerable to deletion upon power loss) to "events are persisted to disk, but are deleted after a short period of time "(seconds, minutes, or perhaps after consumption). For example, a consumer may only receive the messages if it is online at the time the producer writes it to the broker. A consumer coming online later is unable to access the historic events because they are no longer stored anywhere.