https://luoyuxia.github.io/tags/%E6%95%B0%E6%8D%AE%E6%B9%96/ Recent content in 数据湖 on Yuxia's Blog Hugo zh Tue, 14 Apr 2026 21:44:51 +0800 https://luoyuxia.github.io/posts/%E6%B5%85%E6%B5%85%E5%AD%A6%E4%B9%A0%E4%B8%80%E4%B8%8B-mooncake---%E5%A6%82%E4%BD%95%E8%AE%A9-postgres-%E7%9A%84%E6%AF%8F%E4%B8%80%E6%AC%A1%E5%86%99%E5%85%A5iceberg-%E9%83%BD%E8%83%BD%E5%AE%9E%E6%97%B6%E7%9C%8B%E8%A7%81/ Tue, 14 Apr 2026 21:44:51 +0800 https://luoyuxia.github.io/posts/%E6%B5%85%E6%B5%85%E5%AD%A6%E4%B9%A0%E4%B8%80%E4%B8%8B-mooncake---%E5%A6%82%E4%BD%95%E8%AE%A9-postgres-%E7%9A%84%E6%AF%8F%E4%B8%80%E6%AC%A1%E5%86%99%E5%85%A5iceberg-%E9%83%BD%E8%83%BD%E5%AE%9E%E6%97%B6%E7%9C%8B%E8%A7%81/ Mooncake通过GlobalIndex实时生成DeletionVector替代低效EqualityDelete，并结合UnionRead将内存Arrow批次、磁盘Parquet与多级删除信息统一查询，实现Postgres到Iceberg的毫秒级实时同步与分析。 https://luoyuxia.github.io/posts/%E6%B5%85%E6%B5%85%E8%81%8A%E4%B8%80%E8%81%8A%E5%9B%9B%E5%A4%A7%E6%B9%96%E6%A0%BC%E5%BC%8F%E7%9A%84%E5%86%85%E9%83%A8%E6%9C%BA%E5%88%B6%E5%92%8C%E4%B8%80%E8%87%B4%E6%80%A7%E6%A8%A1%E5%9E%8B---hudi-%E7%AF%87/ Sat, 13 Sep 2025 20:07:52 +0800 https://luoyuxia.github.io/posts/%E6%B5%85%E6%B5%85%E8%81%8A%E4%B8%80%E8%81%8A%E5%9B%9B%E5%A4%A7%E6%B9%96%E6%A0%BC%E5%BC%8F%E7%9A%84%E5%86%85%E9%83%A8%E6%9C%BA%E5%88%B6%E5%92%8C%E4%B8%80%E8%87%B4%E6%80%A7%E6%A8%A1%E5%9E%8B---hudi-%E7%AF%87/ 文章深入解析了Hudi的内部机制与一致性模型，重点阐述其基于Timeline和FileGroup的读写流程、乐观并发控制及对写入端的时间戳单调性等严格要求，揭示了其复杂性与潜在数据一致性风险。 https://luoyuxia.github.io/posts/%E6%B5%85%E6%B5%85%E8%81%8A%E4%B8%80%E8%81%8A%E5%9B%9B%E5%A4%A7%E6%B9%96%E6%A0%BC%E5%BC%8F%E7%9A%84%E5%86%85%E9%83%A8%E6%9C%BA%E5%88%B6%E5%92%8C%E4%B8%80%E8%87%B4%E6%80%A7%E6%A8%A1%E5%9E%8B---delta-%E7%AF%87/ Sat, 13 Sep 2025 13:16:11 +0800 https://luoyuxia.github.io/posts/%E6%B5%85%E6%B5%85%E8%81%8A%E4%B8%80%E8%81%8A%E5%9B%9B%E5%A4%A7%E6%B9%96%E6%A0%BC%E5%BC%8F%E7%9A%84%E5%86%85%E9%83%A8%E6%9C%BA%E5%88%B6%E5%92%8C%E4%B8%80%E8%87%B4%E6%80%A7%E6%A8%A1%E5%9E%8B---delta-%E7%AF%87/ Delta通过递增版本的DeltaLog记录写入，采用Copy-on-write或Merge-on-read实现数据更新，并利用PutIfAbsent或表锁解决并发写入冲突，其一致性模型基于分区级冲突检测。 https://luoyuxia.github.io/posts/%E6%B5%85%E6%B5%85%E8%81%8A%E4%B8%80%E8%81%8A%E5%9B%9B%E5%A4%A7%E6%B9%96%E6%A0%BC%E5%BC%8F%E7%9A%84%E5%86%85%E9%83%A8%E6%9C%BA%E5%88%B6%E5%92%8C%E4%B8%80%E8%87%B4%E6%80%A7%E6%A8%A1%E5%9E%8B---paimon-%E7%AF%87/ Sun, 07 Sep 2025 11:10:19 +0800 https://luoyuxia.github.io/posts/%E6%B5%85%E6%B5%85%E8%81%8A%E4%B8%80%E8%81%8A%E5%9B%9B%E5%A4%A7%E6%B9%96%E6%A0%BC%E5%BC%8F%E7%9A%84%E5%86%85%E9%83%A8%E6%9C%BA%E5%88%B6%E5%92%8C%E4%B8%80%E8%87%B4%E6%80%A7%E6%A8%A1%E5%9E%8B---paimon-%E7%AF%87/ Paimon通过LSM树和Deletionvector优化主键表读写，多写者不同bucket无一致性问题，但同bucket写入可能导致更新丢失或悬空Deletionvector。 https://luoyuxia.github.io/posts/%E6%B5%85%E6%B5%85%E8%81%8A%E4%B8%80%E8%81%8A%E5%9B%9B%E5%A4%A7%E6%B9%96%E6%A0%BC%E5%BC%8F%E7%9A%84%E5%86%85%E9%83%A8%E6%9C%BA%E5%88%B6%E5%92%8C%E4%B8%80%E8%87%B4%E6%80%A7%E6%A8%A1%E5%9E%8B---iceberg-%E7%AF%87/ Sat, 06 Sep 2025 18:49:52 +0800 https://luoyuxia.github.io/posts/%E6%B5%85%E6%B5%85%E8%81%8A%E4%B8%80%E8%81%8A%E5%9B%9B%E5%A4%A7%E6%B9%96%E6%A0%BC%E5%BC%8F%E7%9A%84%E5%86%85%E9%83%A8%E6%9C%BA%E5%88%B6%E5%92%8C%E4%B8%80%E8%87%B4%E6%80%A7%E6%A8%A1%E5%9E%8B---iceberg-%E7%AF%87/ 本文深入解析Iceberg数据湖格式的内部机制与一致性模型，涵盖写入流程、快照管理、并发控制及冲突检测机制，确保多写者场景下的数据一致性。