• Optimizing SparkSQL Writes to Hive Dynamic Partitions Liao Jiayi Liao Jiayi #Spark#Apache Spark

    Hive tables are often partitioned by time and populated through SparkSQL dynamic-partition inserts. This reveals an opportunity for optimization.
  • Spark - Cleaning Up Objects with WeakReference Liao Jiayi Liao Jiayi #JVM#Spark#Apache Spark

    A Stack Overflow question about when Spark reclaims Accumulator and Broadcast variables led me to explore how ContextCleaner uses WeakReference and a reference queue to clean up unused objects.