python 从mysql读取数据后,如何转为spark rdd?

作者:高景洋 日期:2021/01/29 18:27 浏览次数:103

直接上代码:


from pyspark import SparkContext,SparkConf

conf = SparkConf()
sc = SparkContext(conf=conf)


list_url_group_data = ListUrlDA().select_list_url_count_group_by_websiteid(list_schedule_website_id) #从mysql读出来的数据 类型 List
list_url_rdd = sc.parallelize(list_url_group_data) # 将List转换为rdd

spark = SparkSession.builder.master("local").appName("SparkMysql").getOrCreate()
schema = StructType([
# true代表不为空
StructField("WebsiteID", StringType(), True),
StructField("count", StringType(), True)
])
df_list_url = spark.createDataFrame(list_url_rdd, schema=schema) # 将rdd转换为dataFrame

df_list_url.show()
calculate_product_count(list_filter_websiteids,df_list_url) # 业务逻辑处理方法
spark.stop()


执行结果:

29-01-2021 17:47:34 CST hbase_scan INFO - +---------+------+
29-01-2021 17:47:34 CST hbase_scan INFO - |WebsiteID| count|
29-01-2021 17:47:34 CST hbase_scan INFO - +---------+------+
29-01-2021 17:47:34 CST hbase_scan INFO - |        1|215923|
29-01-2021 17:47:34 CST hbase_scan INFO - |       17|  7843|
29-01-2021 17:47:34 CST hbase_scan INFO - |       23|  1563|
29-01-2021 17:47:34 CST hbase_scan INFO - |       24|  1720|
29-01-2021 17:47:34 CST hbase_scan INFO - |       71|334890|
29-01-2021 17:47:34 CST hbase_scan INFO - |       94|  6782|
29-01-2021 17:47:34 CST hbase_scan INFO - |      103|     9|
29-01-2021 17:47:34 CST hbase_scan INFO - |      108|   319|
29-01-2021 17:47:34 CST hbase_scan INFO - |      167|  8352|
29-01-2021 17:47:34 CST hbase_scan INFO - |      168|  5417|
29-01-2021 17:47:34 CST hbase_scan INFO - |      171|  8598|
29-01-2021 17:47:34 CST hbase_scan INFO - |      221|    43|
29-01-2021 17:47:34 CST hbase_scan INFO - |      237|  1128|
29-01-2021 17:47:34 CST hbase_scan INFO - |      238|    11|
29-01-2021 17:47:34 CST hbase_scan INFO - |      242|   111|
29-01-2021 17:47:34 CST hbase_scan INFO - |      243|   922|
29-01-2021 17:47:34 CST hbase_scan INFO - |      251|   445|
29-01-2021 17:47:34 CST hbase_scan INFO - |      253|   372|
29-01-2021 17:47:34 CST hbase_scan INFO - |      279|  1739|
29-01-2021 17:47:34 CST hbase_scan INFO - |      282|    59|
29-01-2021 17:47:34 CST hbase_scan INFO - +---------+------+
29-01-2021 17:47:34 CST hbase_scan INFO - only showing top 20 rows

本文永久性链接:
<a href="http://www.zhihu.so/art173.aspx">python 从mysql读取数据后,如何转为spark rdd?</a>
相关文章
友情链接
支付宝打赏(内容对您有帮助的话)
微信交流
知乎编程经验网 - 在工作中总结编程的经验! 备案/许可证编号:鲁ICP备11020152号
QQ:120217215 联系电话:15192695151