AWS Glue Data Catalog as the centralized metastore for Athena & PySpark from EMR

In Industry , Data Analyst vs. Data Engineer don't work , what works is Data Analyst and Data Engineer 🙂 They help each other in achieving a common goal -- "Improving Business " 😊 So while building an environment , the Cloud Architect have to make sure that the env. is efficient for both Data Engineer & Data Analyst.. And that's where AWS Glue helps a lot.. We can leverage the power of Glue to act as a central Metadata store.. Data Analyst / Marketing Analyst can use Athena to execute adhoc queries on DataLake data and Data Engineers can use the same catalog table to read the data from s3 & apply complex Transformation in the data from EMR 😊 This video is on how to setup the Glue Data Catalog as Spark SQL metastore & use the catalog tables in PySpark code from EMR. Documentation Link: ------------------------------------- https://docs.aws.amazon.com/emr/lates... Prerequisite: ---------------------- Using the AWS Glue Data Catalog as the metastore for Hive    • Using the AWS Glue Data Catalog as the met...   PySpark Code used in the code: ---------------------------------------------------- from pyspark.sql import SparkSession from pyspark.sql.functions import * spark = SparkSession\ .builder\ .appName("{App Name")\ .getOrCreate() spark.sql("show databases").show(); df = spark.sql("select * from {database_name}.{table_name}") df.show() df1=df.filter(df['id'].isNotNull()) df2=df1.groupBy("class_name").agg((sum("sepal_length").alias("sum_sepal"))) df2.write.option('path','s3://{}/{}/').saveAsTable('{database_name}.{table_name}') Check this playlist for more AWS Projects in Big Data domain:    • Demystifying Data Engineering with Cloud C...