I have followed the steps outlined to install Developing using the AWS Glue ETL library - Python on Windows found here:
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html
After following the installation instructions, it's unclear how to actually execute a spark job successfully locally
In powershell I have:
a simple pyspark job called my_script.py:
from pyspark.context import SparkContext
from pyspark.sql import SparkSession
sc = SparkContext()
spark = SparkSession(sc)
I have:
- navigated to the aws-glue-libs directory
- in PowerShell attempted
.\bin\gluesparksubmit F:\programming\my_script.py
- the output seems to be nothing
Can you please provide correct example on how to execute a aws glue job locally?
The ultimate goal here is to develop my glue jobs locally in Pycharm before deploying to the AWS Glue Service.