Getting Error while running pig script on google Dataproc Cluster, all the parameter are defined correctly

114 Views Asked by At

This is my pig script -

fs -cp -f gs://$codepath/db_password.sh file://$dataprochome/db_password.sh;
fs -cp -f gs://$codepath/jdbc_daily_load_tables.py file://$dataprochome/jdbc_daily_load_tables.py;
sh chmod +x $dataprochome/db_password.sh;
sh chmod +x $dataprochome/jdbc_daily_load_tables.py;
sh $dataprochome/db_password.sh $dataprochome $stg_gcs_bucket $se_stg_gcs_bucket $target_schema $target_table_stg_gm_add_attributes_orbit $target_table_orbit_delivery_partner_icc $kvenv;

All the input variable are defined properly still getting below error

2023-02-09 20:05:36,221 [main] ERROR org.apache.pig.Main - ERROR 2997: Encountered IOException. org.apache.pig.tools.parameters.ParseException: Encountered "<EOF>" at line 1, column 6.
Was expecting one of:
    IDENTIFIER
    OTHER
    LITERAL
    SHELLCMD
    
Details at logfile: /tmp/17114c5e-af3d-4a09-89c4-324250436a76/pig_1675973135671.log
2023-02-09 20:05:36,240 [main] INFO  org.apache.pig.Main - Pig script completed in 712 milliseconds (712 ms)
2

There are 2 best solutions below

1
OneCricketeer On

That's a bash script. Not pig ... If you're running that in pig, then that perfectly explains why it's failing to parse that file.

You need to use sh rather than pig on shell scripts

0
Sakshi Gatyan On

As per comments in the answer provided by @OneCricketeer, the error was resolved by removing the spaces between the arguments passed to .sh file in the pig script.