[Bug]: Template doesn't compile out of the box

[rafadontal]

Related Template(s)

v2/googlecloud-to-googlecloud

Template Version

v2

What happened?

I am following the template documentation to compile and stage a dataflow template, I've got the same errors with all other templates, but in this case I was trying with v2/googlecloud-to-googlecloud. The documented code is this:

export PROJECT=<my-project>
export BUCKET_NAME=<bucket-name>

mvn clean package -PtemplatesStage  \
-DskipTests \
-DprojectId="$PROJECT" \
-DbucketName="$BUCKET_NAME" \
-DstagePrefix="templates" \
-DtemplateName="PubSub_to_BigQuery_Flex" \
-f v2/googlecloud-to-googlecloud

And the error:

[INFO] 59 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 28.096 s
[INFO] Finished at: 2025-04-28T11:19:19+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) on project googlecloud-to-googlecloud: Compilation failure: Compilation failure:
[ERROR] /C:/Bitbucket/DataflowTemplates/v2/googlecloud-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/templates/FirestoreToBigQuery.java:[74,35] cannot find symbol
[ERROR] symbol: variable XLANG
[ERROR] location: class com.google.cloud.teleport.metadata.Template.TemplateType
[ERROR] /C:/Bitbucket/DataflowTemplates/v2/googlecloud-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/templates/TextToBigQueryStreaming.java:[162,35] cannot find symbol
[ERROR] symbol: variable XLANG
[ERROR] location: class com.google.cloud.teleport.metadata.Template.TemplateType
[ERROR] /C:/Bitbucket/DataflowTemplates/v2/googlecloud-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/templates/TextIOToBigQuery.java:[140,35] cannot find symbol
[ERROR] symbol: variable XLANG
[ERROR] location: class com.google.cloud.teleport.metadata.Template.TemplateType
[ERROR] /C:/Bitbucket/DataflowTemplates/v2/googlecloud-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/templates/TextToBigQueryStreaming.java:[494,9] cannot find symbol
[ERROR] symbol: method parentName()
[ERROR] location: @interface com.google.cloud.teleport.metadata.TemplateParameter.Boolean
[ERROR] /C:/Bitbucket/DataflowTemplates/v2/googlecloud-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/templates/TextToBigQueryStreaming.java:[495,9] cannot find symbol
[ERROR] symbol: method parentTriggerValues()
[ERROR] location: @interface com.google.cloud.teleport.metadata.TemplateParameter.Boolean
[ERROR] /C:/Bitbucket/DataflowTemplates/v2/googlecloud-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/templates/TextToBigQueryStreaming.java:[501,9] cannot find symbol
[ERROR] symbol: method hiddenUi()
[ERROR] location: @interface com.google.cloud.teleport.metadata.TemplateParameter.Boolean
[ERROR] /C:/Bitbucket/DataflowTemplates/v2/googlecloud-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/templates/TextIOToBigQuery.java:[247,23] cannot find symbol
[ERROR] symbol: class JavascriptUdfFile
[ERROR] location: class com.google.cloud.teleport.metadata.TemplateParameter
[ERROR] /C:/Bitbucket/DataflowTemplates/v2/googlecloud-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/options/BigtableChangeStreamsToPubSubOptions.java:[135,7] cannot find symbol
[ERROR] symbol: method parentName()
[ERROR] location: @interface com.google.cloud.teleport.metadata.TemplateParameter.Boolean
[ERROR] /C:/Bitbucket/DataflowTemplates/v2/googlecloud-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/options/BigtableChangeStreamsToPubSubOptions.java:[136,7] cannot find symbol
[ERROR] symbol: method parentTriggerValues()
[ERROR] location: @interface com.google.cloud.teleport.metadata.TemplateParameter.Boolean
[ERROR] /C:/Bitbucket/DataflowTemplates/v2/googlecloud-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/options/BigtableChangeStreamsToPubSubOptions.java:[149,7] cannot find symbol
[ERROR] symbol: method parentName()

Then after a lot of research I've found that I had to edit the documented script a bit to skip some tests:

mvn clean package -PtemplatesStage ^
 -Dmaven.test.skip=true ^
 -DprojectId=%PROJECT% ^
 -DbucketName=%BUCKET_NAME% ^
 -DstagePrefix=templates ^
 -DgcpTempLocation=ods-dynamic-loader-test ^
 -DtemplateName=PubSub_to_BigQuery_Flex ^
 -Dcheckstyle.skip=true ^
 -f v2 ^
 -X

But then still got errors:

[INFO] gcs-to-sourcedb .................................... SUCCESS [01:32 min]
[INFO] googlecloud-to-elasticsearch ....................... SUCCESS [02:04 min]
[INFO] googlecloud-to-googlecloud ......................... FAILURE [ 01:44 h]
[INFO] Google Cloud To MongoDB ............................ SKIPPED
[INFO] googlecloud-to-neo4j ............................... SKIPPED
[INFO] googlecloud-to-splunk .............................. SKIPPED
[INFO] google-ads-to-googlecloud .......................... SKIPPED
[INFO] jdbc-and-googlecloud ............................... SKIPPED
[INFO] jms-to-pubsub ...................................... SKIPPED
[INFO] kafka-to-bigquery .................................. SKIPPED
[INFO] kafka-to-gcs ....................................... SKIPPED
[INFO] kafka-to-kafka ..................................... SKIPPED
[INFO] kafka-to-pubsub .................................... SKIPPED
[INFO] kinesis-to-pubsub .................................. SKIPPED
[INFO] kudu-to-bigquery ................................... SKIPPED
[INFO] MongoDB To Googlecloud ............................. SKIPPED
[INFO] mqtt-to-pubsub ..................................... SKIPPED
[INFO] mysql-to-googlecloud ............................... SKIPPED
[INFO] oracle-to-googlecloud .............................. SKIPPED
[INFO] postgresql-to-googlecloud .......................... SKIPPED
[INFO] pubsub-binary-to-bigquery .......................... SKIPPED
[INFO] pubsub-cdc-to-bigquery ............................. SKIPPED
[INFO] pubsub-to-jms ...................................... SKIPPED
[INFO] pubsub-to-kafka .................................... SKIPPED
[INFO] PubSub To MongoDB .................................. SKIPPED
[INFO] PubSub To Redis .................................... SKIPPED
[INFO] sourcedb-to-spanner ................................ SKIPPED
[INFO] spanner-change-streams-to-sharded-file-sink ........ SKIPPED
[INFO] failure-injection-policies ......................... SKIPPED
[INFO] failure-injected-spanner-service ................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:11 h
[INFO] Finished at: 2025-04-28T14:30:24+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.google.cloud.teleport:templates-maven-plugin:1.0-SNAPSHOT:stage (default) on project googlecloud-to-googlecloud: Template staging failed: Cannot run program "gcloud": CreateProcess error=2, The system cannot find the file specified -> [Help 1]

I was expecting the template to work straight out of the box. It seems to me that the documented process to build and stage a dataflow template is not accurate or up-to-date.

This is my environment:

$ mvn -v
Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937)
Maven home: C:\Program Files\apache-maven-3.9.9
Java version: 17.0.12, vendor: Oracle Corporation, runtime: C:\Program Files\Java\jdk-17
Default locale: en_GB, platform encoding: Cp1252
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

Relevant log output

[shubhamnarang11]

Hi @rafadontal, I was facing the same issue and I think it is due to dependent modules like common and others. You need to first clean install the whole project with this command - mvn clean install -DskipTests -Djib.skip=true and then you can package the template as per the README.

[liferoad]

Check https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/contributor-docs/code-contributions.md