kura - remote installation of deployment package via Mqtt fails

Question

I am having trouble installing a deployment package remotely, following are the metrics I send on the request topic - $EDC/amir-kura/test-client/DEPLOY-V2/EXEC/download

job.id=12345678910L
dp.uri=file:/home/amir/Downloads/org.eclipse.kura.example.hello_osgi/resources/dp/hello_osgi.dp
dp.name=hello_osgi
dp.version=1.0.0
dp.download.protocol=HTTPS
dp.install.system.update=false

The package gets downloaded successfully as I get the following reply on the Notify topic-$EDC/amir-kura/new-client/DEPLOY-V2/NOTIFY/test-client/download

{dp.download.size=0, 
dp.download.progress=100, 
dp.download.status=COMPLETED, 
job.id=12345678910, 
client.id=test-client}

But, the installation fails. Following is my kura console-

15:06:07,357 [MQTT Call: test-client] INFO  CloudServiceImpl:440  - Message arrived on topic: $EDC/amir-kura/test-client/DEPLOY-V2/EXEC/download
15:06:07,382 [pool-2-thread-2] INFO  CloudDeploymentHandlerV2:459  - About to download and install package at URL file:/home/amir/Downloads/org.eclipse.kura.example.hello_osgi/resources/dp/hello_osgi.dp
15:06:07,382 [pool-2-thread-2] INFO  CloudDeploymentHandlerV2:468  - Downloading package from URL: file:/home/amir/Downloads/org.eclipse.kura.example.hello_osgi/resources/dp/hello_osgi.dp
15:06:07,387 [pool-12-thread-1] INFO  DataServiceImpl:441  - Storing message on topic :$EDC/#account-name/new-client/DEPLOY-V2/NOTIFY/test-client/download, priority: 1
15:06:07,389 [pool-12-thread-1] INFO  DataServiceImpl:444  - Stored message on topic :$EDC/#account-name/new-client/DEPLOY-V2/NOTIFY/test-client/download, priority: 1
15:06:07,402 [DataServiceImpl:Submit] INFO  MqttDataTransport:512  - Publishing message on topic: $EDC/amir-kura/new-client/DEPLOY-V2/NOTIFY/test-client/download with QoS: 1
15:06:07,403 [pool-12-thread-1] INFO  DownloadImpl:131  - Ready to install
15:06:07,405 [pool-2-thread-2] INFO  DataServiceImpl:441  - Storing message on topic :$EDC/#account-name/new-client/DEPLOY-V2/REPLY/REQUEST_M2UIS65D0O7DFILVRH9QF80QKE, priority: 1
15:06:07,423 [pool-2-thread-2] INFO  DataServiceImpl:444  - Stored message on topic :$EDC/#account-name/new-client/DEPLOY-V2/REPLY/REQUEST_M2UIS65D0O7DFILVRH9QF80QKE, priority: 1
15:06:07,430 [DataServiceImpl:Submit] INFO  MqttDataTransport:512  - Publishing message on topic: $EDC/amir-kura/new-client/DEPLOY-V2/REPLY/REQUEST_M2UIS65D0O7DFILVRH9QF80QKE with QoS: 0
!SESSION 2017-05-16 14:51:45.651 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.8.0_121
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_IN
Command-line arguments:  -dev file:/home/amir/eclipse/workspaces/kura/.metadata/.plugins/org.eclipse.pde.core/Kura_Emulator_Linux/dev.properties -os linux -ws gtk -arch x86_64 -consoleLog -console

!ENTRY org.apache.felix.deploymentadmin 4 0 2017-05-16 15:06:07.499
!MESSAGE Stream does not contain a valid deployment package: missing manifest!
15:06:07,500 [pool-12-thread-1] INFO  InstallImpl:135  - Install failed!
15:06:07,509 [pool-12-thread-1] INFO  DataServiceImpl:441  - Storing message on topic :$EDC/#account-name/new-client/DEPLOY-V2/NOTIFY/test-client/install, priority: 1
15:06:07,535 [pool-12-thread-1] INFO  DataServiceImpl:444  - Stored message on topic :$EDC/#account-name/new-client/DEPLOY-V2/NOTIFY/test-client/install, priority: 1
15:06:07,550 [DataServiceImpl:Submit] INFO  MqttDataTransport:512  - Publishing message on topic: $EDC/amir-kura/new-client/DEPLOY-V2/NOTIFY/test-client/install with QoS: 1
15:06:07,684 [MQTT Call: test-client] INFO  DataServiceImpl:376  - Confirmed message ID: 1348 to store
15:06:07,935 [MQTT Call: test-client] INFO  DataServiceImpl:376  - Confirmed message ID: 1350 to store

According to the error I am sending an invalid deployment package, as it lacks Manifest, but I have created the .dp file as mentioned in the Hello World Example.

How do I solve this ? Noone else seems to have encountered this problem

Answer 1

I'm adding a second answer after @Amir comment. The other answer does not address the actual issue but may be of help in some other cases.

I think that the problem lies in the metrics sent to DEPLOY-V2 cloudlet.

dp.uri=file:/home/amir/Downloads/org.eclipse.kura.example.hello_osgi/resources/dp/hello_osgi.dp
dp.download.protocol=HTTPS

Also in the notification there's this indication:

dp.download.size=0

The URI says it's a FILE on the local filesystem, but the protocol says to use HTTPS.

If you look at DEPLOY-V2 documentation you will read that

Kura 1.4.0 introduces a new version of the DEPLOY application named “DEPLOY-V2”. The main feature introduced is related to a different approach for package distribution: instead of receiving the package inside a MQTT message, the device receives only a download request with all the information needed to perform an independent download via HTTP.
dp.uri (String). Mandatory. Represents the URI of the deployment package. dp.download.protocol (String) Mandatory. Specifies the protocol to be used to download the bundles/shell scripts. Must be set to HTTP or HTTPS.

You need to provide a valid HTTP URL to download the DP from.

Answer 2

There are 2 things to be aware of when building a Deployment Package.

1. Manifest must be the first entry in the DP file (which is simply a ZIP file)
2. Order of included bundles matters, they must be included in the DP in the same order as they are declared in the Manifest

Manifest as first entry

The MANIFEST.MF file is included in the META-INF/ directory. By ZIP standard you may add an entry in the file to create the directory, and a second entry with the file. This is not supported in DP format, so you must add directly the file with full path as very first entry in the resulting DP file.

For example, using maven-antrun-plugin to package the DP you can use filesonly="true" attribute on a <jar> command:

<jar destfile="${basedir}/../you.final.dp_1.0.0.dp"
     manifest="${project.build.directory}/dp_stage/META-INF/MANIFEST.MF"
     includes="" filesonly="true">
    <filelist dir="${project.build.directory}/dp_stage/" files="${sortedFiles}"/>
</jar>

Order of bundles

If your Deployment Package contains more than one bundle, they must be included in the same order as the Manifest. You can use alphabetical order, which is predictable and works well :-)
Copy your JARs in a folder and add them to the DP in a sorted fashion. Using the same maven-antrun-plugin:

<copy file="${basedir}/../project/target/first.bundle_1.0.0.jar"
      todir="${project.build.directory}/dp_stage/bundles/" />

<copy file="${basedir}/../project2/target/second.bundle_1.0.0.jar"
      todir="${project.build.directory}/dp_stage/bundles/" />

<!-- Files MUST be included in the jar in the same order as they are declared in the DP MANIFEST.
    I used alphabetical order. -->
<!-- https://ant.apache.org/manual/Tasks/jar.html -->
<local name="sortedFiles"/>
<pathconvert property="sortedFiles" pathsep="${line.separator}">
    <sort>
        <fileset dir="${project.build.directory}/dp_stage/" includes="bundles/*"/>
    </sort>
</pathconvert>

---

I have a personal project on github which does exactly this, have a look for reference.
POM which calls maven-antrun-plugin
Ant task to build DP